Recently, I got hit with the sporadic Missing LAG/”Null Value” bug. For those that are unfamiliar with this little feature, I’ll give you the scenario. Perhaps a reboot of the Access Console when eDirectory has not finished a sync with Tomcat or for what ever reason. You open up the Access Console’d iManager and notice that you’re either missing a Access Gateway, a Gateway Cluster or similar.
In investigating, you might see that in the Troubleshooting section a gateway device has become corrupt. You can take this associated ID, and using the View Objects button in iManager, browse to the following container and select the object with that ID to view it.
Since iManager doesn’t have a plug-in for the Access Manager schema, you’ll be presented with the Other tab. Look for the Attribute, “romaAGDeviceXMLDoc” and click Edit. You will notice that it is blank.
Novell TID 7005800 recommends clicking the Repair button, under Audit->Troubleshooting->Configuration, scroll down to Devices with Corrupt Data Store Entries in the Access Console. If this fails, one option is to send Novell Support your last AM backup file along with the password and the value can be sent back to you. Then, you can simply copy and paste the data back into the object as described above, logout of the Access Console, wait 3-5 minutes and log back in to the Console to see the restored Gateway.
Thinking to myself, “That was relatively painless”, I opened FireFox and clicked the bookmark for my Access Console.
Here, is what happened next.
I logged back into the Access Console and was faced with something new and different.
There were no options, just a nice little picture.
I went and checked the logs on the Access Console server (currently a Windows Server). I opened C:\Program Files\Novell\logs\app_sc.0.log and noticed many java errors similar to this:
5853(D)2010-09-04T00:27:00Z(L)application.sc.core(T)29(C)com.volera.vcdn.application.sc.core.Info(M)getDocument(E)org.jdom.input.JDOMParseException: Error on line 2: Attribute name "Manage" associated with an element type "romaAGDevice" must be followed by the ' = ' character.
I went back to iManager and browsed back to that particular AG object in the tree.
Since I knew that the “romaAGDevice”, mentioned in the log, was relative to that attribute “romaAGDeviceXMLDoc”, I opened the related object to look at with further examination.
I found the word Manage and noticed a space and a carriage return (CR) after it. XML does not allow for stray characters within fields. Because Manage had a space after it, the XML parser was expecting an equals “=” and then a value.
The XML tag should have read, “ManagementAddress=192.168.1.2″. I, very carefully, removed the space and CR and applied my change. I logged off the Console and restarted Tomcat on the Access Console server.
When I logged back into the Access Console, I got the same screen as above and immediately checked the logs again. Apparently there were more of these stray spaces in the attribute’s value. This time, I went through all of the values to make sure they were clean. I found 4 other places where a space had been inserted. I cleaned the value and applied my changes. Restart Tomcat once again.
I logged in to the Access Console, once again and found that all was right with the world and I could also see and open the Access Gateways.
Since I had just remedied the issue, I immediately took a new backup of the Access Console.
Access Manager does not have a built in backup scheduler and must be backed up manually from the Access Console. It is recommended that a backup be performed prior to any changes to any of the components, Identity Providers (IDP), Access Gateway configurations or Reverse Proxy. Then after the changes have been made and shown successful, perform another backup to ensure a current backup of the new configuration.
If I had not had a fairly recent backup of the environment, there is a good chance that the my LAGs would have had to be redone and re-added back into the cluster.