Just when you think it is safe to go back in the water, someone throws in a Shark. I have spent some time getting familiar with the SysVinit processes, and how to run NDSD Command Options. Anybody familiar with troubleshooting eDirectory probably knows about this;
rcndsd start -ndb
That starts NDSD without loading the database. Useful when trying to repair the dib set when it refuses to load properly.
Did you know there were, at least, two other command options?
-rdb Restricted Mode. Skips some of the security features.
-igrfl Loads without loading Roll Forward Logging (useful, as we will see in a bit).
A little background, we are talking about eDirectory 220.127.116.11 (also known as Snowman) running on Red Hat 7.2. This was a DEV server, but we had an ongoing development effort that was nearing production roll-out. No production animal’s user objects were harmed during this process.
We recently ran into a situation that everyone dreads, the full disk. Turns out Roll Forward Logging was turned on without setting up a separate nds.rfl partition (oops…). Not only that, but someone setup a test driver, in IDM, and left the trace file size at unlimited, trace level 5, with the driver running in a looping state. I have never seen a 17 gigabyte trace file before! So, we had a panic mode operation. Disk space was down to zero on the partition that holds the DIB set. My brother was in the Navy, and I’ll always remember one of his Naval Axioms… When in Panic, When in Doubt, Run in Circles, Scream and Shout! In his favor, he was in Submarines, and if someone did that on a Submarine, they usually got shot out of the Torpedo Tube. They are very level headed on Submarines. How my brother got there is beyond me, but I digress…
Ok, start deleting files. Trace files, gone… RFL Files, gone… Free space now at 48%… We are good to go… Oops, forgot to turn off Roll Forward Logging. Now, restart eDir and BOOM, instant failure. It turns out eDirectory is not able to find what should be the latest Roll Forward Log and the DIB set is locked. We can no longer access the locked DIB Set so nothing is working. No development, no drivers, no LDAP. no nothing!
Not to worry, there is a very good TID that describes how to fix that:
In there they say to do this…
rcndsd start -igrfl This turns off Roll Forward Logging for this load, which allows eDirectory to startup…
… and then do…
dsbk setconfig -l This turns off Roll Forward Logging for good.
Restart edirectory and all should be good, right?
Did I mention, this was on Red Hat 7.2???
Now, with Red Hat 7, and SLES 12, someone found a solution looking for a problem, called SystemD. This changes how services are setup and loaded. All that knowledge you accumulated on SysVInit, does not apply. You may have seen the ubiquitous error message on these servers when you try to run rcndsd. It says you cannot do that anymore (Grrrr!!!). There is a new sheriff in town! Ok, so how in the world do you get the -igrfl command option in there with this great new system???
Google was not very friendly in this case, as nothing could be found online. Not in the forums, not anywhere. So, time to start hacking. First stop, ndsmanage. Since that is the accepted method for starting and stopping eDir from the command line now, see if we could slip the command option in there, somewhere on the command line. No Joy! It only accepts what it needs to start an instance, or get to the menu. Everything else is ignored (how rude!).
Well FUBAR!!! Next up, start hacking the scripts. We did manage to find our way to the actual script line that loaded the ndsd binary. From there, we added the -igrfl option and let it fly! No Joy!!! Right about know I felt like Charlie Brown trying to kick a field goal! Watching the command execute in the logs, we could see the -igrfl set, but it did nothing. We still got the Roll Forward Logging -6024 error. Seeing the command option there should have been a clue. We could see the …sbin/ndsd -igrfl, but it did nothing. That made no sense, but it did not set any ideas to look elsewhere.
What was left, besides beating our head against the wall (which feels real good when you stop) was to open a Service Request.
Now, here I am going to put in a plug for NTS. This was a holiday weekend, followed by a week between Christmas and New Years, when nobody really seems to want to work. Yet here we were with a stalled Development effort and a pending Production Roll Out. Still, NTS is there for you! We had to call and have handoffs rearranged, but we got our answer in a timely manner.
What we found out, and what several Knowledge Partners (who shall remain nameless), and myself, did not catch, was that those nifty command options we have grown to know, are not really command options! They get turned into Environment settings somewhere within the myriad of interconnecting scripts. Somewhere there is code that interprets what options are there, and converts them.
Here is what we received from the folks at NTS:
If you take a look at the ndsd script (even in a SLES 12 box/RHEL 7 server), you’ll see that what we used to specify as command line options to pass in to ndsd are actually translated into environment variables:
if [ “$1” = “start” ]; then if [ “$2” = “-ndb” ]; then NDSD_DONT_OPEN_AGENT=Y export NDSD_DONT_OPEN_AGENT StartNdsd elif [ “$2” = “-rdb” ]; then RESTRICTED_MODE=Y export RESTRICTED_MODE StartNdsd elif [ “$2” = “-igrfl” ]; then NDSD_IGNORE_RFL=Y export NDSD_IGNORE_RFL StartNdsd Else
So, if you want to start up eDirectory ignoring rfl, you would need to pass in the NDSD_IGNORE_RFL=Y variable in the environment. In SLES 12/RHEL 7 this is accomplished by modifying either the env or the env_custom file, normally located in /etc/opt/novell/eDirectory/conf/.
I am not going to get into a detailed technical discussion on how SystemD works. Here is a document that helps: SystemD Unit
What we have is a system of files that describe what is needed, and how to get things loaded. Here is the service file for NDSD on our Red Hat 7.2 server (someone got a little carried away with the name):
root@idmserver conf]# cat /usr/lib/systemd/system/ndsdtmpl-var-opt-novell-eDirectory-conf-nds.conf@.service [Unit] Description=eDirectory service for %I After=network.target local-fs.target [Service] Type=forking RemainAfterExit=no PIDFile=/var/opt/novell/eDirectory/data/ndsd.pid LimitCORE=infinity EnvironmentFile=%I EnvironmentFile=-/etc/opt/novell/eDirectory/conf/env_idm EnvironmentFile=-/etc/opt/novell/eDirectory/conf/env_custom ExecStartPre=-//opt/novell/eDirectory/sbin/pre_ndsd_start_custom ExecStartPre=-//opt/novell/eDirectory/sbin/pre_ndsd_start_factory ExecStart=/opt/novell/eDirectory/sbin/ndsdwrapper ExecStartPost=-//opt/novell/eDirectory/sbin/post_ndsd_start_custom ExecStartPost=-//opt/novell/eDirectory/sbin/post_ndsd_start_factory ExecStopPost=-//opt/novell/eDirectory/sbin/post_ndsd_stop_custom ExecStopPost=-//opt/novell/eDirectory/sbin/post_ndsd_stop_factory TimeoutStopSec=180 [Install] WantedBy=multi-user.target
In there, you can see three EnvironmentFile settings. The first is %I. I am still trying to figure that one out, but it appears to be a system variable, so we won’t mess with that. The other two are files, kindly provided with full path settings. Neither of them existed in that folder on our server. There is, however, an env.template file:
NDS_CONF="/etc/opt/novell/eDirectory/conf/nds.conf" TEXTDOMAINDIR="NDSHOME/opt/novell/eDirectory/share/locale" libdir="NDSHOME/opt/novell/eDirectory/lib64" LD_LIBRARY_PATH=NDSHOME/opt/novell/eDirectory/lib64:NDSHOME/opt/novell/eDirectory/lib64/nds-modules:NDSHOME/opt/novell/eDirectory/lib64/apr:NDSHOME/opt/novell/lib64:/opt/novell/lib64:$LD_LIBRARY_PATH PATH=$PATH:/usr/local/bin:/usr/bin:/bin sbindir="NDSHOME/opt/novell/eDirectory/sbin"
Look familiar? That is very similar to what the ndspath script sets up. Ok, now we are getting somewhere. Copy the env.template to a new env_idm file. Note: it says env_idm in the services file, not env as mentioned by NTS. This may be a difference between Red Hat 7.2 and SLES 12. Add in the NDSD_IGNORE_RFL=Y and fire up ndsd. If you are doing this, it helps to open a second command line window and do a tailf on /var/opt/novell/eDirectory/logs/ndsd.log. Watching this, you can see what happens quickly. What we saw happen, quickly, was ndsd crashing and burning. It seems the env.template settings are more for crashing eDirectory on startup than providing a sample env_<whatever> sample. That could be due to the settings already being in place, who knows. I did not investigate that situation.
SO, remove the template, add back just the environment setting and bada-boom, bada-bing, eDirectory is now started without missing it’s Roll Forward Logging files. Roll Forward Logging is now disabled for this load, we can do the dsbk setconfig -l (lower case L) and turn Roll Forward Logging off for good. Once you get eDirectory started on it’s own (no -igrfl), do not forget to turn Roll Forward Logging back on if you still need it.
Bottom line lessons learned here.
One thing I did not try, and may be an alternative. Set the environment variable from the command line, and start up eDirectory using ndsmanage:
That may be a more expedient way to get things done. Next person can try that out and post a comment here. Just don’t forget to clear the environment variable when you are done, or you may have a problem when you try to do a restore later and find you have no Roll Forward Logs.
This is not intended to be a lesson on SystemD. I’ll leave that for others. This is a “get you back on the road, quickly, before it starts raining” type of fix. That is what comes from fixing motorcycle breakdowns, on the side of the road, with zip ties and Gorilla Tape.
Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment. It just worked for at least one person, and perhaps it will be useful for you too. Be sure to test in a non-production environment.