Environment
eDirectory
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12
RedHat 6.X
RedHat 7.X
Situation
eDirectory 8.8.8.X
eDirectory 9.X
SLES 11.X
SLES 12.X
RedHat 6.X
RedHat 7.X
ndsd memory usage grows beyond expected memory usage.
Expected memory can be determined by observing a trend over time or by calculating the expected memory adding: dib cache (see the cache option in the _ndsdb.ini, 300-400 MB for indexes and application overhead, 1 MB per thread in use, and any Java Max Heap settings added to drivers.
Note: ndsd memory footprint on Linux is expected to retain memory that it has requested, has been allocated and has been freed until the kernel requires the memory and reallocates memory.
Resolution
Enabling Google Perftools with ndsd
1. Add the following lines:
init.d systems - SLES 11.X & RedHat 6.X to the pre_ndsd_start script (default - /opt/novell/eDirectory/sbin/pre_ndsd_start)
export HEAP_PROFILE_INUSE_INTERVAL=52428800
export HEAP_PROFILE_ALLOCATION_INTERVAL=52428800
export HEAPPROFILE=/tmp/novell/ndsd_heap
export HEAP_PROFILE_MMAP=1
systemd systems - SLES 12.X & RedHat 7.X to the env file (default - /etc/opt/novell/eDirectory/conf/env)
HEAP_PROFILE_INUSE_INTERVAL=52428800
HEAP_PROFILE_ALLOCATION_INTERVAL=52428800
HEAPPROFILE=/tmp/novell/ndsd_heap
HEAP_PROFILE_MMAP=1
2. Modify the ndsd startup script to use libtcmalloc instead of libtcmalloc_minimal
init.d systems - SLES 11.X & RedHat 6.X (/etc/init.d/ndsd)
#if [ "$MALLOC_CHECK_" = "" ] && [ -f /usr/$libfldr/libtcmalloc_minimal.so.0 ]; then
if [ "$MALLOC_CHECK_" = "" ] && [ -f /usr/$libfldr/libtcmalloc.so.0 ]; then
if [ -f /etc/novell-release ]; then
#LD_PRELOAD="/usr/$libfldr/libtcmalloc_minimal.so.0" $sbindir/ndsd
LD_PRELOAD="/usr/$libfldr/libtcmalloc.so.0" $sbindir/ndsd
else
#LD_BIND_NOW=1 LD_PRELOAD="/usr/$libfldr/libtcmalloc_minimal.so.0" $sbindir/ndsd
LD_BIND_NOW=1 LD_PRELOAD="/usr/$libfldr/libtcmalloc.so.0" $sbindir/ndsd
fi
systemd systems - SLES 12.X & RedHat 7.X (/opt/novell/eDirectory/sbin/ndsdwrapper)
EXAMPLE:
if [ "$MALLOC_CHECK_" = "" ]; then
if [ -f /etc/novell-release ]; then
#LD_PRELOAD="/usr/lib64/libtcmalloc_minimal.so.4" $sbindir/ndsd
LD_PRELOAD="/usr/lib64/libtcmalloc.so.4" $sbindir/ndsd
else
#LD_BIND_NOW=1 LD_PRELOAD="$libdir/libtcmalloc_minimal.so.4" $sbindir/ndsd
LD_BIND_NOW=1 LD_PRELOAD="$libdir/libtcmalloc.so.4" $sbindir/ndsd
fi
else
$sbindir/ndsd
fi
Note: The actual libtcmalloc.so.0 may be named slightly different - Check the /usr/$libfldr (typically /usr/lib64) for libtcmalloc.so* and modify the file name as appropriate.
Note 2: Make sure MALLOC_CHECK_ is not set. This option configures ndsd to use malloc instead of tcmalloc which will negate the tcmalloc configuration.
3. Create the directory specified in HEAPPROFILE
EXAMPLE:
mkdir /tmp/novell
4. Modify the dib cache setting so that it is about 500 MB
EXAMPLE:
/var/opt/novell/eDirectory/data/dib/_ndsdb.ini
cache=500000000
5. Restart ndsd
init.d
/etc/init.d/ndsd restart
or
rcndsd restart
or
ndsmanage stopall && ndsmanage startall
systemd
ndsmanage stopall && ndsmanage startall
After the restart of ndsd heap files should be created in the HEAPPROFILE location if everything is configured correctly.
6. The script in the additional information section should be copied and run in conjunction with the Google Profiling configuration. The script run pprof against the heap files to generate profiling information. After pprof has been run by the script, the script will remove any excess heap files to prevent excessive accumulation of heap files.
7. Allow ndsd to run in conjunction with the memory profiling and the script until ndsd has substantially exhibited the memory consumption issue.
8. Create a gcore of the ndsd process and bundle with novell-getcore
The gdb package installed with the OS contains a gcore that will be used to create a core of the ndsd process
Be sure to run the gcore command which located in a directory that contains adequate space to create a core file the same size as the ndsd memory footprint. If ndsd is using 5 GB, make sure there is plenty of space to create at least a 5 GB file (suggest leaving room to spare)
EXAMPLE:
If the pid # of ndsd is 2354; then run:
gcore 2354
or
gcore `pidof ndsd`
Bundle the gcore core using novell-getcore - https://support.microfocus.com/kb/doc.php?id=3078409
9. Submit the novell-getcore bundle, all the files under the HEAPPROFILE directory, a support config and any steps that can be used to cause the memory consumption; if known, to technical support.
Cause
Many modules run within the ndsd memory space. If a module requests memory but doesn't properly free it under certain conditions, the memory footprint of ndsd can grow beyond the expected usage.
Tools like Google Perftools log memory allocations and release to help determine in a module could be failing to properly release memory.
Additional Information
HEAP_PROFILE_INUSE_INTERVAL - Dump heap profiling information whenever the high-water memory usage mark increases by the specified number of bytes.
HEAP_PROFILE_ALLOCATION_INTERVAL - Dump heap profiling information each time the specified number of bytes has been allocated by the program. Typically we recommend to to be set to 500MB so that a new heap profiling file is generated at every 500MB allocated by ndsd.
HEAPPROFILE - Sets the location where the heap profiling files are created and the prefix of the heap profiling file. The included script is configured to process files in /tmp/novell with the prefix ndsd_heap.
HEAP_PROFILE_MMAP - Sets profiling mmap to true
Script:
#! /bin/bash
# Note this script will remove heap files created by google profiler older than 10 minute so it does not fill up the disk space with heap files. If you observe heap files being created faster than the 10 minute interval keeps disk space usage under control, the value can be reduced as needed.
#Use crontab -e as root and put in the following to run the script every 10 minutes:
#Memory Profiler
#*/10 * * * * /root/googleMemoryProfiler.sh
LOCATION=/tmp/novell
KEEPLOCATION=/tmp/novell/keepheapfiles
createDir(){
if [ ! -d $LOCATION ]
then
/bin/mkdir -p $LOCATION && echo "Created $LOCATION directory for outputfiles"
else echo "Novell directory already created for outputfiles"
fi
}
createDir
createDir2(){
if [ ! -d $KEEPLOCATION ]
then
/bin/mkdir -p $KEEPLOCATION && echo "Created $KEEPLOCATION directory for outputfiles"
else echo "Novell directory already created for outputfiles"
fi
}
createDir2
OF=$LOCATION/googleMemoryProfiler.txt
FILENAMES=heap*
PATHTOCONF=/etc/opt/novell/eDirectory/conf/nds.conf
VARDIR=`/bin/cat $PATHTOCONF |grep -i vardir |awk 'BEGIN {FS="="} {print $2}'`
NDSDPID=`/bin/cat $VARDIR/ndsd.pid`
DATE=`which date`
TOP=`which top`
FREE=`which free`
FIND=`which find`
PPROF=/opt/novell/eDirectory/bin/pprof
CAT=`which cat`
echo " " >>$OF
echo "--------------------" $($DATE)", Start Data Collection --------------------" >>$OF
echo "##### TOP STATS FOR NDSD PROCESS #####">>$OF
$TOP -b -n1 -p $NDSDPID >>$OF
echo " " >>$OF
echo "##### FREE STATS #####">>$OF
$FREE -m >>$OF
echo " " >>$OF
#echo "#####/proc/meminfo#####" >>$OF
#$CAT /proc/meminfo >>$OF
echo " " >>$OF
echo "##### HEAP FILES REMOVAL #####" >>$OF
$DATE >>$OF
HEAPFILES=`ls /tmp/novell/ndsd_heap* |wc -l`
if [ $HEAPFILES -gt 1 ]
then
LATESTHEAPFILE=`$FIND $LOCATION/ndsd_heap* -type f | sort -nr |cut -d: -f2- |cut -d/ -f2- |head -1 |awk -F / '{print $3}'`
OLDESTHEAPFILE=`$FIND $LOCATION/ndsd_heap* -type f | sort -nr |cut -d: -f2- |cut -d/ -f2- |tail -1 |awk -F / '{print $3}'`
echo "Latest heap file which will be used by pprof = $LOCATION/$LATESTHEAPFILE" >> $OF
echo "Oldest heap file which will be used by pprof = $LOCATION/$OLDESTHEAPFILE" >> $OF
$PPROF --text /opt/novell/eDirectory/sbin/ndsd $LOCATION/$LATESTHEAPFILE >>$OF
$PPROF --svg /opt/novell/eDirectory/sbin/ndsd --base $LOCATION/$OLDESTHEAPFILE $LOCATION/$LATESTHEAPFILE >> $LOCATION/"SVG-"$OLDESTHEAPFILE"-"$LATESTHEAPFILE"."svg
$PPROF --pdf /opt/novell/eDirectory/sbin/ndsd --base $LOCATION/$OLDESTHEAPFILE $LOCATION/$LATESTHEAPFILE >> $LOCATION/"PDF-"$OLDESTHEAPFILE"-"$LATESTHEAPFILE"."pdf
echo "Moving $LOCATION/$LATESTHEAPFILE to $KEEPLOCATION" >> $OF
mv $LOCATION/$LATESTHEAPFILE $KEEPLOCATION
echo "Moving $LOCATION/$OLDESTHEAPFILE to $KEEPLOCATION" >> $OF
mv $LOCATION/$OLDESTHEAPFILE $KEEPLOCATION
echo "Deleting all heapfiles which do not = $LOCATION/$LATESTHEAPFILE" >>$OF
$FIND $LOCATION/ndsd_heap* -maxdepth 0 -name "$LATESTHEAPFILE" -prune -o -exec rm -f '{}' ';'
echo "--------------------" $($DATE)", Finished Data Collection --------------------" >>$OF
else
echo "Less than 2 files" >> $OF
LATESTHEAPFILE=`$FIND $LOCATION/ndsd_heap* -type f | xargs stat --format '%Y :%y %n' 2>/dev/null |sort -nr |cut -d: -f2- |cut -d/ -f2- |head -1 |awk -F / '{print $3}'`
$PPROF --text /opt/novell/eDirectory/sbin/ndsd $LOCATION/$LATESTHEAPFILE >>$OF
echo "Did not execute DELETING OF HEAP FILES because less than two heap files in /tmp/novell directory" >>$OF
echo "------------------------------------------------------------------------------------------------" >>$OF
echo " " >>$OF
fi
NOTE: For svg and pdf to work: graphviz and gv need to be installed