8.2 Improving Bulkload Performance

eDirectory 8.8 provides you with new options to increase the bulkload performance.

The following are the tunable parameters for bulkload performance using the NetIQ Import Convert Export (ICE) utility.

Also refer to the various operating system tunable parameters.

8.2.1 eDirectory Cache Settings

To optimize the bulkload performance, allocate a higher percentage of the eDirectory cache for block cache.

For more details refer to Tuning eDirectory Subsystems in the NetIQ eDirectory 8.8 SP8 Tuning Guide.

8.2.2 LBURP Transaction Size Setting

The LBURP transaction size sets the number of records that are sent from ICE to the LDAP server during a single transaction. Increasing this value can improve bulkload performance, assuming that you have adequate memory and that the increase does not cause I/O contention.

The default transaction size is 25, which is appropriate for small LDIF files (fewer than 100,000 operations) but not for a large number of records. The LBURP transaction size can be set between 1 and 350.

Modifying the Transaction Size

To modify the transaction size, modify the required value for the n4u.ldap.lburp.transize parameter in /etc/opt/novell/eDirectory/conf/nds.conf. In ideal scenarios, a higher transaction size ensures faster performance. However, the transaction size must not be set to arbitrarily high values for the following reasons:

  • A larger transaction size requires the server to allocate more memory to process the transaction. If the system is running low on memory, this can cause a slowdown due to swapping.

  • The LDIF file should be free of errors and any entries already existing in eDirectory should be commented out. Even if a single error exists in the transaction (including cases where the object to be added already exists in the directory), eDirectory ignores the LBURP transaction setting and performs a commit after each operation to ensure data integrity.

    See Debugging LDIF Files for more information.

  • LBURP optimization works only for leaf objects. If the transaction contains both a container and its subordinate objects, eDirectory treats this as an error. To avoid this, we recommend loading the container objects first using a separate LDIF file or enables the use of forward references.

    For more information, see Enabling Forward References in the NetIQ eDirectory 8.8 SP8 Troubleshooting Guide.

8.2.3 Increasing the Number of Asynchronous Requests in ICE

This refers to the number of entries the ICE client can send to the LDAP server asynchronously before waiting for any result back from the server.

The number of asynchronous requests can be set between 10 and 200. The default value is 100. Any value less than the minimum value (10) would fallback to the default. The minimum value is appropriate for small LDIF files.

In ideal scenarios, a higher window size ensures faster performance. However, the window size must not be set to arbitrarily high values because a larger window size requires the client to allocate more memory to process the entries in the LDIF file. If the system is running low on memory, this can cause a slowdown due to swapping.

You can modify the number of asynchronous requests in ICE using either the ICE command line option or iManager.

Using ICE Command Line Option

The number of asynchronous requests can be specified using the ICE command line option -Z. This is available as part of the LDAP destination handler.

To set the number of asynchronous requests sent by the ICE client to 50, you would enter the following command:

ice -SLDIF -f LDIF_file -a -c -DLDAP -d cn_of_admin -Z50 -w password

Using iManager ICE Wizard

To set the number of asynchronous requests sent by the ICE client through iManager:

  1. Click the Roles and Tasks button Roles Icon.

  2. Click eDirectory Maintenance > Import Convert Export Wizard.

  3. Type the value in the LBURP Window Size field in the LDAP Destination Handler screens in both the Importing Data from a File and Migrating Data between LDAP Servers tasks.

  4. Click Next.

    For more information, refer to the help provided in the Wizard.

8.2.4 Increased Number of LDAP Writer Threads

The LDAP server now has multiple writer threads. Use the -F ICE command line option for enabling forward referencing to avoid any possible errors due to concurrent processing as follows:

ice -SLDIF -f LDIF_file -a -c -DLDAP -d cn_of_admin -w password  -F

8.2.5 Disabling Schema Validation in ICE

Use the -C and -n ICE command line options to disable schema validation at the ICE client as follows:

ice -C -n -SLDIF -f LDIF_file -a -c -DLDAP -d cn_of_admin -w password

8.2.6 Disabling ACL Templates

You can disable the Access Control List (ACL) templates to increase the bulkload performance. The implication of this is that some of the ACLs will be missing. However, you can resolve this by adding the required ACLs to the LDIF file or applying them later.

  1. Run the following command:

    ldapsearch -D cn_of_admin -w password -b cn=schema -s base objectclasses=inetorgperson 
    

    The output of this command would be similar to the following:

    dn: cn=schema
    objectClasses: ( 2.16.840.1.113730.3.2.2 NAME ’inetOrgPerson’ SUP
     organizationalPerson STRUCTURAL MAY ( groupMembership $ ndsHomeDirectory
     $ loginAllowedTi meMap $ loginDisabled $ loginExpirationTime $
     loginGraceLimit $ loginGraceRem aining $ loginIntruderAddress $
     loginIntruderAttempts $ loginIntruderResetTim e $
     loginMaximumSimultaneous $ loginScript $ loginTime $
     networkAddressRestri ction $ networkAddress $ passwordsUsed $
     passwordAllowChange $ passwordExpirationInterval $
     passwordExpirationTime $passwordMinimumLength $ passwordRequired $
     passwordUniqueRequired $ printJobConfiguration $ privateKey $ Profile $ 
     publicKey $ securityEquals $ accountBalance $ allowUnlimitedCredit $
     minimum AccountBalance $ messageServer $ Language $ UID $
     lockedByIntruder $ serverHolds $ lastLoginTime $ typeCreatorMap $
     higherPrivileges $ printerControl $ securityFlags $ profileMembership $
     Timezone $ sASServiceDN $ sASSecretStore $ sASSecretStoreKey $
     sASSecretStoreData $ sASPKIStoreKeys $ userCertificate
     $nDSPKIUserCertificateInfo $ nDSPKIKeystore $ rADIUSActiveConnections $
     rADIUS AttributeLists $ rADIUSConcurrentLimit $ rADIUSConnectionHistory
     $ rADIUSDefa ultProfile $ rADIUSDialAccessGroup $ rADIUSEnableDialAccess
     $ rADIUSPassword $ rADIUSServiceList $ audio $ businessCategory $
     carLicense $ departmentNumbe r $ employeeNumber $ employeeType $
     givenName $ homePhone $ homePostalAddress  $ initials $ jpegPhoto $
     labeledUri $ mail $ manager $ mobile $ pager $ ldap Photo $
     preferredLanguage $ roomNumber $ secretary $ uid $ userSMIMECertifica te
     $ x500UniqueIdentifier $ displayName $ userPKCS12 ) X-NDS_NAME ’User’ X
    -NDS_NOT_CONTAINER ’1’ X-NDS_NONREMOVABLE ’1’ X-NDS_ACL_TEMPLATES ( ’2#subtree#[Self]#[All Attributes Rights]’ ’6#entry#[Self]#loginScript’ ’1#subtree#[Root Template]#[Entry Rights]’ ’2#entry#[Public]#messageServer’ ’2#entry#[Root Template]#groupMembership’ ’6#entry#[Self]#printJobConfiguration’ ’2#entry#[Root  Template]#networkAddress’) )
    
  2. In the output noted in the previous step, delete the information marked in bold.

  3. Save the revised output as an LDIF file.

  4. Add the following information to the newly saved LDIF file:

    dn: cn=schema
    changetype: modify
    delete: objectclasses
    objectclasses: ( 2.16.840.1.113730.3.2.2 )-add:objectclasses
    

    Therefore, your LDIF should now be similar to the following:

    dn: cn=schema
    changetype: modify
    delete: objectclasses
    objectclasses: ( 2.16.840.1.113730.3.2.2) 
    -
    add:objectclasses
    objectClasses: ( 2.16.840.1.113730.3.2.2 NAME ’inetOrgPerson’ SUP
     organization alPerson STRUCTURAL MAY ( groupMembership $ ndsHomeDirectory
     $ loginAllowedTimeMap $ loginDisabled $ loginExpirationTime $
     loginGraceLimit $ loginGraceRem aining $ loginIntruderAddress $
     loginIntruderAttempts $ loginIntruderResetTime $
     loginMaximumSimultaneous $ loginScript $ loginTime $
     networkAddressRestri ction $ networkAddress $ passwordsUsed $
     passwordAllowChange $ passwordExpirationInterval $
     passwordExpirationTime $ passwordMinimumLength $ passwordRequired
     $passwordUniqueRequired $ printJobConfiguration $ privateKey $ Profile $ 
     publicKey $ securityEquals $ accountBalance $ allowUnlimitedCredit $
     minimum AccountBalance $ messageServer $ Language $ UID $
     lockedByIntruder $ serverHolds $ lastLoginTime $ typeCreatorMap $
     higherPrivileges $ printerControl $ securityFlags $ profileMembership $
     Timezone $ sASServiceDN $ sASSecretStore $ sASSecretStoreKey $
     sASSecretStoreData $ sASPKIStoreKeys $ userCertificate $
     nDSPKIUserCertificateInfo $ nDSPKIKeystore $ rADIUSActiveConnections $
     rADIUSAttributeLists $ rADIUSConcurrentLimit $ rADIUSConnectionHistory $
     rADIUSDefa ultProfile $ rADIUSDialAccessGroup $ rADIUSEnableDialAccess
     $rADIUSPassword $ rADIUSServiceList $ audio $ businessCategory $
     carLicense
     $ departmentNumbe r $ employeeNumber $ employeeType $ givenName $
     homePhone $ homePostalAddress  $ initials $ jpegPhoto $ labeledUri $ mail
     $ manager $ mobile $ pager $ ldap Photo $ preferredLanguage $ roomNumber
     $ secretary $ uid $ userSMIMECertifica te $ x500UniqueIdentifier $
     displayName $ userPKCS12 ) X-NDS_NAME ’User’ X-ND S_NOT_CONTAINER ’1’ X
    -NDS_NONREMOVABLE ’1’)
    
  5. Enter the following command:

    ldapmodify -D cn_of_admin -w password -f LDIF_file_name
    

For more information on working with ACLs, refer to the NetIQ eDirectory 8.8 SP8 Tuning Guide.

8.2.7 Backlinker

Backlinker is a background process that checks the referential integrity among other checks runs 50 minutes after the eDirectory server comes up. The subsequent time it runs is after 13 hours. Ensure that backlinker does not run during the bulkload process. In case backlinker runs, depending on the time and the number of objects loaded, backlinker can hinder the bulkload.

8.2.8 Enabling/Disabling Inline Cache

You can enable or disable the Inline Change Cache for a server. You can disable Inline Change Cache only when Outbound Synchronization is disabled. Enabling Outbound Synchronization also enables Inline Change Cache.

Disabling Inline Change Cache marks the change cache as invalid for this replica and tags it with an invalid flag in Agent Configuration > Partitions. Enabling Inline Change Cache removes the invalid change cache flag when the change cache is rebuilt.

8.2.9 Increasing the LBURP Time Out Period

By default, the time out period for a client is 20 minutes (1200 seconds). But during bulkload, with the LBURP transaction size as high as 250, objects with large number of attributes with huge values for these attributes, and with LBURP concurrent processing enabled at the server, the server gets busy processing data pumped in by the ICE client without responding to the client in the stipulated time. This times out the ICE client.

Therefore, we recommend you to increase the time out period. You can do this by exporting the environment variable LBURP_TIMEOUT with high values (in seconds).

For example, to export the LBURP_TIMEOUT variable with 1200 seconds, enter the following:

export ICE_LBURP_TIMEOUT=1200