How to Configure Access Gateway Embedded Service Provider to Reduce Access Gateway Load and Improve Performance



By: ncashell

November 23, 2009 11:01 am

Reads: 617

Comments:6

Rating:0

Introduction:

The goal of the following document is to explain how to improve the Linux Access Gateway server performance and stability by including all attributes referenced by protected resource policies in a SAML assertion sent at authentication time.

In large production environments, it is commonplace to overload the Access Gateway to the point where utilization and server performance are negatively impacted. This document describes how attribute maps and SAML assertions can be used to significantly reduce traffic between Novell Access Manager Identity Servers and Access Gateways.

By understanding and taking advantage of some enhancements to Access Manager beginning with the release of 3.1 Support Pack 1 Interim Release 2, a lot of unnecessary work can be offloaded from the Access Gateway, improving performance, stability and the user experience.

Background information:

The Access Gateway (AG) is responsible for

  • Protecting web applications/services based on their distinct URLs
  • Providing required attributes to allow single sign on to back end applications/services.

Protection of such services often requires authentication. Because authentication is done at the Identity Server (IDS), the Access Gateway must be able to communicate with this IDS server to receive the authentication details. Such authentication details are sent via a SAML assertion.

The protection of services also requires authorization or single sign on decisions. Attributes required in the decision making process must be retrieved from the IDS, over what is called the SOAP back channel.

One can already design a solution leveraging roles (http://www.novell.com/documentation/novellaccessmanager31/policies/data/b995x1b.html) but there are many setups that require additional LDAP attributes.

Communication flow:

Although the AG host may have multiple proxy services defined, only ONE of those services hosts the Embedded Service Provider (ESP, and also known as the federation service) used to talk to the IDS via the SOAP back channel. Typically, the first reverse proxy/proxy service is used to host the federation service (via a reserved path of /nesp) although this is configurable. Any time the IDS needs to be invoked for authentication or policy evaluation, the user session details are always sent to the federation service (ESP) on the AG first before being redirected to the IDS. This AG federation service knows how to generate the required federation messages to send to the IDS.

When the AG proxy needs to execute a locally enabled policy (Identity Injection, Form Fill, or Authorization) the following steps are executed

  1. The AG proxy sends a SOAP request to its local ESP to evaluate the policy.
  2. If the ESP has all required identity information already cached, then the policy is evaluated locally and the response is returned to the proxy.

    If the ESP does not have the information cached, it will query the authoritative ESP (the AG ESP that was originally involved in authenticating the user and so establishing the user session). If this is NOT a clustered environment, then this step is omitted. This query of the authoritative ESP involves:

    1. Identifying which ESP in the cluster holds the user session details
    2. Proxy’ing the SOAP request to the authoritative ESP to try and evaluate the information required to satisfy the policy
  3. If the authoritative ESP has the required identity information, then it is returned to allow the policy to be evaluated. If not, then the initial ESP will query an IDS.
  4. If the IDS has the user information already cached, then it is returned. If not, and the IDS is not authoritative for this user session then the same process used by the ESP is used to locate the IDS holding the user session. The IDS then , and then sends the SOAP request to that authoritative IDS.

    As with the ESP communication above, this latter step is omitted if not in a clustered environment.

  5. If the authoritative IDS does not have the identity information already cached, then it will make an LDAP call to retrieve the required information from the user store and return it.

    As can be seen, the overhead (especially in clustered environments) of constantly communicating over the SOAP back channel during authentication and policy evaluation can have a major impact on performance.

    Typical issues seen in clustered environments include ESP CPU utilization going very high, Tomcat running out of threads, LDAP server overloading, data read timeout during the proxy’ing of requests to the authoritative ESP/IDS servers.

Performance improvement options:

In order to mitigate such overhead, administrators should consider the benefits of identifying the attributes required by all ESP policies and including them in SAML assertions sent between the IDS and ESP at authentication time. The assertion includes authentication statements (about the subject that it is authenticating) as well as attribute statements (attributes about the user). The attributes from this attribute statement will be cached at the ESP and used locally when evaluating policies. The end result is an elimination of the requests to the IDS to retrieve user attributes required to satisfy the policy.

The following guidelines, when configured correctly, will result in a huge reduction in traffic over the SOAP back channel. This resulting traffic reduction will generate a corresponding performance increase on the AG servers.

  1. Identify all attributes required on all policies enabled for each protected resource.

    Identify all policies that are enabled on each defined protected resource. Go through each of these policies and note the attributes that are required by this policy. In the following example, a single policy is enabled on one protected resource. Although this may not be realistic, this exercise will show how to verify that the policy requires the following attributes: all user roles, LDAP cn, LDAP roomNumber, LDAP mail and LDAP title attributes.

  2. Define an attribute set that will contain all attributes required by the AG enabled policies.

    After identifying all the required attributes from the previous step, the administrator must go to the ‘Shared Settings’ tab on the IDP configuration and define a new attribute set. After creating a new attribute set and giving it a logical name, add each attribute required by clicking the new option. In this example there will be 5 entries: all roles, LDAP cn, LDAP roomNumber, LDAP mail and LDAP title attributes.

    Note that the local attribute must include the attribute that the IDS will evaluate. There is an option to define the remote attribute name, but this is ignored for communications between IDS and ESP.

  3. Select the AG or AG cluster configuration where the newly defined Attribute Set will be used

    Go to the IDS configuration, and select the ‘Liberty’ tab. Under ‘Trusted Providers’, there will be a link to the AG cluster configuration name. If there is no AG cluster configuration, the IP address of the AG server will appear under this ‘Trusted Providers’ link.

  4. Add the newly defined Attribute Set to the Liberty relationship between IDS and selected ESP.

    After selecting the Trusted AG Service provider, select the attribute set defined in step 2 from the drop down menu. Once the attribute set is selected, the list of attributes from that Attribute set will appear on the right hand side of the screen. These are the attributes available for selection. Select each attribute and make sure that it moves across to the ‘Send with Authentication’ menu. Doing this will force the attributes to be resolved at authentication time, so that they are sent with the subject details in the SAML assertion.

  5. Define the attribute refresh rate on the policy

    When using LDAP attributes in an Identity Injection or Form Fill policy, the option to define a refresh rate exists. This refresh rate determines how often the AG proxy must go back to the ESP to determine whether the data is valid or stale. For performance purposes it is recommended that the ‘Session’ setting be defined, so that we only retrieve the attributes once during the session lifetime. Although no requests will go back over the back channel to the IDS server, it will reduce communication between the AG proxy and the ESP.

  6. Injecting IDS user name and password to back end Web server

    If the policy requires that the credential profile username and password be sent across to the back end Web server, the attribute map created above must include the credential profile details. Unlike regular LDAP attributes in the above example, these credential profile attributes MUST be mapped to a ‘Remote Attribute’ name. Note that this remote attribute name is case sensitive. The three credential profile attributes that need to be mapped are as follows:

    When defining the attributes to send to the back end Liberty ESP, we will only need to send the UserName and userPassword. The userDN may be left in the available list as it is already sent over in a SAML assertion by default at authentication time.

Validating configuration

Before rolling out the changes in production, there are a number of simple tests that an administrator can perform to confirm that no unnecessary SOAP back channel requests are being made from the ESP to the IDS server when policies are being evaluated.

  1. Turn on verbose logging at IDS server temporarily: Select the Identity Provider configuration tab in the Administration Console and click ‘Logging’. Under ‘Component File Logger’ set the following components to verbose : Application, Liberty, Web Service Provider and Consumer.

  2. Access a policy enabled protected resources on the AG and check the ESP and IDS log files. Look at the output of the catalina.out file on both the IDS/ESP servers

    The catalina.out file includes all debug information, assuming the above IDP log settings are enabled. This file is located in /var/opt/novell/tomcat5/logs/ on both IDP/ESP servers. Navigating these log files in debug mode can be confusing to say the least, so key entries have been identified to look for, and which server and log file they are found in.

    1. Verify that the SAML assertion sent at authentication time includes the AttributeStatement containing all defined attributes. This can be done by searching the catalina.out file on either the IDP or ESP for the “AttributeStatement” string. When a user authenticates to the above example setup, the output shown on both IDP/ESP servers is the following:
      <saml:AttributeStatement>
                        <saml:Subject>
                           <saml:NameIdentifier Format="urn:liberty:iff:nameid:one-time" NameQualifier="https://lag129.lab.novell.com:443/nesp/idff/
      metadata">
                            9xVBI/rKBwt4wLXim82945nMv+yYyrjmOFkNFg==
                           </saml:NameIdentifier>
                           <saml:SubjectConfirmation>
                              <saml:ConfirmationMethod>
                               urn:oasis:names:tc:SAML:1.0:cm:artifact
                              </saml:ConfirmationMethod>
                           </saml:SubjectConfirmation>
                        </saml:Subject>
                        <saml:Attribute AttributeName="ldapcn" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                         </saml:Attribute>
                        <saml:Attribute AttributeName="ldapmail" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                        </saml:Attribute>
                        <saml:Attribute AttributeName="userRoles" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                        </saml:Attribute>
                        <saml:Attribute AttributeName="title" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                           <saml:AttributeValue>
                            XX
                           </saml:AttributeValue>
                         </saml:Attribute>
                        <saml:Attribute AttributeName="roomnum" AttributeNamespace="urn:oasis:names:tc:SAML:1.0:assertion">
                           <saml:AttributeValue>
      					 
      

      When examining the log entry, note the following:

      Some attributes are multivalued (such as the userRoles attribute ), and will therefore have multiple “AttributeValue” entries.

      For security purposes, the “AttributeValue” includes an XX string and not the appropriate value

    2. Verify that the attribute statement is processed correctly and that the attribute values are added to the local cache. This is visible from the catalina.out file on the ESP, where a separate entry should exist for each attribute added. The one below shows the mail attribute being added to an internal ESP structure.
      <amLogEntry> 2009-09-30T11:10:25Z DEBUG NIDS WSC:
      Method: WSCCacheAlreadyReadCache.add
      Thread: http-127.0.0.1-8080-Processor22
      Added WSCCacheAlreadyReadCacheSet:
      NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~2
      2mail~22~5D </amLogEntry>
      
      
    3. Identify a AG policy referencing the required attributes and note the policy ID. This step involves looking through the /var/log/ics_dyn.log file on the Linux Access Gateway (LAG) when the LOG_LEVEL setting in /etc/laglogs.conf entry is set to 7 (default is 5). Search for the URL where the Identity Injection policy is enabled and locate the ‘Sending eval’ string. This includes an eval request and policyID that we can search for in the ESP to confirm that the data has been retrieved from local cache. The snippet below shows an eval request number of 2641 and a policyID of “57M81710-NL1N-L610-O816-54MM7N558146″. We can specifically see the LAG sending a SOAP request to the ESP, and whether or not a response was obtained.
      Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
      5C3FF802: AMEVENTID#410: IdInjection enabled for  the protected resource
      :
      Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
      5C3FF802: AMEVENTID#410: II:a5304b64 Sending EVAL Request 2641 policyId 57M81710-NL1N-L610-O816-54MM7N5
      58146
      Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#7079: proce
      ssSoapRequests - size 2 processed 1, deleted 0 (0, conFail 0 conTimeout 0) 0 (0)
      Sep 30 12:12:28 lag129 : AM#504515000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#0: Connection Established with peer 127.0.0.1:8080 (src 127.0.0.1:0)
      Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#2641: sentsoapRequest 2641  app a91de4c8  II
      Sep 30 12:12:28 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#2641: backchannel receivedResp (app   a91de4c8  II )   (2641)[seg:0xa4b87430:0xa59355a0:1248]
      Sep 30 12:12:28 lag129 : AM#504506000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#591E1B49DACAF72693531BCA
      5C3FF802: AMEVENTID#410: Received response for IdInjection EVAL request
      
      
    4. Confirm that the ESP received the request, and returned the response from cache. Failure to see these steps probably indicates that the ESP has had to contact the IDS to retrieve the requested information. This is done by searching the catalina.out on the ESP for the policyID or EVAL number. When the policy is found, verify that the attributes we are requesting are filled from cache. The snippet below shows the LDAP cn attribute being retrieved from cache.

      Note: The debug messages change with different versions. If your logs do not include the strings below, you can also confirm that there are no unnecessary SOAP back channel requests by searching for requests to the IDP /nidp/services/LDAPService URL in the ESP catalina.out file when the policy is evaluated. If there are entries, it indicates that a SOAP back channel request is being made for that attribute.

      <amLogEntry> 2009-09-30T11:12:28Z VERBOSE NIDS Application: AM#501101020: AMDEVICEID#esp-7AA324FFCBA4D4ED: NXPESID#2641:  <?xml version="1.0"
      encoding="UTF-8"?><Evaluate PolicyId="57M81710-NL1N-L610-O816-54MM7N558146" Verbose="on">
                                      <ContextDataElement Enum="2551" Value="591E1B49DACAF72693531BCA5C3FF802"/>
                              </Evaluate> </amLogEntry>
      
      <amLogEntry> 2009-09-30T11:12:28Z INFO NIDS Application: AM#501101050: AMDEVICEID#esp-7AA324FFCBA4D4ED: PolicyID#57M81710-NL1N-L610-O816-54MM7
      N558146: NXPESID#2641:  Evaluating policy </amLogEntry>
      
      :
      :
      
      <amLogEntry> 2009-09-30T11:12:28Z DEBUG NIDS WSC:
      Method: WSC.fillFromCache
      Thread: http-127.0.0.1-8080-Processor21
      Processing set: NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22cn~22~5D </amLogEntry>
      
      <amLogEntry> 2009-09-30T11:12:28Z DEBUG NIDS WSC:
      Method: WSC.fillFromCache
      Thread: http-127.0.0.1-8080-Processor21
      Request filled from WSC Already Read Cache! </amLogEntry>
      
      <amLogEntry> 2009-09-30T11:12:28Z INFO NIDS WSP: AM#500103001: AMDEVICEID#esp-7AA324FFCBA4D4ED: AMAUTHID#591E1B49DACAF72693531BCA5C3FF802:  Fi
      lled the user attribute request from data already in the web service consumer cache. </amLogEntry>
      
      <amLogEntry> 2009-09-30T11:12:28Z INFO NIDS Application: AM#501101056: AMDEVICEID#esp-7AA324FFCBA4D4ED: AMAUTHID#591E1B49DACAF72693531BCA5C3FF
      802: PolicyID#57M81710-NL1N-L610-O816-54MM7N558146: NXPESID#2641:  Data retrieval ok:  from cached String[] value </amLogEntry>
      
      

      After validating this information, ensure that all log levels are set back to the default.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)

Tags: ,
Categories: Uncategorized

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment.  It just worked for at least one person, and perhaps it will be useful for you too.  Be sure to test in a non-production environment.

6 Comments

  1. By:mrandall_cerner_com

    Great info, Neil! Does this also mean now that if I create an Identity Injection policy and insert SAML Credentials -> SAML Assertion that the assertions will also carry over? We’ve been looking at how to duplicate having SP code in our SAML aware apps and the Access Gateway to consume assertion data.

    • By:ncashell

      Hi Matt, I just saw this post – sorry about the delay. The II option to post the assertion takes the assertion we receive from the IDP during auth, and injects the subject details (authenticationStatement) back to the ESP. It does not appear to send the attributes. If your back end just needs the user specific info e.g. NameIdentifier info, we will be fine.

      HTH, Neil

  2. By:Elfstone2

    While at the top of the article it recommends setting VERBOSE on each of the four component types, to follow along the examples in your own environment you actually need to be in DEBUG mode (as even shown in the screenshots of various logs).

  3. By:jwilleke

    With the latest support packs, does this still add an performance improvement?

Comment