NetIQ Access Manager IDP server generates invalid SAML2 response messages signature

  • 7024052
  • 07-Aug-2019
  • 19-Nov-2019

Environment

  • Access Manager 4.4
  • Access Manager 4.5

Situation

  • NetIQ Access Manager IDP cluster with 4 nodes
  • Only one node out of the cluster generates signed SAML Response messages which can be validated
  • All nodes are running the exact same NAM version

  • The following configuration files have been confirmed to be the same
    • /opt/novell/nids/lib/webapp/WEB-INF/classes/nidpconfig.properties
    • /opt/novell/nam/idp/conf/tomcat.conf
    • /opt/novell/nam/idp/conf/server.xml (only IP addresses are different)
    • /opt/novell/nam/idp/conf/web.xml
    • /opt/novell/nids/lib/webapp/WEB-INF/web.xml
    • /opt/novell/nids/lib/webapp/META-INF/context.xml

  • The following packages / librariess have been confirmed to be the same
    • all installed rpm packages which had been installed
    • /opt/novell/nids/lib/webapp/WEB-INF/lib/xmlsec.jar

  • All Java keystores have been validated to store the same signing certificates with the correct alias
    • /opt/novell/devman/jcc/certs/idp/signing.keystore

Resolution

  • remove the "/opt/novell/nids/lib/webapp/WEB-INF/classes/org/apache/xml/security/utils/XMLUtils.class"
    (not shipped with a NAM standard installation) and restart the IDP server

  • As seen from the class loader log example (Cause Section of this TID) it is important to load the XMLUtils.class from the higgins-sts-xmlsecurity-apache_1.0.0.jar. We have seen situations where this will not be the case even there is no "/opt/novell/nids/lib/webapp/WEB-INF/classes/org/apache/xml/security/utils/XMLUtils.class". A JAR is basically a zip file which can be extracted.

    In order to force this you can extract the
    higgins-sts-xmlsecurity-apache_1.0.0.jar. The "\higgins-sts-xmlsecurity-apache_1.0.0\org\apache\xml\security\utils" will include the XMLUtils.class which you can copy over to the "/opt/novell/nids/lib/webapp/WEB-INF/classes/org/apache/xml/security" to make sure it is getting loaded before any JAR will be loaded.

    File and directory ownership should be in all cases "novlwww.novlwww"

    Note
    : for the solution above removing the class the wrong XMLUtils.class had been placed in

  • This issue has been addressed as well to engineering in order to find a solution avoiding such situations in the future

Cause

  • On the cluster nodes failing to generate a correct signature an extra / old XMLUtils.class has been found in the "/opt/novell/nids/lib/webapp/WEB-INF/classes/org/apache/xml/security/" directory. This class is usually part of the "/opt/novell/nids/lib/webapp/WEB-INF/lib/xmlsec.jar" that is part of the "novell-nidp-server-[NAM Verson].noarch.rpm"

  • Tomcat will load classes before JARs. Therefore the above mentioned XMLUtils.class got loaded before any class shipped with the xmlsec..jar

    Class loader debug output from a failing server
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    [Loaded org.apache.xml.security.c14n.InvalidCanonicalizerException from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.utils.XMLUtils from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/classes/]
    [Loaded org.apache.xml.security.c14n.CanonicalizationException from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.utils.XMLUtils$1 from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/classes/]
    [Loaded org.apache.xml.security.c14n.CanonicalizerSpi from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.c14n.implementations.CanonicalizerBase from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.c14n.implementations.Canonicalizer20010315 from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

    Class loader debug output from a working server
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    Loaded org.apache.xml.security.exceptions.AlgorithmAlreadyRegisteredException from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]

    [Loaded org.apache.xml.security.c14n.InvalidCanonicalizerException from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.utils.XMLUtils from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/higgins-sts-xmlsecurity-apache_1.0.0.jar]
    [Loaded org.apache.xml.security.c14n.CanonicalizationException from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]
    [Loaded org.apache.xml.security.c14n.CanonicalizerSpi from file:/opt/novell/nam/idp/webapps/nidp/WEB-INF/lib/xmlsec.jar]

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Additional Information


Note: With SAML using the XML signature standard a SAML Assertion or a signed SAML Assertion can be send enclosed in a signed SAML Response Message.


Tools used to debug the signing issue:
  • Fiddler header trace tool in order to access to the posted SAML Assertion. You can find the message on which the SAML Response gets posted (Base64 encoded). If you select the "WebForms" view you will see the SAML Response Field. Right click on the value (even if the field looks to be empty) and run "send to TextWizard". Usually fiddler will automatically run a Base64 decode. You can copy and paste this Assertion / SAML Response in order to save it into a text file for validating any signatures included.

    Note: Using the Firefox SAML Tracer browser plugin from https://github.com/UNINETT/SAML-tracer will fail using copy and paste from the SAML tab. The Assertion rendered in this tab has been modified to make it more human readable and therefore cannot be used to validate signatures. You can save the trace here as a JSON file and get the raw / unmodified Basa64 encoded SAML response from there if you want to sue this tool

  • IDP cluster auditing and logging
    • File Logging: enabled
    • Echo to Console: enabled
    • Component File Logger Levels
    • File Logging  Enabled
    • Echo To Console: checked
    • Application, Liberty, SAML2: debug
  • xmlsec debug logging with log4j
    • cp /var/opt/novell/iManager/nps/WEB-INF/lib/log4j-1.2.17.jar /opt/novell/nam/idp/endorsed/
    • chown novlwww.novlwww /opt/novell/nam/idp/endorsed/log4j-1.2.17.jar
    • vi /opt/novell/nids/lib/webapp/WEB-INF/classes/log4j.properties
    • add the following "log4j.logger.org.apache.xml.security=DEBUG" entry
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      # Set root logger priority to FATAL and its only appender to CONSOLE.

      log4j.rootCategory=DEBUG, CONSOLE
      CONSOLE is set to be a ConsoleAppender using a PatternLayout.
      log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
      log4j.appender.CONSOLE.Threshold=DEBUG
      log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
      log4j.appender.CONSOLE.layout.ConversionPattern=- %m%n

      # XMLSEC debugding
      log4j.logger.org.apache.xml.security=DEBUG
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    • restart the IDP server
    • you will see messages like in the catalina.out:
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      Signing will use certificate [O=novell, OU=accessManager, CN=test-signing] having serial no [545818454437813568918420645375657259130939441557] <
      /amLogEntry>

      - Create URI "http://www.w3.org/2000/09/xmldsig#rsa-sha1" class "class org.apache.xml.security.algorithms.implementations.SignatureBaseRSA$Signa
      tureRSASHA1"
      - Request for URI http://www.w3.org/2000/09/xmldsig#rsa-sha1
      - Created SignatureDSA using SHA1withRSA
      - Transforms.addTransform(http://www.w3.org/2000/09/xmldsig#enveloped-signature)
      - Create URI "http://www.w3.org/2000/09/xmldsig#enveloped-signature" class "class org.apache.xml.security.transforms.implementations.TransformEn
      velopedSignature"
      - The NodeList is null
      - Transforms.addTransform(http://www.w3.org/2000/09/xmldsig#enveloped-signature)
      - Transforms.addTransform(http://www.w3.org/2001/10/xml-exc-c14n#)
      - Create URI "http://www.w3.org/2001/10/xml-exc-c14n#" class "class org.apache.xml.security.transforms.implementations.TransformC14NExclusive"
      - The NodeList is null
      - Transforms.addTransform(http://www.w3.org/2001/10/xml-exc-c14n#)
      - Request for URI http://www.w3.org/2000/09/xmldsig#sha1
      - setElement("ds:SignatureMethod", "null")
      - Create URI "http://www.w3.org/2000/09/xmldsig#rsa-sha1" class "class org.apache.xml.security.algorithms.implementations.SignatureBaseRSA$Signa
      tureRSASHA1"
      - Request for URI http://www.w3.org/2000/09/xmldsig#rsa-sha1
      - Created SignatureDSA using SHA1withRSA
      - Request for URI http://www.w3.org/2000/09/xmldsig#sha1
      - I was asked to create a ResourceResolver and got 0
      -  extra resolvers to my existing 4 system-wide resolvers
      - check resolvability by class org.apache.xml.security.utils.resolver.ResourceResolver
      - quick fail for empty URIs and local ones
      - check resolvability by class org.apache.xml.security.utils.resolver.ResourceResolver
      - check resolvability by class org.apache.xml.security.utils.resolver.ResourceResolver
      - State I can resolve reference: "#idr3cN4ANgZgrnW3IKN555OXFXp0U"
      - getElementByIdType() Search for ID idr3cN4ANgZgrnW3IKN555OXFXp0U
      - I could find an Element using the simple getElementByIdType method: samlp:Response
      - Try to catch an Element with ID idr3cN4ANgZgrnW3IKN555OXFXp0U and Element was [samlp:Response: null]
      - setElement("ds:Transforms", "null")
      - setElement("ds:Transform", "null")
      - Preform the (0)th http://www.w3.org/2000/09/xmldsig#enveloped-signature transform
      - setElement("ds:Transform", "null")
      <amLogEntry> 2019-07-30T13:31:52Z DEBUG NIDS Application:
      Method: XMLSignable.postSigning
      Thread: https-jsse-nio-192.168.10.100-8443-exec-2
      Signature completed for object! Name: Response </amLogEntry>
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

    • Tomcat / JAVA class loader debugging to review which classes / jars are getting loaded in which order
      add the following entry into the "/opt/novell/nam/idp/conf/tomcat.conf" Note: make sure you disable this feature after testing as this will slow down the system dramatically and create a huge amount of logging entries
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      “JAVA_OPTS="${JAVA_OPTS} -verbose:class"
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      This will dump messages like:
      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      [Opened /opt/novell/nam/idp/endorsed/commons-logging-1.2.jar]
      [Opened /opt/novell/nam/idp/endorsed/gson-2.2.4.jar]
      [Opened /opt/novell/nam/idp/endorsed/httpclient-4.5.jar]
      [Opened /opt/novell/nam/idp/endorsed/log4j-1.2.17.jar]
      [Opened /opt/novell/nam/idp/endorsed/auCoreLib.jar]
      [Opened /opt/novell/nam/idp/endorsed/commons-codec-1.9.jar]
      [Opened /opt/novell/nam/idp/endorsed/httpcore-4.4.1.jar]
      [Opened /opt/netiq/common/jre/lib/rt.jar]
      [Loaded java.lang.Object from /opt/netiq/common/jre/lib/rt.jar]
      [Loaded java.io.Serializable from /opt/netiq/common/jre/lib/rt.jar]
      [Loaded java.lang.Comparable from /opt/netiq/common/jre/lib/rt.jar]
      [Loaded java.lang.CharSequence from /opt/netiq/common/jre/lib/rt.jar]
      ......
      .......

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  • zipdiff tool from http://zipdiff.sourceforge.net/ in order to compare JARs
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    java -jar zipdiff.jar -file1 xmlsec44.jar -file2 xmlsec45.jar -comparetimestamps -comparecrcvalues -outputfile diffs--nam44and45.html
    File 1 = xmlsec181.jar
    File 2 = xmlsec185.jar
    No differences found.

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  • xmlsec1 tool from http://www.aleksey.com installed on a Debian Linux in order to validate SAML Response Message Signature
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    klaus@debian:~/Assertions$ xmlsec1 --verify --trusted-pem IDPRH.pem --id-attr:ID urn:oasis:names:tc:SAML:2.0:protocol:Response nam441-enc-assertion.xml
    OK
    SignedInfo References (ok/all): 1/1
    Manifests References (ok/all): 0/0
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++