32.3.2 Troubleshooting 100101043 and 100101044 Liberty Metadata Load Errors

Identity Server is the identity provider for other Access Manager components. Access Gateways have Embedded Service Providers. When a device is imported into Administration Console and an Identity Server configuration is selected for them, a trusted relationship is established with Identity Server by using test certificates. When you change these certificates or change from using HTTP to HTTPS, you need to ensure that the trusted relationship is reestablished. Metadata is used for establishing trusted relationships.

The metadata exchanged between service providers and identity providers contains public key certificates, key descriptors for message signing, a URL for the SSO service, a URL for the SLO (single logout) service, and so on. With Access Manager, this metadata is accessible on both Identity Server and the Embedded Service Provider of the device. Errors are generated when either the identity provider could not load the service provider’s metadata (100101043), or the service provider could not load the metadata of the identity provider (100101044).

If users are receiving either of these errors when they attempt to log in, verify the following:

If these steps do not solve your problem, try the following:

Metadata

If you change the base URL of the Identity Server, all service providers, including Embedded Service Providers, need to be updated so that they use the new metadata:

Embedded Service Provider Metadata

If you change the base URL of the Identity Provider, all Access Manager devices that have an Embedded Service Provider need to be updated so that new metadata is imported. To force a re-import of the metadata, you need to configure the device so it does not’ have a trusted relationship with Identity Server, update the device, reconfigure the device for a trusted relationship, then update the device. The following steps explain how to force Access Gateway to re-import the metadata of Identity Server.

  1. In Administration Console Dashboard, click Devices > Access Gateways > Edit > Reverse Proxies/Authentication.

  2. Select None for the Identity Server Cluster option, click OK twice, then update Access Gateway.

  3. Click Edit > Reverse Proxies/Authentication.

  4. Select an Identity Server configuration for the Identity Server Cluster option, click OK twice, then update Access Gateway.

Service Provider Metadata

If you have set up federation with another provider over the Liberty, SAML 1.1, SAML 2.0, or WS Federation protocol and you change the base URL of Identity Server, you need to update the provider with the new metadata to reestablish the trusted relationship. If the provider is another Identity Server, follow the procedure below to update the metadata; otherwise, follow the provider’s procedures.

  1. In Administration Console Dashboard, click Devices > Identity Servers > Edit > [Protocol] > [Provider] > Metadata.

  2. Click Reimport.

  3. Follow the steps in the wizard.

    For more information, see Section 2.7.7, Managing Metadata.

DNS Name Resolution

When the service provider tries to access the metadata on the identity provider, it sends the request to the hostname defined in the base URL configuration of Identity Server. The base URL in Identity Server configuration is used to build all the metadata end points.

To view the metadata of Identity Server with a DNS name of idpcluster.lab.novell.com, enter the following URL:

https://idpcluster.lab.novell.com:8443/nidp/idff/metadata

Scan through the document and notice the multiple references to https://idpcluster.lab.novell.com/... You should see lines similar to the following:

<md:SoapEndpoint>
   https://idpcluster.lab.novell.com:8443/nidp/idff/soap
</md:SoapEndpoint>

<md:SingleLogoutServiceURL>
   https://idpcluster.lab.novell.com:8443/nidp/idff/slo
</md:SingleLogoutServiceURL>

<md:SingleLogoutServiceReturnURL>
   https://idpcluster.lab.novell.com:8443/nidp/idff/slo_return
</md:SingleLogoutServiceReturnURL>

The Embedded Service Provider of Access Gateway must be able to resolve the idpcluster.lab.novell.com hostname of Identity Server. To test that it is resolvable, send a ping command with the hostname of Identity Server. For example, from Access Gateway:

ping idpcluster.lab.novell.com

The same is true for Identity Server. It must be able to resolve the hostname of Access Gateway. To discover the URL for Access Gateway metadata:

  1. In Administration Console Dashboard, click Devices > Access Gateways > Edit > Reverse Proxy/Authentication.

  2. View the Embedded Service Provider section.

    The URL of the metadata is displayed in this section.

To view the metadata, enter the displayed URL. Scan through the document and notice the multiple references to the hostname of Access Gateway.

You should see lines similar to the following. In these lines, the hostname is ag1.provo.novell.com.

<md:SoapEndpoint>
   http://ag1.provo.novell.com:80/nesp/idff/spsoap
</md:SoapEndpoint>

<md:SingleLogoutServiceURL>
   http://ag1.provo.novell.com:80/nesp/idff/spslo
</md:SingleLogoutServiceURL>

<md:SingleLogoutServiceReturnURL>
   http://ag1.provo.novell.com:80/nesp/idff/spslo_return
</md:SingleLogoutServiceReturnURL>

To test that Identity Server can resolve the hostname of Access Gateway, send a ping command with the hostname of Access Gateway. For example, from Identity Server:

ping ag1.provo.novell.com

To view sample log entries that are logged when a DNS name cannot be resolved, see The Embedded Service Provider Cannot Resolve the Base URL of Identity Server.

Certificate Names

Ensure that the certificates for Identity Server and the Embedded Service Provider match the hostnames defined in the metadata URL (see DNS Name Resolution).

When Identity Server and Access Gateway are enabled for HTTPS, all communication to these devices requires that the devices send back a server certificate. Not only must the certificate be assigned to the appropriate device, but the subject name of the device certificate must match the hostname of the device it is assigned to.

To verify the certificate name of Identity Server certificate:

  1. In Administration Console Dashboard, click Devices > Identity Servers > Edit.

  2. Click the SSL Certificate icon.

    The SSL Connector keystore is displayed.

  3. Verify that the subject name of the certificate matches the DNS name of Identity Server.

    • If the names match, a certificate name mismatch is not causing your problem.

    • If the names do not match, you need to either create a certificate that matches or import one that matches. For information about how to create a certificate for Identity Server, see Section 19.0, Enabling SSL Communication.

To verify the certificate name of Access Gateway certificate:

  1. In Administration Console Dashboard, click Devices > Access Gateways > Edit > [Name of Reverse Proxy].

  2. Read the alias name of the server certificate, then click the Server Certificate icon.

  3. Verify that the Subject name of the server certificate matches the published DNS name of the proxy service of Access Gateway.

    • If the names match, a certificate name mismatch is not causing your problem.

    • If the names do not match, you need to either create a certificate that matches or import one that matches. For information about how to create an Access Gateways certificate, see Configuring Access Gateway for SSL.

To view sample log entries that are logged to the catalina.out file when the certificate has an invalid name, see The Server Certificate Has an Invalid Subject Name.

Certificates in the Required Trust Stores

Ensure that the issuers of Identity Server and Embedded Service Provider certificates are added to the appropriate trusted root containers.

When the server certificates are sent from the identity provider to the service provider client, and from the service provider to the identity provider client, the client needs to be able to validate the certificates. Part of the validation process is to confirm that the server certificate has been signed by a trusted source. By default, well known external trusted certificates are bundled with Access Manager. You can view this list here: Administration Console > Security > Certificates > External Trusted Roots. If the issuer of server certificate is not present in the External Trusted Root list, the import the issuers of the server certificate (intermediate and trusted roots) into the correct trusted root stores:

  • The intermediate and trusted roots of the Embedded Service Provider certificate must be imported into the NIDP Trust Store.

  • The intermediate and trusted roots of Identity Server certificate must be imported into the ESP Trust Store.

For more information, see Section 15.5, Importing a Signed Certificate.

If you use certificates generated by Administration Console CA, the trusted root certificate is the same for Identity Server and the Embedded Service Provider. If you are using external certificates, the trusted root certificate might not be the same, and there might be intermediate certificates that need to be imported.

To verify the trusted root certificates:

  1. In Administration Console Dashboard, click Security > Certificates.

  2. Determine the issuer of Identity Server certificate and the Embedded Service Provider certificate:

    1. Click the name of Identity Server certificate, note the name of the Issuer, then click Close.

    2. Click the name of the Embedded Service Provider certificate of Access Gateway, note the name of the Issuer, then click Close.

    3. (Conditional) If you do not know the names of these certificates, see Certificate Names.

  3. To verify the trusted root for Identity Server, click Devices> Identity Servers > Edit > Security > NIDP Trust Store.

  4. In the Trusted Roots section, scan for a certificate subject that matches the issuer of the Embedded Service Provider certificate, then click its name.

    • If the Issuer has the same name as the Subject name, then this certificate is the root certificate.

    • If the Issuer has a different name than the Subject name, the certificate is an intermediate certificate in the chain. Click Close, and ensure that another certificate in the trust store is the root certificate. If it isn’t there, you need to import it and any other intermediate certificates between the one you have and the root certificate.

  5. To verify the trusted root for the Embedded Service Provider, click Devices > Access Gateways > Edit >Service Provider Certificates> Trusted Roots.

  6. In the Trusted Roots section, scan for a certificate subject that matches the issuer of Identity Server certificate, then click its name.

    • If the Issuer has the same name as the Subject name, then this certificate is the root certificate.

    • If the Issuer has a different name than the Subject name, the certificate is an intermediate certificate in the chain. Click Close, and ensure that another certificate in the trust store is the root certificate. If it isn’t there, you need to import it and any other intermediate certificates between the one you have and the root certificate.

  7. (Optional) If you have clustered your Identity Servers and Access Gateways and you are concerned that not all members of the cluster are using the correct trusted root certificates, you can re-push the certificates to the cluster members.

    1. Click Troubleshooting > Certificates.

    2. Select the Trust Store of your Identity Servers and Access Gateways, then click Re-push certificates.

    3. Update the Identity Severs and Access Gateways.

    4. Check the command status of each device to ensure that the certificate was pushed to the device. From Identity Servers page or Access Gateways page, click the Commands link.

To view sample log entries that are logged to the catalina.out file when a trusted root certificate is missing, see Trusted Roots Are Not Imported into the Appropriate Trusted Root Containers.

Enabling Debug Logging

You can enable Identity Server logging to dump more verbose Liberty information to the catalina.out file on both Identity Server and the Embedded Service Provider of Access Gateway.

  1. In Administration Console Dashboard, click Devices > Identity Servers > Edit > Auditing and Logging.

  2. Select Enabled for File Logging and Echo to Console.

  3. In the Component File Logger Levels section, set Application and Liberty to a debug level.

  4. Click OK, update Identity Server, then update Access Gateway.

  5. After enabling and applying the changes, duplicate the issue to add specific details to the log file for the issue.

    If the error is the 100101044 error, look at the log file on the Embedded Service Provider for the error code

    If the error is the 100101043 error, look at the log file on Identity Server for the error code.

    On Linux, look at the catalina.out file, and on Windows, look at the stdout.log file.

  6. (Conditional) To view the log files from Administration Console, click Auditing > General Logging, then select the file and download it.

  7. (Conditional) To view the log files on the device, change to the log directory.

    • On Linux, change to the /var/opt/novell/nam/logs/idp directory.

    • On Windows Server, change to the /Program Files/Novell/Tomcat/logs directory.

Below are a few typical entries illustrating the most common problems. They are from the catalina.out file of the Embedded Service Provider:

The Embedded Service Provider Cannot Resolve the Base URL of Identity Server

When the Embedded Service Provider cannot resolve the DNS name of Identity Server, the metadata cannot be loaded and a hostname error is logged. In the following entries, the Embedded Service Provider cannot resolve the idpcluster.lab.novell.com name of Identity Server.

<amLogEntry> 2009-08-06T16:24:56Z INFO NIDS Application: AM#500105024:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: ESP is requesting metadata from IDP https://
idpcluster.lab.novell.com/nidp/idff/metadata </amLogEntry>

<amLogEntry> 2009-08-06T16:24:56Z SEVERE NIDS IDFF: AM#100106001:
AMDEVICEID#esp-09C720981EEE4EB4: Unable to load metadata for Embedded
Service Provider: https://idpcluster.lab.novell.com/nidp/idff/
metadata, error: AM#300101046: AMDEVICEID#esp-09C720981EEE4EB4::   Attempted to connect to a url with an unresolvable host name 
</amLogEntry>

<amLogEntry> 2009-08-06T16:24:56Z INFO NIDS Application: AM#500105039:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: Error on session id 2CA1168DF7343A42C7879E707C51A03C,
error 100101044-esp-09C720981EEE4EB4, Unable to authenticate.
AM#100101044: AMDEVICEID#esp-09C720981EEE4EB4:: Embedded Provider
failed to load Identity Provider metadata </amLogEntry>

Trusted Roots Are Not Imported into the Appropriate Trusted Root Containers

When the trusted roots are not imported into the appropriate trusted root containers, a certificate exception is thrown and an untrusted certificate message is logged. In the following log entries, the Embedded Service Provider is requesting metadata from Identity Server, but the Embedded Service Provider does not trust Identity Server certificate because the trusted root of the issuer of Identity Server certificate is not in the Embedded Service Provider’s trusted root container.

<amLogEntry> 2009-08-05T16:07:53Z INFO NIDS Application: AM#500105024:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: ESP is requesting metadata from IDP https://idpcluster.lab.novell.com/nidp/idff/metadata </amLogEntry>

<amLogEntry> 2009-08-05T16:07:53Z SEVERE NIDS IDFF: AM#100106001: AMDEVICEID#esp-09C720981EEE4EB4: Unable to load metadata for Embedded ServiceProvider: https://idpcluster.lab.novell.com/nidp/idff/metadata, error: java.security.cert.CertificateException: Untrusted Certificate-  chain </amLogEntry>

<amLogEntry> 2009-08-05T16:07:53Z INFO NIDS Application: AM#500105039:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: Error on session id D983B08C28D35221D139D33E5324F98F, error 100101044-esp-09C720981EEE4EB4, Unable to authenticate. AM#100101044: AMDEVICEID#esp-09C720981EEE4EB4:: Embedded Provider failed to load Identity Provider metadata </amLogEntry>

The Server Certificate Has an Invalid Subject Name

When the certificate has an invalid subject name, the handshake fails. In the log entries below, the Embedded Service Provider is requesting metadata from Identity Server. The server certificate name does not match, so the Embedded Service Provider is unable to authenticate and get the metadata necessary to establish the trusted relationship.

<amLogEntry> 2009-07-05T16:07:53Z INFO NIDS Application: AM#500105024:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: ESP is requesting metadata from IDP 
https://idpcluster.lab.novell.com/nidp/idff/metadata </amLogEntry>

<amLogEntry> 2009-07-05T16:07:53Z SEVERE NIDS IDFF: AM#100106001: AMDEVICEID#esp-09C720981EEE4EB4: Unable to load metadata for Embedded Service Provider: https://idpcluster.lab.novell.com/nidp/idff/metadata, error: Received fatal alert: handshake_failure </amLogEntry>

<amLogEntry> 2009-07-05T16:07:53Z INFO NIDS Application: AM#500105039:
AMDEVICEID#esp-09C720981EEE4EB4: AMAUTHID#YfdEmqCT2ZutwybD1eYSpfph8g5a5aMl6MGryq1hIqc=: Error on session id D983B08C28D35221D139D33E5324F98F, error 100101044-esp-09C720981EEE 4EB4, Unable to authenticate. AM#100101044: AMDEVICEID#esp-09C720981EEE4EB4: : Embedded Provider failed to load Identity Provider
metadata </amLogEntry>

Testing Whether the Provider Can Access the Metadata

To test whether the metadata is available for download, enter the metadata URL of the identity provider and service provider. If the DNS name of the identity provider is idpcluster.lab.novell.com, open a browser at Identity Server and enter the following URL:

https://idpcluster.lab.novell.com:8443/nidp/idff/metadata

Open a browser on Access Gateway Service, then enter the same URL.

Because Access Gateway Appliance does not have a graphical interface, you need to use the curl command to test whether Access Gateway Appliance can access the metadata of Identity Server. If the DNS name of the identity provider is idpcluster.lab.novell.com, enter the following command from Access Gateway machine:

curl -k https://idpcluster.lab.novell.com:8443/nidp/idff/metadata

To test whether Identity Server can access the metadata URL of Access Gateway, open a browser on Identity Server machine. If the published DNS name of service provider is www.aleris.net, enter the following URL:

https://www.aleris.net/nesp/idff/metadata

Manually Creating Any Auto-Generated Certificates

Occasionally, there are issues where the subject name was auto-generated and the entire configuration appears to be correct, but the 100101044/100101043 error is still reported. Delete the auto-generated certificate and manually re-create the server certificate, making sure that it is added to the relevant devices and stores.