Getting Started Building a SOAP Driver for IDM – Part 1

geoffc

By: geoffc

August 25, 2010 3:05 pm

Reads: 543

Comments:0

Rating:4.5

This entry is part 1 of 9 in the series Getting Started Building a SOAP Driver for IDM

Novell Identity Manager is a great product that has a bunch of different drivers that connect to a variety of different systems.

Some are really specific drivers, like the Lotus Domino (Notes) driver. It only works with Lotus Domino servers. Then there are some that are most generic like the LDAP or JDBC drivers, which can work with most LDAP or JDBC instances. This class can be both easier and harder, since there are so many possibilities, but also more basic configurations that help out.

The SOAP driver is one that falls into a different category. In fact I think it is the only one in this category. It provides more of a transport, than anything else. Basically this is because SOAP is a transport protocol, not really an application per se.

It happens to ship with some configurations, for DSML and SPML (As of Identity Manager 3.6.1 SPML V1.0 support is available, support for SPML 2.0 is coming with Identity Manager 4 when it ships), two common approaches to implementing a complaint SOAP application. But the reality is that most SOAP implementations are unique and one of a kind. Or at least sufficiently different to not work with the standards.

Recently I had the entertaining task of building a SOAP driver to Salesforce.com. Salesforce.com, henceforth known as SFDC, has an interesting SOAP API that is quite powerful.

I thought it might be interesting to work through some of the steps I needed to take to get this working properly.

It was both harder and easier than I expected. It was easier, because once I got the basics done, stuff just started to work, except for some minor details. It was harder because the devil is literally in the details. The details will totally suck the time out of a project.

One of the things about the SOAP driver is that it is considered one of those drivers that really needs to be done mostly in XSLT. Well perhaps this was true in the past, but at some point, Novell added a very nice configuration option in the driver that basically wraps the SOAP document (Usually a <soapenv:Envelope> or <soap-env:Envelope> parent node inside an <nds> and <input> or <output> node.

This has the benefit of allowing DirXML Script policy to work on it. If you have ever done a Delimited Text driver, it is this issue that gets you locked into XSLT. DirXML Script cannot touch XML unless it is somewhat XDS complaint, and that means, starts with an <nds> node for the most part.

If the same option to wrap the <record> node with an <nds> node existed in the Delimited text driver, you could probably do most everything you needed without using XSLT. (Yes, I have an enhancement request at the http://www.novell.com/rms web site. So far I have 2 things I requested reported as implemented. The rest are in a variety of states. I have about 20 some odd submitted.).

Once you realize that you can do almost everything in Policy it makes life a lot easier.

The very first thing you need to do is get hold of the WSDL (Web Services Definition Language) which is the way that web services describe themselves. Then get a copy of soapUI (http://www.soapui.org/ ) which is a great tool for playing around with SOAP. It looks somewhat Eclipse based, but stripped down.

You can open a WSDL, explore the functions, see what the input XML for the SOAP API call should look like, and then try to connect to an endpoint to test any ideas that you have.

Then see if there are any other custom tools that the vendor supplies for their interface. In the case of SFDC, there is an excellent tool called Apex Explorer (http://wiki.developerforce.com/index.php/Apex_Explorer ) that is something like DBVis (http://www.dbvis.com/) which is a database browsing tool. In this case Apex Explorer is a visual tool to explore the SFDC database.

This is great for two reasons. Firstly you can explore and understand the object classes and attributes in SFDC. You can see the syntax and the possible values (in the case of the picklist attributes). Secondly you can build SOQL (the Salesforce Object Query Language, that SFDC uses in queries) statements, test them and see what the actual underlying data looks like.

My experience was that the team that built the SFDC application was not really an application development group, and that the underlying schema and data layout are different than how I might have done it. But once you can see the data, you can get a better understanding of what is there, and how it is being used.

One interesting limitation to realize is that SFDC does not really support multi valued attributes. The closest you can get to a multi valued attribute is a multi-picklist, which is where you get a picklist box, and can select 0 to many values. The data is actually stored, semi colon separated in the underlying data tables, which is good to know.

They do support reference attributes which is similar to eDirectory’s Distinguished Name syntax, but unlike eDirectory, these cannot be multi valued, so if there are references from one object to many, it is really recorded as many objects pointing at one object.

Queries in both directions work, so this is more of a quibble than a real issue.

Now that you have a tool to generate SOAP documents (soapUI) to better understand what the transactions should look like and a tool to explore the underlying data (Apex Explorer) you are ready to go.

So what transaction to do you need first? Well how do you connect? There are a number of possible authentication approaches used in SOAP web services. The SOAP driver from Novell by default uses the header based authentication. That is, your HTTP connection includes the authentication information. (In soapUI, this would be the Auth tab on any particular API function you have open). This is actually how you would connect to the User Applications SOAP end points if you wanted to say, make a SOAP call to terminate a running workflow. (But that is the story for another article series).

In the case of SFDC, they use session based authentication, which in some ways is easier. You have to use the <logon> function, to connect with a username and password, and get an endpoint URL, and sessionId to work with for the following transactions.

You can model this in soapUI and see how this works. In the case of SFDC, you need to learn that a password is not always just a password. It turns out to further secure API access to SFDC, they require a username (Usually your email address), a password, and a Security Token. Every time you do a password change a new Security token is issued and non-web based access (Apex Explorer included) requires the password value to be the password followed immediately by the Security Token value. Making for a MUCH longer password string.

My initial approach was sort of a cheat. I was not sure I would be able to do everything in Policy when I began, but I really wanted to avoid the XSLT approach. So I started using a trick I learned with the JDBC driver (Calling Stored Procedures with the IDM JDBC Driver) which builds an XML document in a local variable (I start with a GCV, and some fields that need to be replaced) XML Parse it into a nodeset variable, and then make a Java call to destCommandProcessor with the XML document from the Subscriber channel (Thus the destination is the connected system). I have used this with success in the JDBC driver to call stored procedures, as well as in the SAP HR driver to build the RELATIONSHIP queries outside of XSLT. (The issue in the SAP HR driver is that the Query doc for this very specific query is not one that the standard Query token can generate. It needs an extra node. So usually the approach for custom XDS docs is to use XSLT. But this approach works well there.

The GCV I used looked something like this:

		<definition display-name="Storage for the XML doc used to login to SFDC     (sfdc-login-doc)" multi-line="true" name="sfdc-login-doc" type="string">
			<description>The login to SFDC needs to send an XML doc, and by storing in a GCV, we can replace the Username/Password, when we are ready to use it, and then XML Parse it, and use srcCommandProcessor to send it to the shim.</description>
			<value><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:enterprise.soap.sforce.com">
   <soapenv:Header>
   </soapenv:Header>
   <soapenv:Body>
      <urn:login>
         <urn:username>XXUser</urn:username>
         <urn:password>YYPwd</urn:password>
      </urn:login>
   </soapenv:Body>
</soapenv:Envelope></value>
		</definition>
		
		

Note: I do not know how to represent that here, but the less than/greater than signs for the XML inside the GCV value nodes, actually needs to be ampersand lt semi colon, i.e. escaped for use inside XML.

You can see that the <urn:username> and <urn:password> nodes I had values that are easily searched and replaced.

Anyway, I used that to send the login XML document that SFDC expected. Now the problem I had was, when to send it? Well I had planned on using the driver heartbeat which shows up in the Publisher channel every minute (at its most frequent value, you can make it higher, but not lower than 1 minute).

The problem is that on the publisher channel, sending a srcCommandProcessor command gets a response document that informs you that the event is queued on the Subscriber channel, and the driver moves on. Doh! No, I need to wait for a response.

In hindsight, I wonder if Peter L’s approach in this article ( Sending a XDS Message from one driver to another)
would help there. Will have to think about that one.

Anyway my solution was to use the heartbeat, cast the event out there into the queue, and when it eventually returns, process the <loginResponse> document, in an Input transform rule, and use it to set a pair of driver scoped variables, one for the endpoint URL, and the other for the sessionId.

In order to do this, I had to use XPATH to detect the <loginResponse> document, (XPATH of self::soapenv:Envelope/soapenv:Body/urn:loginResponse ) and pick out the values of the two returned values of interest. This is straightforward XPATH, just with namespaces thrown in for fun, so remember to use the right namespace prefixes and to add the namespace declarations to all your policies involving them. You can add them by hand in the <policy> node of the XML inside your policy object, or you can use the Namespace editor in Designer, (It is in the upper right hand corner next to the Simulator button.)

What I also did was in the Input transform, I would convert any response from SFDC over SOAP into <instance> documents, with attribute names that made sense. So in the case of loginResponse, I decided to only send back an <instance> document, (making it look like a response to a query) with two attributes, sessionId, and serverUrl with the values returned.

This way the engine can handle everything easily. <instance> docs are meant to be the way to get data back from queries, so converting most responses to them, makes handling it very easy.

Now to convert the <loginResponse> document to an <instance> document I could have done in XSLT and built up the document the old fashioned way, but instead I decided to do it all in Policy. To me this was an easier approach. I concede it is a little harder to read, but it does work well.

Basically, you use the Append XML element token to add element nodes (first an <instance> node, then an <attr> node under it) to the event document, Set XML attribute to add a class-name XML attribute, like <instance class-name=”loginResponse”> or <attr attr-name=”sessionId”> append a <value> node under the <attr> node, and then use the Append XML Text token to add some text to the <value> node, which you get from using XPATH to copy out the value of interest.

A couple of tips. You are going to use a lot of the last() function. This is important since in XPATH if you set an XML Attribute, lets say attr-name and you specify to set it, to the XPATH of ../instance/attr and you have three <attr> nodes in the document, all three are going to get the XML attribute added. You need to take great care and be specific in what you want.

As long as you work in order, you can use the predicate ../instance[last()]/attr[last()]/value[last()] to be sure you are sticking your data in the correct place.

In this space, the Simulator in Designer is your greatest asset. Copy the XML from the returned SOAP document, and paste it into the Comment for your rule. Then when you need it, copy it out, and paste it into Simulator. Then you can run through quickly and see if your approach worked.

Then as a last step, use Strip by XPATH to get rid of the SOAP XML and leave behind just a nice and valid XDS document for the engine to work with.

The next thing you need to do is to get a Query XDS converted to a SOAP query, and then handle its response. Once these two things (Login and Query) are working, most everything else is a detail.

Stay tuned for the next article in this series where I explain one approach to handling query events.

As an example, here is how I handled the loginResponse doc as a full rule, pasted in:

<rule>
	<description>[icap] Handle loginResponse documents, with soapenv: header left behind</description>
	<comment xml:space="preserve">Lets see if we can do this in Policy instead of XSLT.  Well yes, yes we can!

Because the SOAP doc comes back with a namespace definition, this causes us pain, so the test is for self::urn:loginResponse.  This works because we use the urn: namespace to ask for it.

Next we build the instance doc, with the two values we care about, serverUrl and sessionId. Step by step, element by element we build it.

Note:
Append XML Element adds a <node>
Append XML text sticks a text string between <node>text</node>
Set XML attribute sets the class-name='User' in the <node class-name='User'>

Rinse and repeat for each attr.

Set the driver scoped variables that we need to prove we are logged in.

Finally remove the SOAP doc from the XDS doc with a strip XPATH.

</comment> <comment name="author" xml:space="preserve">Geoffrey Carman</comment> <comment name="version" xml:space="preserve">2</comment> <comment name="lastchanged" xml:space="preserve">Apr 27, 2010</comment> <conditions> <or> <if-xpath op="true">self::soapenv:Envelope/soapenv:Body/urn:loginResponse</if-xpath> </or> </conditions> <actions> <do-append-xml-element expression=".." name="instance"/> <do-set-xml-attr expression="../instance" name="class-name"> <arg-string> <token-text xml:space="preserve">login</token-text> </arg-string> </do-set-xml-attr> <do-set-xml-attr expression="../instance" name="src-dn"> <arg-string> <token-text xml:space="preserve">sfdc</token-text> </arg-string> </do-set-xml-attr> <do-append-xml-element expression="../instance" name="attr"/> <do-set-xml-attr expression="../instance/attr[last()]" name="attr-name"> <arg-string> <token-text xml:space="preserve">sessionId</token-text> </arg-string> </do-set-xml-attr> <do-append-xml-element expression="../instance/attr[@attr-name='sessionId']" name="value"/> <do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="sessionId"]/value'> <arg-string> <token-xpath expression="*//*[local-name()='sessionId']/text()"/> </arg-string> </do-append-xml-text> <do-append-xml-element expression="../instance" name="attr"/> <do-set-xml-attr expression="../instance/attr[last()]" name="attr-name"> <arg-string> <token-text xml:space="preserve">serverUrl</token-text> </arg-string> </do-set-xml-attr> <do-append-xml-element expression="../instance/attr[@attr-name='serverUrl']" name="value"/> <do-append-xml-text expression='../instance[@class-name="login"]/attr[@attr-name="serverUrl"]/value'> <arg-string> <token-xpath expression="*//*[local-name()='serverUrl']/text()"/> </arg-string> </do-append-xml-text> <do-set-local-variable name="SfdcTargetURL" scope="driver"> <arg-string> <token-xpath expression="*//*[local-name()='serverUrl']/text()"/> </arg-string> </do-set-local-variable> <do-set-local-variable name="SessionID" scope="driver"> <arg-string> <token-xpath expression="*//*[local-name()='sessionId']/text()"/> </arg-string> </do-set-local-variable> <do-strip-xpath expression="self::soapenv:Envelope"/> <do-break/> </actions> </rule>
VN:D [1.9.22_1171]
Rating: 4.5/5 (2 votes cast)
Getting Started Building a SOAP Driver for IDM - Part 1, 4.5 out of 5 based on 2 ratings
Series NavigationGetting Started Building a SOAP Driver for IDM – Part 2 >>

Tags: ,
Categories: Identity Manager, Technical Solutions

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment.  It just worked for at least one person, and perhaps it will be useful for you too.  Be sure to test in a non-production environment.

Comment