This entry is part 2 of 3 in the series Delving into and beyond the current-op

 
The first article in this series covered the basics aspects related to the current operation, including an overview of how the engine actually handles the current operation and the key technical details related to the current operation.

This second article will outline general best practices related to working directly with nodes other than the current-op within the “current operation”. It will also outline some useful toolbox rules and expressions that can be used to access and manipulate the entire XDS document.

Finally, this article will specifically address the following question as raised in the first article:

  • Where are all the other operations in the XDS document, can they be accessed and even changed?

What this article will not address is direct (out of band) operations, as a subsequent article will address these.

Whilst the engine processes a policy and applies the rules contained in that policy, the source XDS document is visualised best as two components (each of which can be further split into logical sub-components).

  1. The “current operation” node set
    1. The current-op node
    2. The current-op variable
    3. Sibling elements/nodes added by prior rules (including pending direct operations)
  2. Remaining nodes in the XDS document
    1. Nodes already processed by this policy
    2. Placeholder/marker for the current-operation node set (visible only to the engine).
    3. Nodes awaiting processing.

Once all relevant rules in the policy are applied, the engine merges the two main components above (XDS document and “current operation” node set). The engine repeats the entire process using the next node awaiting processing (if any remain).

current-op node

At any point in time, there can only ever be a maximum of one current-op node (At least while the driver is actively processing events!)

By definition, the current-op node is a single node containing a valid operation that the engine uses as the default destination when applying rules in the policy. Rules can of course specify that they should operate on another object than the one represented in the current operation.

Additionally, the current-op node must be a node already present in the “current operation” node set. This makes changing the current-op always a two rule process.

  1. Append a node to the “current operation” node set.
  2. Set the current-op nodeset variable to point at this new node

current-op variable

The current-op variable exists mostly as a way to change which node set within the current operation is the actual current-op node.

It is a node set and as such can contain multiple element nodes. When this variable (by policy) is set to a node or node set, then first node in this variable becomes the current-op node (however this node must also be a sibling element node of the “current operation” node set). The naming is arguably confusing, but once you understand the distinction, it does make sense.

This variable is not the same as the “current operation” node set. If they were, and the requirement that the first node of the current-op variable was always the current-op node, then there would be no way to add sibling nodes prior to the current-op node!

Sibling Elements (nodes)

During the application of a policy against the current-op node, rules often add sibling elements.  These siblings can appear either prior or after the current-op node, (they can be regular operations or pending direct write operations).

The combination of the sibling element nodes and the current-op node make up the “current operation” node set.

If you veto, or strip via XPath the current operation, and there are no sibling elements, then the current-op node would point to an empty node set.

Variable scope

One thing that I wondered about, but never actually had around to testing was

“When you set a local variable to scope=’policy’, what does that actually mean?”

As mentioned in the first article, there rules within a policy apply to each operation in the XDS document.

The scope of a local variable is limited to just a single operation within a policy.

This makes sense, as you would end up with odd bugs if a local variable for one operation still hung around for the next operation and then reused accidentally by the next policy without first being reinitialised.

Therefore, the hierarchy of variable scope is as follows:

  1. Policy (per operation, persists only until the policy has been applied to that specific operation, read-write)
  2. Driver (valid for all subsequent operations, persists until the next driver restart, read-write)
  3. Global (otherwise known as Global Configuration Values) persists until the next driver restart, read-only)

In addition, there is operation data that is per operation, but persists across policies. The downside of that is it can really bloat the trace.

Within the existing framework, one has no choice but to abuse driver scope variables whenever one needs a variable that persists longer than a specific policy applied to a specific operation). Even if the persistence is limited to just within a specific policy.

It would be nice to have more levels of variables. I would particularly like to see a variable that is has a per channel scope (they do run as independent threads after all) at the very least. At least one of the NetIQ provided drivers (Managed Systems Gateway) actually has code to detect which channel currently has a “lock” on a driver scoped global variable. This is somewhat clumsy, as the other channel “sleeps” for a period of time (1 second if I recall correctly) and tries to get the lock again. Therefore, blocking an entire channel during this period (as each channel is single-threaded).

Best Practices (when working beyond the current-op)

  • Reserve the techniques explained in this series for only when you cannot find a way to solve your problem via the standard engine functionality.
  • When changing the XDS document, scope your rules so each rule only runs once (per XDS document).
  • Write your code generically enough that it can handle processing either an input or output XDS document.
  • When testing or manipulating the XDS document directly, test carefully the scenario where the XDS document contains just a single node.
  • Remember to strip the current operation if you have explicitly copied it back into the XDS document.
  • Try to avoid overuse of the do-append-xml-element, do-append-xml-text and do-set-xml-attr. Instead, ask yourself, how can I do this with the existing object/attribute tokens? This can often make for more legible and more resilient code. It also allows the engine to be smarter with caching and optimizations.
  • Add a comment or trace to indicate when switching to another current-op. This avoids confusion.

Toolbox ​​XPath expressions

Select all nodes from within thecurrent operation”

This will select all nodes within the “current operation”, not just the node actually considered the current-op!

Depending on one’s requirements, this might often be further refined to for example, select only nodes that matched a specific operation type, say add operations.

../*

Select just the current-op node from within the “current operation”

The single node within the “current operation” considered the current-op.

self::*

This can also be represented using the $current-op variable, but technically the current-op variable can contain more than one node, which makes the resultant XPath slightly longer.

$current-op[1]

Select all valid operation nodes in the current document.

All operation nodes, within the “current operation”.

  1. Remaining operation nodes from XDS document (regardless of direction)
  2. The current-op node and its siblings (if any exist) from the operation the current operation.
/nds/input/* | /nds/output/* | ../*

This is a key expression when you choose to work outside of the current operation node set.

Count total number of valid operation nodes in the current document.

Useful if you need to log the number of operations and possibly email or report on this.

Alternatively, one may have a requirement to compare against the number of operations that actually resulted in a change to the destination system.

count(/nds/input/* | /nds/output/* | ../*)

Determine which direction the current XDS document operates.

The result will be either “input” or “output”. I strongly suggest that you set this as a local variable and include this as the first rule in policies as it really comes in handy.

name(/nds/input | /nds/output)

Select the actual root of the current XDS document (irrelevant of direction).

This requires that you first determine the direction and set that as a local variable. (See the first toolbox rule) Without this trick (which should be used in every toolbox rule that manipulates the XDS document directly), most code would be much longer. This is because two copies of the logic would be required (one to refer to the input node and the other to refer to the output node).

In addition, this forces one to write code that is more generic rather than assuming only an “input” direction for example.

/nds/*[name()= $inout]

Moving operations between XDS and “current operation”

If one is at all familiar with the standard clone and strip XPath tokens, then this is quite straightforward.
NOTE: Be sure to always pair a clone with a strip, otherwise you risk ending up with duplicate copies of the nodes.

Move only the current operation from the “current operation” node set the end of the XDS document

Default is to place this at the end of the existing operations.

<do-clone-xpath dest-expression="/nds/*[name()= $inout]" src-expression="self::*"/>
<do-strip-xpath expression="self::*"/>

Specify a desired placement via a before expression: (for example copy prior to the current first operation)

<do-clone-xpath before="*[1]" dest-expression="/nds/*[name()= $inout]" src-expression="self::*"/>
<do-strip-xpath expression="self::*"/>

An alternative representation using the current-op variable.

Which (though longer) appears to be functionally equivalent and potentially more legible.

 <do-clone-xpath dest-expression="/nds/*[name()= $inout]" src-expression="$current-op[1]"/>
 <do-strip-xpath expression="$current-op[1]"/>

Move all nodes in the “current operation” node set to the XDS document

<do-clone-xpath dest-expression="/nds/*[name()= $inout]" src-expression="../*"/>
<do-strip-xpath expression="../*"/>

Test if the source XDS document contains only one operation.

In this scenario, the engine copies the single node to the “current operation” node set and the XDS document is empty. One should always account for this scenario, as many XPath expressions require adjustment so that they work correctly when the XDS document is empty.

Note that when the source XDS document contains no valid operations, the engine does not even attempt to apply rules so there is no way to check for that scenario via DirXML Script.

<if-xpath op="true">count (/nds/*[name()= $inout]/*) = 0</if-xpath>

Toolbox Rules

Determine if input or output and set as a local variable.

The variable will be contain either “input” or “output”.

Make sure to add this toolbox rule prior to any other rule that manipulates the XDS document directly.

<rule>
    <description>Determine if input or output</description>
    <conditions/>
    <actions>
        <do-set-local-variable name="inout" scope="policy">
            <arg-string>
                <token-xpath expression="name(/nds/input | /nds/output)"/>
            </arg-string>
        </do-set-local-variable>
    </actions>
</rule>

Bookend (Add START/END nodes to XDS document) – DirXML Script

Bookend any valid operation nodes in the current XDS document with fake “BEGIN/END” nodes. This rule requires that you place the “Determine if input or output” in the same policy and schedule it prior to this rule.

This addresses a common request in the NetIQ Identity Manager Support Forums for those using the delimited text driver. It only works when all operations are included in same XDS document (as is somewhat common in the delimited text driver).

Make sure to veto these fake “BEGIN/END” nodes (in a subsequent policy) once they are no longer required. Otherwise, the engine will try and process them as valid add operations, which will most likely fail and generate an error.

<rule>
    <description>Tag Begin and End of Current Import</description>
    <comment xml:space="preserve">Bookend any valid operation nodes in the current XDS document with a fake "BEGIN/END" node.
Be sure to veto these out once they are no longer required.
Prequsisite: ToolBox rule "Determine if input or output"</comment>
    <conditions>
        <or>
            <if-local-variable mode="regex" name="inout" op="equal">input|output</if-local-variable>
        </or>
        <or>
            <if-xpath op="true">/nds/*[name()= $inout]/*[1]/@class-name[. != 'BookEnd']</if-xpath>
            <if-xpath op="true">count (/nds/*[name()= $inout]/*) = 0</if-xpath>
        </or>
    </conditions>
    <actions>
        <do-add-dest-object class-name="BookEnd" when="before">
            <arg-dn>
                <token-text xml:space="preserve">Vault/Data/Common/DummyObject</token-text>
            </arg-dn>
        </do-add-dest-object>
        <do-add-dest-attr-value class-name="BookEnd" name="Document" when="before">
            <arg-dn>
                <token-text xml:space="preserve">Vault/Data/Common/DummyObject</token-text>
            </arg-dn>
            <arg-value type="string">
                <token-text xml:space="preserve">BEGIN</token-text>
            </arg-value>
        </do-add-dest-attr-value>
        <do-add-dest-object class-name="BookEnd" when="after">
            <arg-dn>
                <token-text xml:space="preserve">Vault/Data/Common/DummyObject</token-text>
            </arg-dn>
        </do-add-dest-object>
        <do-add-dest-attr-value class-name="BookEnd" name="Document" when="after">
            <arg-dn>
                <token-text xml:space="preserve">Vault/Data/Common/DummyObject</token-text>
            </arg-dn>
            <arg-value type="string">
                <token-text xml:space="preserve">END</token-text>
            </arg-value>
        </do-add-dest-attr-value>
        <do-clone-xpath before="*[1]" dest-expression="/nds/*[name()= $inout]" src-expression="../*[position() &lt; last()]"/>
        <do-clone-xpath dest-expression="/nds/*[name()= $inout]" src-expression="../*[last()]"/>
        <do-strip-xpath expression="../*"/>
    </actions>
</rule>

As you can see, the conditions are designed such that the rule will only one once per XDS document and specifically only on the first operation within the aforementioned document.

The use of “before” and “after” ensures that the new fake add nodes and their attributes are correctly positioned in the “current operation” node set.

The rules clone all nodes from the “current operation” directly into the overall XDS document.
Note the various positioning criteria used in these expressions:

  1. before=”*[1]” – this means before the first node (if any nodes exist)
  2. ../*[position() &lt; last()] – this means everything in the current operation except the last operation. In this case, that is the fake “BEGIN” operation and the actual current-op node.
  3. ../*[last()] – the last operation, that is the fake “END” operation.
  4. That the two src-expressions are the exact inverse of each other, so together they select everything in the “current operation” node set.

Then do-strip-xpath is used to clear the “current operation” node set. At this point, the “current operation” is an empty node set, which the engine copies back into the XDS document.

Bookend (Add START/END nodes to XDS document) – XSLT stylesheet

For reference, here is some sample code to achieve the same thing via an XSLT stylesheet code

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- identity transformation template -->
    <!-- in the absence of any other templates this will cause -->
    <!-- the stylesheet to copy the input through unchanged to the output -->
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>
    <!-- Add a new object after the last add: -->
    <xsl:template match="//add[last()]">
        <xsl:call-template name="identity"/>
                <add class-name="BookEnd" dest-dn="Vault/Data/Common/DummyObject">
            <add-attr attr-name="Document">
                <value type="string">
                    <xsl:value-of select="BEGIN" />
                </value>
            </add-attr>
        </add>
    </xsl:template>
    <!-- Add a new object before the first add: -->
    <xsl:template match="//add[1]">
                <add class-name="BookEnd" dest-dn="Vault/Data/Common/DummyObject">
            <add-attr attr-name="Document">
                <value type="string">
                    <xsl:value-of select="BEGIN" />
                </value>
            </add-attr>
        </add>
        <xsl:call-template name="identity"/>
    </xsl:template>
    <xsl:template match="node()|@*" name="identity">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

The above XSLT stylesheet will only bookend add operations (though it could easily be adapted to be more generic). It is roughly the same length when you exclude comments and the like.

The DirXML-Script version is arguably easier to understand and maintain.

Number nodes (identify position in the current document)

Nodes are numbered via an operation property (which will persist until manually stripped or the operation is completed). Also note that any subsequent changes which add or remove operations/nodes (veto, add object etc.) will break the numbering.

This requires that rules only make changes whilst processing the first two operations in a node set. However the second rule does contain a “do-for-each” loop which iterates over every operation in the node set, so overall this should scale linearly with large XDS documents.

Include the two rules below (plus the ToolBox rule “Determine if input or output”) in a policy scheduled immediately prior to a policy/rule where you might need to know the position of each operation node.

Rule #1 (First Pass)

Add a dummy node, used later to determine position. Alternatively, if this is the only node in the XDS document, set its position to 1 (which will also ensure that rule #2 is not applied)

<rule>
    <description>Number Nodes - 1st Pass</description>
    <comment xml:space="preserve">Nodes not yet numbered because current-op position has not yet been determined.
Insert a dummy node and perform remainder of the processing on the next operation.
Prequsisite: ToolBox rule "Determine if input or output"</comment>
    <conditions>
        <and>
            <if-local-variable mode="regex" name="inout" op="equal">input|output</if-local-variable>
            <if-op-property name="position" op="not-available"/>
            <if-xpath op="not-true">//*[@class-name='dummy']</if-xpath>
        </and>
    </conditions>
    <actions>
        <do-if>
            <arg-conditions>
                <and>
                    <if-xpath op="true">count (//nds/*[name()= $inout]/*) > 0</if-xpath>
                </and>
            </arg-conditions>
            <arg-actions>
                <do-add-dest-object class-name="dummy" when="after">
                    <arg-dn>
                        <token-text xml:space="preserve">dummy</token-text>
                    </arg-dn>
                </do-add-dest-object>
            </arg-actions>
            <arg-actions>
                <do-set-op-property name="position">
                    <arg-string>
                        <token-text xml:space="preserve">1</token-text>
                    </arg-string>
                </do-set-op-property>
            </arg-actions>
        </do-if>
        <do-break/>
    </actions>
</rule>

Rule #2 (Second Pass)

Now we now know exactly where we are and the current operation node does not yet have a “position” operation property set. Therefore, we can determine the position of all nodes.

  1. Copy “current operation” back into XDS in same position as dummy node using do-clone-xpath.
  2. Remove dummy node from the XDS document via do-strip-xpath as it is no longer required.
  3. Clear “current operation” via do-strip-xpath.
  4. Number the nodes using a “do-for-each” loop that iterates over each node in the XDS document.

<rule>
    <description>Number Nodes - 2nd Pass</description>
    <comment xml:space="preserve">Now we know where we are.

1. Copy "current operation" back into XDS in same position as dummy node.
2. Remove dummy node
3. Clear "current operation"
4. Number the nodes
Prequsisite: ToolBox rule "Determine if input or output"</comment>
    <conditions>
        <and>
            <if-local-variable mode="regex" name="inout" op="equal">input|output</if-local-variable>
            <if-xpath op="true">//*[@class-name='dummy']</if-xpath>
        </and>
    </conditions>
    <actions>
        <do-set-local-variable name="counter" scope="policy">
            <arg-string>
                <token-text>0</token-text>
            </arg-string>
        </do-set-local-variable>
        <do-clone-xpath before="*[@class-name='dummy']" dest-expression="//nds/*[name()= $inout]" src-expression="../*"/>
        <do-strip-xpath expression="/nds/*[name()= $inout]/*[@class-name='dummy']"/>
        <do-strip-xpath expression="../*"/>
        <do-for-each notrace="true">
            <arg-node-set>
                <token-xpath expression="/nds/*[name()= $inout]/*"/>
            </arg-node-set>
            <arg-actions>
                <do-append-xml-element expression="$current-node[not(operation-data)]" name="operation-data"/>
                <do-set-local-variable name="counter" scope="policy">
                    <arg-string>
                        <token-xpath expression="$counter + 1"/>
                    </arg-string>
                </do-set-local-variable>
                <do-set-xml-attr expression="$current-node/operation-data" name="position">
                    <arg-string>
                        <token-local-variable name="counter"/>
                    </arg-string>
                </do-set-xml-attr>
            </arg-actions>
        </do-for-each>
    </actions>
</rule>            

Notice, that within the for-each loop, I make use of do-append-xml-element and do-set-xml-attr despite the best practices indicating that one should try to use the built in tokens where possible. This is always something where one should use common sense and choose a balance between the least intrusive, easiest to read and most efficient code.

In this case, using code to make each current-node variable temporarily the current operation added excessive overhead. The number of actions with the “do-for-each” loop increased to seven! Other actions were also required outside the “do-for-each” loop for proper clean up.

The two examples above gives a practical example of how one can combine many of the previously listed toolbox XPath and action tricks.

The next article in this series deals exclusively with direct operations, it will outline the technical details, review general best practices related to working with direct operations. It will outline how to manipulate and transform direct operations and provide some useful toolbox rules related to direct operations.

Series Navigation<< Delving into and beyond the current-op – Part 1Delving into and beyond the current-op – Part 3 >>
1 vote, average: 4.00 out of 51 vote, average: 4.00 out of 51 vote, average: 4.00 out of 51 vote, average: 4.00 out of 51 vote, average: 4.00 out of 5 (1 votes, average: 4.00 out of 5)
You need to be a registered member to rate this post.
Loading...Loading...

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment.  It just worked for at least one person, and perhaps it will be useful for you too.  Be sure to test in a non-production environment.

Leave a Reply

No Comments
Alexander McHugh
Jun 17, 2014
5:00 am
Reads:
1,032
Score:
4