How does the Synthetic Add process work on IDM 3.5.x

Here is the explanation on how the engine evaluates which attributes should go into a Synthetic add and how to disable the optimization piece. Excerpts from a level 3 trace will be used to exemply what is being discussed as needed.

When an event is cached by the engine, that event's timestamp is also saved in the cache. From a sample trace:

[06/24/08 08:17:36.302]:GroupWise ST:Start transaction.
[06/24/08 08:17:36.339]:GroupWise ST:Processing events for transaction.
[06/24/08 08:17:36.397]:GroupWise ST:
<nds dtdversion="3.5" ndsversion="8.x">
<source>
<product version="3.5.11.20080307 ">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify cached-time="20080624141736.191Z" class-name="User" event-id="NW65SP7#20080624141736#1#1" qualified-src-dn="O=idm\OU=users\CN=tburt6" src-dn="\OESNW_TREE\idm\users\tburt6" src-entry-id="33106" timestamp="1214317056#1">
<modify-attr attr-name="Description">
<add-value>
<value timestamp="1214317056#1" type="string">Sample</value>
</add-value>
</modify-attr>
</modify>
</input>
</nds>

  In the trace excerpt  above, the event's timestamp is 1214317056#1, as can be seem in the "timestamp" attribute of the <modify> XML tag. The first part of the timestamp is the number of seconds since 1/1/1970 GMT time, and the second part (after the # sign) is the event number within that second.

  Since the modify event does not have an <association> tag and its coming from eDirectory, that means that the object does not have an association.  Therfor, the event will need to be converted from a modify to an add. To do this, the event will pass to the end of the Subscriber Event Transformation Policy Set as a <modify>, then the engine will do the conversion.

  To do the conversion, the engine evaluates the driver's filter, and queries eDirectory based on the filter. An instance document is sent back from eDirectory with the attributes found among the ones queried, as well as their timestamps. Bellow you can see the event reaching the conversion point, followed by the query back into eDirectory:

[06/24/08 08:17:36.529]:GroupWise ST:
<nds dtdversion="3.5" ndsversion="8.x">
<source>
<product version="3.5.11.20080307 ">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<modify cached-time="20080624141736.191Z" class-name="User" event-id="NW65SP7#20080624141736#1#1" qualified-src-dn="O=idm\OU=users\CN=tburt6" src-dn="\OESNW_TREE\idm\users\tburt6" src-entry-id="33106" timestamp="1214317056#1">
<modify-attr attr-name="Description">
<add-value>
<value timestamp="1214317056#1" type="string">Sample</value>
</add-value>
</modify-attr>
</modify>
</input>
</nds>
[06/24/08 08:17:36.533]:GroupWise ST:Subscriber processing modify for \OESNW_TREE\idm\users\tburt6.
[06/24/08 08:17:36.534]:GroupWise ST:Converting <modify> to <add>
[06/24/08 08:17:36.546]:GroupWise ST:Reading relevant attributes from \OESNW_TREE\idm\users\tburt6.
[06/24/08 08:17:36.554]:GroupWise ST:
<nds dtdversion="3.5" ndsversion="8.x">
<source>
<product version="3.5.11.20080307 ">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<query class-name="User" dest-dn="\OESNW_TREE\idm\users\tburt6" dest-entry-id="33106" scope="entry">
<read-attr attr-name="CN"/>
<read-attr attr-name="company"/>
<read-attr attr-name="Description"/>
<read-attr attr-name="Facsimile Telephone Number"/>
<read-attr attr-name="Generational Qualifier"/>
<read-attr attr-name="Given Name"/>
<read-attr attr-name="Initials"/>
<read-attr attr-name="Login Disabled"/>
<read-attr attr-name="Login Expiration Time"/>
<read-attr attr-name="NGW: GroupWise ID"/>
<read-attr attr-name="nspmDistributionPassword"/>
<read-attr attr-name="OU"/>
<read-attr attr-name="personalTitle"/>
<read-attr attr-name="Surname"/>
<read-attr attr-name="Telephone Number"/>
<read-attr attr-name="Title"/>
</query>
</input>
</nds>

  And here is the instance document coming back as a result of the query. Notice that all attributes have timestamps as well:

[06/24/08 08:17:36.561]:GroupWise ST:Pumping XDS to eDirectory.
[06/24/08 08:17:36.568]:GroupWise ST:Performing operation query for \OESNW_TREE\idm\users\tburt6.
[06/24/08 08:17:36.708]:GroupWise ST:Read result:
[06/24/08 08:17:36.712]:GroupWise ST:
<nds dtdversion="3.5" ndsversion="8.x">
<source>
<product version="3.5.11.20080307 ">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<output>
<instance class-name="User" qualified-src-dn="O=idm\OU=users\CN=tburt6" src-dn="\OESNW_TREE\idm\users\tburt6" src-entry-id="33106">
<attr attr-name="CN">
<value naming="true" timestamp="1214227967#23" type="string">tburt6</value>
</attr>
<attr attr-name="Description">
<value timestamp="1214317056#1" type="string">Sample</value>
</attr>
<attr attr-name="Given Name">
<value timestamp="1214317057#1" type="string">tom</value>
</attr>
<attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
</attr>
<attr attr-name="Surname">
<value timestamp="1214227967#3" type="string">burt</value>
</attr>
</instance>
<status level="success"></status>
</output>
</nds>

  Now, by default values that are newer than the modify that triggered the synthetic add are intentionally excluded from the add, with the expectation that they will come through in a subsequent modify. That is the expected behavior, but if this behavior is undesirable in a given scenario, it can be overridden by stripping the timestamp from the original modify in the Subscriber Event Transformation Policy Set. Since in the trace above "Given Name" has a timestamp that is newer than the one in the original event, the resulting add won't have it, as shown bellow:

[06/24/08 08:17:36.720]:GroupWise ST:Synthetic add:
[06/24/08 08:17:36.722]:GroupWise ST:
<nds dtdversion="3.5" ndsversion="8.x">
<source>
<product version="3.5.11.20080307 ">DirXML</product>
<contact>Novell, Inc.</contact>
</source>
<input>
<add class-name="User" event-id="NW65SP7#20080624141736#1#1" qualified-src-dn="O=idm\OU=users\CN=tburt6" src-dn="\OESNW_TREE\idm\users\tburt6" src-entry-id="33106">
<add-attr attr-name="CN">
<value naming="true" timestamp="1214227967#23" type="string">tburt6</value>
</add-attr>
<add-attr attr-name="Description">
<value timestamp="1214317056#1" type="string">Sample</value>
</add-attr>
<add-attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
</add-attr>
<add-attr attr-name="Surname">
<value timestamp="1214227967#3" type="string">burt</value>
</add-attr>
</add>
</input>
</nds>

In case there is the need to disable this optimization of removing future events from the <add> document, a sample policy is provided bellow as an example on how to strip the original's event timestamp. This policy shold be properly tested before use in production environments:

<policy>
<rule>
<description>Remove event timestamp in object not associated</description>
<comment xml:space="preserve">Sample rule created to show customers one way of disabling the Synthetic add optimization. This rule should go in its own policy within the driver's Subscriber Event Transformation Policy Set.</comment>
<conditions>
<and>
<if-operation mode="case" op="equal">modify</if-operation>
<if-association op="not-associated"/>
</and>
</conditions>
<actions>
<do-strip-xpath expression="@timestamp"/>
</actions>
</rule>
</policy>