License:
MPL (Mozilla Public License)

Download GenFileDriver_0_8A.zip

TOC

Introduction
New features and fixes
Installation
Configuration

 

Introduction

The Generic File Driver is similar to the Text Driver shipped with IDM, but has more options, and has the capability to read virtually any file type. Out of the box, the following file types are supported: XML, CSV and XLS (the latter using POI from Apache).

Main differentiators to NetIQ’s driver:

  • Produces and consumes regular NDS documents (no need for XSLT conversion).
  • Accepts add, modify or instance documents on the subscriber channel.
  • Auto-query back for missing attributes on modify commands.
  • GCV (Ecmascript) for generating associations and source DN’s.
  • Publisher channels processes files in a streaming mode. A maximum of 20 records are prefetched/buffered in memory (whenever support by the file type: CSV and XML).
  • Publisher channel supports interruption (driver stop command) during file processing.
  • Publisher channel can add metadata about the file or record processed (eg: record number, filename, last record indicator, etc).

The original version I wrote in 2007 is located at http://www.novell.com/coolsolutions/tools/18671.html. This update comes with an IDM 4 package making the installation simpler and has some new features and fixes (see below).

See the referenced version for a list of the initial features.

 

New features and fixes

version 0.8A, 3-MARCH-2015 (package 0.7)

  • Shim feature – general – reverted back to java 1.6 compatible ‘move file’ operation.

version 0.8, 14-SEPT-2014 (package 0.7)

  • Package bug: fixed typo in example xslt template for XML publisher files. See forum thread.
  • Technical – general refactoring of the shim.
  • Shim feature – publisher – added option to remove old publisher files. Disabled by default for backward compatibility.
  • Shim feature – publisher – added option to disable the publisher channel.
  • Shim feature – publisher – added additional options in the command to generate (“Dynamic Input based” and “Delete”)
  • Shim feature – subscriber – added option to disable the “max number of records in generated files” by setting it to 0 (zero).
  • Shim feature – subscriber – when a modify command is ’empty’ (=all fields defined in the schema are missing), but the content indicates that the output file should be closed (eg: the file close field is set to true), no query back is performed in order to add a record to the output file. It is assumed that the only purpose of this modify command is to close the output file.
  • Shim feature – general – added option to replace illegal ECMA script characters in field names (eg for calculating the association). Disabled by default for backward compatibility.
  • Shim feature – general – added option to include the driver’s ECMA script libraries into the shim (for complex ECMA script operation). Disabled by default for backward compatibility.

version 0.7b, 14-MAY-2014 (package 0.6)

  • Shim bug: case issue with the new configuration parameter to configure the command generated on the publisher channel

version 0.7, 07-MAY-2014 (package 0.6)

  • Shim bug: specifying no encoding on the XML reader caused an Exception (see also this cool solution post)
  • Shim feature: added option to configure the command generated on the publisher channel (add, modify or dynamic).
  • Package bug: xml and xls file readers had invalid class names (see the same cool solutions post)
  • Package feature: changed xls entry field to multi-line

version 0.6, 28-FEB-2014 (package 0.5)

  • Shim/package feature: Added configuration parameter to use Quartz scheduler to close subscriber generated files using a cron-like expression string.
  • Shim feature: Added slf4j binding (separate jar) for Idm tracing, allowing Quartz logs to be shown in idm trace files.
  • Shim bug: possible NPE when the publisher folder becomes unreachable (eg due to network outage).
  • Shim feature: Added option for every strategy to receive driver shutdown notification by implementing IShutdown.
  • Shim bug/feature: Return an empty instance document when we receive an associated query on the subscriber channel. Reason: support of the merge attribute filter rules in case of a ‘sync’ event.

version 0.5, 21-JAN-2013 (package 0.4)

  • Shim bug: Fixed an issue wrt the driver not adding the metadata on the publisher channel.

version 0.4, 24-APR-2012 (package 0.4)

  • Shim bug: Fixed an issue with the driver schema (appeared to be hard coded).
  • Shim feature: Added beta support for query back. Query back only supported on publisher channel (when reading in a file). Feedback welcome.
  • Package bug: Fixed an issue that caused the driver configuration not to be imported when the package was imported.
  • Package bug: Changed creation policy from validating the Given Name to validating the Surname.
  • Package feature: Added package ‘Delimited Text Driver Password Synchronization Package’ as an optional package for this driver.
  • Package feature: Added remote loader prompts to the package.

version 0.3, 12-OCT-2011

  • Shim feature: automatic association and src-dn calculation (using ecmascript in the driver configuration). No longer need for style sheets for just creating association and src-dn values. The driver shim now does this for you.
  • Shim bug: fixed missing remove-all-values on the publisher documents.
  • Shim feature: added the ability (csv output files) to flush the file buffer after every record written (useful for when you are developing/testing your driver).
  • Shim feature: when receiving a modify event on the subscriber channel, the shim will query back for all missing attribute values in order to be able to write a complete record (no need for a stylesheet to query for missing attributes and keeping this stylesheet in sync with the schema).
  • Shim bug/feature: the subscriber channel now fully support add, modify and instance events. Instance and add events are assumed to contain all data and do not perform a query back. Only the modify event performs a query back

 

Installation

The attached zip file (GenFileDriver_0_8.zip) contains (beside some documentation) 2 main jar files and a lib folder. One jar is the package, one jar is the shim. The lib folder contains optional jars.

 

Shim: GenFileDriverShim_0.7.jar
Copy the shim to you eDirectory server (DirXML’s classes folder). If you want to use XLS support, also download poi from Apache. Restart eDirectory after this.

Note: if you have an older version of the shim, be sure to delete that jar.
Package: SVCGENFILEB_0.0.6.20140508075632.jar
Import this package into your designer project. Drag and drop the “Delimited Text” driver that is located under “Tools” and select the “Generic File Base” package, answering all the questions as needed.
From this point on, you can use and modify it as any other driver.

lib folder:
This folder contains some optional jars. These are only required if you plan to use quartz scheduler to close the generated files using a cron-like string. Jars required when doing this are: quartz-2.2.1.jar, slf4j-api-1.7.5.jar and slf4j-idmTrace.jar.

Policies

This driver only has 3 policies; all of them on the publisher channel. If required, you could add the password policies as available from the standard text driver (by creating your own copy from Novell’s package and adding this to the driver).
SVCGENFILEB-pub-mp
Match on Internet Email Address if any available.
SVCGENFILEB-pub-cp
Checks for mandatory attributes:
Surname: required due to eDir constraints
Internet Email Address: required due to use in matching policy
SVCGENFILEB-pub-pp
Placement based on a driverset GCV (idv.dit.data.users).

Configuration

All configuration options should have a good description. Most of them are also described on the initial release page.

4 votes, average: 5.00 out of 54 votes, average: 5.00 out of 54 votes, average: 5.00 out of 54 votes, average: 5.00 out of 54 votes, average: 5.00 out of 5 (4 votes, average: 5.00 out of 5)
You need to be a registered member to rate this post.
Loading...Loading...

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment.  It just worked for at least one person, and perhaps it will be useful for you too.  Be sure to test in a non-production environment.

Leave a Reply

26 Comments

  • joakim_ganse says:

    Hi,

    The wizard did not create the driver config properly for me.
    Had to copy and paste one from your previous version.

  • al_b says:

    Hi,
    Could you advice please, how I can receive access to file metadata?

    I added in Publisher option elements of metadata:

    <definition display-name=”List of meta data elements that should be added:” name=”pub_metaData” type=”string”>
    <description>Publisher: List of meta data elements that should be added(recordNumber,isLastRecord,filePath,fileName,fileSize). The schema as given in the driver parameter ‘schema’ will be extended with all meta data elements listed. Separate each item with a comma.</description>
    <value xml:space=”preserve”>recordNumber,isLastRecord,fileName,fileSize</value>
    </definition>

    but I can’t see in a trace, that any of this elements available for me. :(
    I found ugly way to get access to recordNumber information via init-params, but I believe, that you foreseen more elegant and simple way.

    00:06:03 914 Drvrs: </add>
    <init-params>
    <publisher-state filePath=”c:\work\2013.01.20_00.06.03\dts.csv” fileReader=”info.vancauwenberge.filedriver.filereader.csv.CSVFileReader” recordNumber=”1″/>
    </init-params>
    </input>
    </nds>

    I tried to use version 0.4 (GenFileDriverShim_0.4.jar, SVCGENFILEB_0.0.4.jar), and may be some bug (or misunderstanding) prevent me from see this info.

    Thank you for your help,
    Alex

  • joakim_ganse says:

    Hi,

    This is a truly brilliant driver :) Starting to replace all text drivers with this one.
    One thing I have noticed is that when using it with a Remote Loader on Winows I can not brows for the class.
    What I did was to choose a diffrent class and then edit the txt file.
    This is not enough though, I got a java Lang error.
    So I discovered I had to edit the registry as well, and pase in the classname there to.
    After that I could edit the config the normal way, had to put in the passwords again.

    The classname is info.vancauwenberge.filedriver.shim.driver.GenericFileDriverShim

    Cheers

  • mJg2XW says:

    Hi.

    Greate driver:)

    Do the driver have some sort of PostProcessor funktion on the Publisher?
    /Michael

  • joer999 says:

    Exactly what I needed: a text driver which can produce CSV’s with a header. Also neat is the feature of the Work & Destination directory.
    Great work.

  • mJg2XW says:

    Hi

    Sorry for my delay.

    I would like to send a XDS document to a diffrent driver or bumping up a “counter” on a specific object or just to do /generate some sort of events that signal that the .CSV driver i finish, so other driveres begin doing some work.
    /Michael

    • scauwe says:

      Hi Michael,
      You can have metadata in the driver to notify you when it is the last record of the file. You can always, in the command transformation on the publisher, react on this and do whatever you want after the last record (eg: increase a counter somewhere, create another object (workorder?),… so that another driver is triggered).

  • mJg2XW says:

    Hi

    Thanks for your replay, I will give it a try.

    /Michael

  • aarondickinson says:

    Hi,

    I was just wondering if there was any tricks to getting the Gen File driver connected to a Remote Loader?

    I’ve coppied the appropriate .jar across and (manually) edited the instance config to use the correct class but I keep getting this fatal error:

    Unable to validate that there is a non-empty driver object password (a Publisher-channel Policy may be incorrect)

    And before I start to diagnose the stylesheets (I believe that is there the issue might be), I was just wondering if there was anything else I should keep in mind as I didn’t see anything about connecting to an RL.

    Thanks in advance,

    Aaron

    • scauwe says:

      Hi Aaron,

      I run it in a remote loader all the time, both on Linux and on Windows, and never had any issues. For windows, it might be more tricky (see joakims comments in https://www.netiq.com/communities/cool-solutions/cool_tools/generic-file-driver-idm-v-003/#_comment-10776).
      What style sheet are you referring to?

      Stefaan

      • aarondickinson says:

        Oh Great. I missed that reply.
        I noticed the same issue with the drop down myself, but didn’t catch the java error as a 3rd party is running the RL.
        Will pass this info on.

        The StyleSheet I’m referring to is probably my Pub Pre-Processor XSLT.
        I see that when the driver requests the driver obj password from the RL, it gets a response (so the query is being processed by the sub XSLTs), but then the pre-processor converts the query response into an add event… :/
        The XSLT was made by a contractor so I may just have to add in a template for queries.

      • scauwe says:

        No problem.
        Note that the ‘standard’ xslt (the one used with NetIQ’s file driver) is no longer needed when using this shim. You get plain xds documents from the shim.

  • pcook says:

    I’m unable to get this driver to start on a brand-new, fully patched eDir 8.8.8/IDM 4.0.2 server. I get a fatal error upon attempting to start the driver:

    java.lang.IllegalArgumentException: No enum constant info.vancauwenberge.filedriver.shim.FileDriverPublicationShim.GenerateCommnd.add

    I’ve tried all the other “generate command” options (modify,dynamic) with the same result. What could be causing this?

  • SchoellU says:

    Hi,
    I’m unable to get the Driver (v0.8 and earlier) to start on a 4.0.2 IDM-Server. Is fully patched eDir 8.8.8/IDM 4.0.2. Did receive a fatal error upon attempting to start the driver:
    Code(-9010) An exception occurred: java.lang.NoSuchMethodError: java.io.File.toPath()Ljava/nio/file/Path;
    at info.vancauwenberge.filedriver.util.Util.moveFile(Util.java:179)

    Code(-9203) Error saving driver persistent data: Code(-9010) An exception occurred: com.novell.nds.io.JCIOException: modifyEntry: -608
    at com.novell.nds.io.NDSStreamAttributeOutputStream.(NDSStreamAttributeOutputStream.java:111)

    Any idea?
    Thanks

  • SchoellU says:

    additional Information:
    the error occures during the following activity:
    Moving file from /var/opt/novell/…/jobinfo.csv to /var/opt/novell/…/work/jobinfo.csv

    Thanks in advance
    Uwe

    • scauwe says:

      Hi Uwe,

      Since version 0.8 of the driver, I changed the method of moving files. I switched from the java File.rename() to Files.move()
      Files.move requires a Path argument. It now seems that File.toPath() is introduced in java 1.7. IDM ships with java 1.6.
      The part that confuses me is: how comes it works on my systems… It should not.
      I’ll try to publish a patched 0.8 version ASAP that does not use File.toPath().

      Stefaan

      • SchoellU says:

        Hi Stefaan,
        thanks for answer
        Now i am confused too, we had also tested with version 0.5 and Java 1.6 with the same result…
        Regards
        Uwe

  • kellywm1 says:

    Is there a patched version that fixes the Files.move() issue yet?

    An exception occurred: java.lang.NoSuchMethodError: java.io.File.toPath()Ljava/nio/file/Path;

    Occurs when driver is trying to move input csv file from IN to WIP directory. Java version on server is ;

    java version “1.6.0_21″
    Java(TM) SE Runtime Environment (build 1.6.0_21-b07)
    Java HotSpot(TM) Client VM (build 17.0-b17, mixed mode, sharing)

    I cannot update the server to java 1.7 or later, other java apps used by other teams require the 1.6.0_21 specific version.

    Thanks

  • joer999 says:

    Great enhancement over the standard text driver, thanks.

By: scauwe
Jan 31, 2013
5:33 am
Reads:
5,817
Score:
5