4.5 F/X Monitor Configuration File

The F/X Monitor ships with some sample configuration files for the Monitor and an example. For more information, see Section C.0, Logging Configuration File.

F/X Monitors are configured using an XML configuration file (with the exception of logging configuration).

The configuration file structure logically follows the F/X Monitor architecture. It defines the following items:

  • F/X Monitor level settings

  • Adapter definitions, which are the location and type of recipients

  • Reader definitions, which are of collectors and the parsers they use

  • Parser configurations, which are named parser configurations that are referenced by reader definitions

    Currently, there is an n‑to‑1 relationship between reader definitions and parser configurations.

As an XML document, the header in the F/X Monitor configuration file is an XML syntax declaration. Also the F/X Monitor configuration must be encapsulated within F/X Monitor open and close tags.

The following code shows the XML tag structure of an F/X Monitor configuration file:

<?xml version="1.0"?>

<fxmonitor>
  <adapters>
      <!--
        Add adapter definitions here Adapter definitions are the destinations
        for alarms generated by readers
      -->
  </adapters>
  
  <settings>
    <!--
        Optional ability to specify a filestore for temporary files created by
        the FX Monitor if not specified the FX Monitor uses a temporary
       directory underneath the O/S reported temporary directory.
    -->
      <filestore></filestore>
  </settings>

   <readers>
     <!--
      Add reader definitions here.  Readers combine a collector and a parser to
     collect and parse data in order to generate alarms
      -->
  </readers>
  
  <parsers>
    <!--
      Add parser definitions here. Parser definitions define a named parser
      configuration,which uses one of the provided Parser classes and
      configures it to process data content as required to generate alarms 
      with a set of properties (or 'columns' or 'fields')
      -->
  </parsers>
  
</fxmonitor>

The following sections describe the F/X Monitor Configuration file definitions and tags in further detail:

4.5.1 Adapter Definitions

The first main component of the F/X Monitor configuration file is the adapters section, which identifies the target/s for alarms generated by the F/X Monitor.

Specify any number of adapter targets and optionally put them into arbitrary groups that determine how the F/X Monitor responds to failure to deliver alarms to targets.

Adapter targets are defined in the configuration file within an adapters tag. Each target definition is contained within an adapter tag, as illustrated:

<?xml version="1.0"?>
<fxmonitor>
  <adapters>
    <adapter>
    <!-- individual adapter settings go here -->
    </adapter>
  </adapters>
  <!-- remaining fx monitor configuration content goes here -->
</fxmonitor>

There are no limitations on the number of adapter definitions.

An excerpt from a configuration file is shown in the following example. It defines an adapter for the F/X Monitor to send alarms to as an F/X adapter on the host mainserver.mycorp.com on port 50001.

<adapter>
<name>Main FX Adapter</name>
  <type>PRIMARY</type>
  <group>My Production Group</group>
  <alarmformat>XML</alarmformat>
  <adapterport>50001</adapterport>
  <adapterhost>mainserver.mycorp.com</adapterhost>
  <adapterretry>60</adapterretry>
</adapter>

The following list describes the required and optional adapter parameters:

  • <adapterhost>: (Required) The hostname or IP address of the listening adapter:

    <adapterhost>hostname_or_IP_address</adapterhost>
    
  • <adapterport>: (Required) The TCP/IP port on which the adapter is listening for connections:

    <adapterport>adapter_port</adapterport>
    
  • <adapterretry>: (Required) The frequency (in seconds) with which to attempt connection in the event of failure:

    <adapterretry>retry_frequency</retry>
    
  • <alarmformat>: Specifies the alarm format produced for the adapter. For F/X adapters, use the default XML type:

    <alarmformat>format_type</alarmformat>
    

    where format_type is one of the following:

    • RL: Reception Log format for legacy universal and TEC adapters.

    • XML: FX XML event format (default).

    The value is not case-sensitive.

  • <group>: The group the adapter definition belongs to:

    <group>group_name</group>
    

    If unspecified, the DEFAULT_GROUP group name is used.

  • <maxwait>: The maximum time (in seconds) that an alarm waits if it cannot be sent:

    <maxwait>maximum_wait_time</maxwait>
    

    The default is 365 days. If an alarm is not sent to the target within the <maxwait> time period, it is deleted from the delivery queue by the F/X Monitor.

  • <name>: Used only for reference:

    <name>text_name_of_adapter</name>
    

    Specify a text name representing the adapter definition.

  • <type>: Specifies the operational role of the adapter within its adapter group:

    <type>operational_role</type>
    

    where operational_role is one of the following:

    • primary: Preferred target within the group.

    • backup: Fail-over target within the group.

      Groups can have 1 primary and any number of backup definitions. If more than one primary is specified, the second and subsequent definitions are downgraded to backup by the F/X Monitor.

    • multicast: Indicates that this target is in a multicast group and always receives alarms irrespective of the availability of other members of the group.

    The value is not case-sensitive.

4.5.2 Monitor Configuration Setting Definitions

There is only one F/X Monitor-level configuration parameter, and it is optional. The filestore tag provides the directory location where the F/X Monitor stores temporary files. The default is a subdirectory of the System temporary directory.

  <settings>
      <filestore>directory_location</filestore>
  </settings>

4.5.3 Reader Definitions

Readers are the definition of a collector to gather file or URI sourced data and a parser to process that data into alarms. The majority of a reader definition consists of the configuration of the chosen collector. The F/X Monitor maintains all readers as separate threads, ensuring that any latency incurred by the collection or parsing process does not impact other collection threads.

To enable parser definitions to be reused across readers, they are referenced by name and defined in a separate <parsers> section of the configuration file, as illustrated:

<readers>
  <reader>
    <name>name_of_reader</name>
    <class>class_name_of_collector_to_use</class>
    <parser>parser_name_to_use</parser>
    <settings>
      <!-- collector configuration here -->
    </settings>
  </reader>
</readers>

An example reader definition is shown in the following example. Note the name, class, and parser definitions and the detail of the collector configuration contained in the <settings> tag.

<readers>
   <reader>
    <name>Blah News Reader</name>
    <class>com.mosol.integration.fx.collectors.URICollector</class>
    <parser>Generic RSS Parser</parser>
    <settings>
      <!-- Every 60 seconds get the latest RSS news feed -->
      <uri>http://blah.com/feeds/rss.xml</uri>
      <timeout>10</timeout>
      <poll>60</poll>
      <retry>10</retry>
    </settings>
  </reader>
</readers>

The following list describes the required parameters for the <reader> tag:

  • <name>: Text name representing the reader:

    <name>name_of_this_reader</name>
    
  • <parser>: Name of the parser to use with this reader:

    <parser>name_of_the_parser_to_use</parser>
    

    The named parser must be defined in the <parsers> section of the configuration file.

  • <class>: The Java class and package name of the collector to use:

    <class>full_collector_class_name</class>
    

The above tags are children of the <reader> tag and are not part of the <settings> tag. The <settings> tag contains only the configuration for the chosen collector.

The standard collectors delivered are:

  • Directory Collector: Monitors the directory specified for files matching the file name matching criteria (defined by a regular expression). When new files are written to the directory or files are updated, the contents are read and passed to the associated parser configuration for processing.

  • FTP Collector: Monitors an FTP server, with optional directory location, for files meeting the specified file matching criteria. When new files are available or files are updated, the file is transferred to a local cache and its contents passed to the associated parser configuration for processing.

  • URI Collector: Connects to a URI (HTTP or HTTPS) and returns the content for processing by the associated parser. The collector caches the returned content and only passes it to the parser in the event of a content change.

  • Command Collector: Issues a command to the F/X Monitor host and returns the content from stdout and (optionally) stderr for the command.

The following defines common collector parameters specified inside the settings tag are common for all collectors. The only exception for the required parameters is in the case of user-developed collectors, which do not require a polling model. All of the standard F/X collectors implement polling, thereby requiring these properties.

  • <delay>: Delay is an optional stagger to defer collection in situations where an F/X Monitor has a large number of collectors, which the administrator does not want to invoke simultaneously:

    <delay>delay_in_seconds</delay>
    

    The default is 0 (zero).

  • <poll>: (Required) The time period in which the collector polls the source to identify new or updated content:

    <poll>poll_frequency_in_seconds</poll>
    
  • <retry>: The frequency to attempt connection in the event of failure:

    <retry>retry_frequency_in_seconds</retry>
    

Each collector might add extra configuration settings to a set of standard collector parameters, which are described in the following sections:

Directory Collector Parameters

Classname: com.mosol.integration.fx.collectors.DirectoryCollector

The Directory Collector adds a <directory> and <filematch> setting to the standard set.

The following lists the Directory Collector parameters:

  • <directory>: (Required) Directory to monitor for file activity:

    <directory>explicit_directory</directory>
    
  • <encoding>: Enables the encoding of the file to be specified:

    <encoding>encoding_type</encoding>
    

    Default is UTF‑8.

    To specify UNICODE encoding instead of default UTF‑8:

    <encoding>UNICODE</encoding>
    
  • <filematch>: (Required) Regular expression match on the file names to qualify for collection:

    <filematch>regular_expression</filematch>
    

    For example, to collect all .csv files:

    <filematch>.+\.csv</filematch>
    
  • <filemaxage>: Restricts file matching based on the last update time reported by the operating system:

    <filemaxage type="seconds|minutes|hours|DAYS|months">number_of_units</filemaxage>
    

    Matched files can be no older than the specified maximum file age, defined in terms of a number of seconds, minutes, hours, days, or months. Case insensitive.

    The default is days. The value must be a positive integer.

    If no <filemaxage> tag is specified, then the collector does not differentiate between files based on their date stamp (last modified time).

    Examples:

    To process files less than 12 hours old:

    <filemaxage type="hours">12</filemaxage>
    

    To process files less than 2 months old:

    <filemaxage type="months">2</filemaxage>
    
  • <persist>: Set to yes and the F/X Monitor persists the details of processed files:

    <persist>yes|NO</persist>
    

    If a Monitor stops and restarts, it continues from the last recorded point and does not resend alarms previously delivered.

    This tag applies only to F/X Monitors with session names. Also, progress information is maintained between F/X Monitors with the same name.

    For more information on F/X Monitor startup and naming, see Section 4.2, Starting an F/X Monitor. Progress information persists only between F/X Monitor sessions of the same name because it is possible for more than one F/X Monitor to match files from the same source, but have differing progress in processing files.

    If the tag is not added or is set to NO, persistence does not occur.

    This feature works for both tail and nontail readers across all collector types.

  • <sendfile>: Instructs the F/X Monitor to send an alarm describing the file, whether content was processed or not:

    <sendfile>ALL|CHANGES</sendfile>
    

    where:

    • ALL: Send an alarm for each file, irrespective of whether it has changed since the last poll.

    • CHANGES: Only send an alarm for new, or changed, files.

    The alarm details the file name, size, and date stamp.

  • <tailfile>: Set to yes to provide the ability to process only recently added content, rather than rescanning complete files:

    <tailfile>yes|NO</tailfile>
    

    If an updated file is a smaller size than the version found during the previous scan, it is assumed that the file has changed fundamentally and the Monitor automatically reprocesses the complete file.

FTP Collector

Classname: com.mosol.integration.fx.collectors.FTPCollector

The FTP Collector adds a <directory> and <filematch> setting to the standard set much like the Directory Collector, but also adds FTP server authentication settings.

The following lists the FTP Collector parameters:

  • <directory>: (Required) Directory to monitor for file activity on the FTP server:

    <directory>relative_directory</directory>
    

    This is a relative directory on the FTP server.

  • <encoding>: Enables specifying encoding type of the file:

    <encoding>encoding_type</encoding>
    

    The default is UTF‑8.

    To specify UNICODE encoding instead of default UTF‑8:

    <encoding>UNICODE</encoding>
    
  • <filematch>: (Required) Regular expression match on the file names to qualify for collection:

    <filematch>regular_expression</filematch>
    

    For example, to collect all .csv files;

    <filematch>.+\.csv</filematch>
    
  • <filemaxage>: Restricts file matching based on the last update time reported by the operating system:

    <filemaxage type="seconds|minutes|hours|DAYS|months">number_of_units</filemaxage>
    

    Matched files can be no older than the specified maximum file age, defined in terms of a number of seconds, minutes, hours, days, or months. Case insensitive.

    The default is days. The value must be a positive integer.

    If no <filemaxage> tag is specified, then the collector does not differentiate between files based on their date stamp (last modified time).

    Examples:

    To process files less than 12 hours old:

    <filemaxage type="hours">12</filemaxage>
    

    To process files less than 2 months old:

    <filemaxage type="months">2</filemaxage>
    
  • <filepolicy>: Determines the policy to be applied to files transferred to the monitor for parsing:

    <filepolicy [suffix="renamed file suffix"]>LEAVE|DELETEALL|DELETELOCAL| DELETEREMOTE|RENAME</filepolicy>
    

    The F/X Monitor transfers and caches a local copy of all processed (matched) files.

    The options are:

    • LEAVE: Leave transferred files on the FTP server.

    • DELETEALL: Delete files on the FTP server and the F/X Monitor’s local copy.

    • DELETELOCAL: Delete just the F/X Monitor’s local copy after parsing.

    • DELETEREMOTE: Delete just the FTP server’s file.

    • RENAME: Rename the file on the FTP server after transferring, appending the suffix value defined in the filepolicy suffix argument.

  • <password>: (Required) Shows FTP password in the clear:

    <password>clear_password</password>
    

    TEA (tiny encryption engine) encrypts the password. For example:

    <password cipher="TEA">encypted_password</password>
    
  • <persist>: Set to yes and the F/X Monitor persists the details of processed files:

    <persist>yes|NO</persist>
    

    If a Monitor stops and restarts, it continues from the last recorded point and does not resend alarms previously delivered.

    This tag applies only to F/X Monitors with session names. Also, progress information is maintained between F/X Monitors with the same name.

    For details on F/X Monitor startup and naming, see Section 4.2, Starting an F/X Monitor. Progress information persists only between F/X Monitor sessions of the same name because it is possible for more than one F/X Monitor to match files from the same source, but have differing progress in processing files.

    If the tag is not added or is set to NO, persistence does not occur.

    This feature works for both tail and nontail readers across all collector types.

  • <port>: (Required) FTP server port:

    <port>port_number</port>
    

    The default is 22

  • <sendfile>: Instructs the F/X Monitor to send an alarm describing the file, whether content was processed:

    <sendfile>ALL|CHANGES</sendfile>
    

    where:

    • ALL: Send an alarm for each file, irrespective of whether it has changed since the last poll.

    • CHANGES: Only send an alarm for new, or changed, files.

    The alarm details the file name, size, and date stamp.

  • <server>: (Required) Hostname or IP address of the FTP server:

    <server>hostname_or_IP_address_of_FTP_Server</server>
    
  • <user>: (Required) FTP user name:

    <user>relative_directory</user>
    

URI Collector

The URI Collector takes a URI and uses it to collect content for parsing.

The following lists the URI Collector parameters:

  • <uri>: (Required) URI, such as a Web URL:

    <uri>URL</uri>
    

    For example, to recover the CNN RSS feed:

    <uri>http://rss.cnn.com/rss/cnn_topstories.rss</uri>
    
  • <timeout>: Timeout value for connection attempts:

    <timeout>timeout_in_seconds</timeout>
    

Command Collector

(Required) The Command Collector takes a command string, which the F/X Monitor executes each poll period, returning the stdout and stderr content for parsing. The Command Collector also inherits the standard collector properties.

The following is the syntax for the Command Collector parameter:

<command [timeout=timeout_seconds] [includestderr="NO|yes"]>command_to_execute</command>

This tag defines the command to run and optionally provides a timeout value. If the command has not returned (completed) by the timeout period, the invoked process is terminated by the F/X Monitor.

On a Windows system, this example lists the users, but terminate if the command takes more than 5 seconds to complete:

<command timeout=5>cmd /c net user</command>

You can specify that any stderr output of the process also be captured for processing using the includestderr argument. On a UNIX system, this example lists the running processes that contain mos and capture stderr:

<command timeout=10 includestderr="yes">ps –ef | grep mos*</command>

4.5.4 Parser Definitions

As previously described, parsers are utilized by defining parser configurations. Parser configurations are defined with a name for reference. They identify the parser class to use, then supply the configuration for that parser class.

In a similar manner to collectors, parsers inherit common configuration parameters and can extend them further to the processing type. This document describes the standard (common) parameters, then details the additional parameters available with each Operations Center parser.

Parser definitions appear in the F/X configuration file within the <parsers> section, in the following format:

<parsers>
  <parser>
    <name>name of parser</name>
    <class>class name of parser to use</class>
      <settings>
      <!-- parser configuration here -->
      </settings>
  </parser>
</parsers>

Two parameters are required beneath the <parser> tag:

  • <name>: Text name representing the parser:

    <name>parser_name</name>
    
  • <class>: The java class and package name of the parser to use:

    <class>full_parser_class_name</class>
    

The parser specific configuration parameters are contained within the <settings> tag.

For all of the Operations Center supplied parsers, the class name to use is:

com.mosol.integration.fx.parsers.parser_class

The following describes the standard class parsers:

  • DelimitedFileParser: Processes delimited content. The capabilities include:

    • Definition of the delimiter to use

    • Automatic property naming using headerline

    • Handles nested delimited content

  • CSVParser: A specialization of the Delimited File Parser for comma-separated files.

  • TabDelimiterParser: A specialization of the Delimited File Parser for tab-delimited files.

  • XMLParser: Processes XML datasets. Its capabilities include:

    • XPath-based identification of sections of interest for alarm generation

    • Relative addressing (incorporation) of parent and child tag values and attribute values

    • Automated conversion of date and numeric formats

    • Date conversions to specify alarm last updated time from content

  • RegExpParser: Applies regular expression grouping by file line. Enables splitting of files into alarms with properties. Severity matching is based on content matching. It specifies the alarm last updated time using the content.

  • SimpleTextParser: Performs line by line processing of text files, using content match to determine which lines are promoted as alarms.

The following topics cover using parsers:

Standard Parser Configuration Parameters

Since the original release of the F/X adapter, a great effort was made to make available to all parsers a standard set of parsing configuration parameters. The result is the following standard configuration parameters that apply to dataset processing using any parser:

  • <cleandata>: Enables cleaning of column data by providing a regular expression that identifies a group to extract and set as the column value:

    <cleandata column="column_to_clean">matcher</cleandata>
    

    If the matcher does not match the column value, no changes are made. This also enables multiple <cleandata> tags to be defined for the same column, if required.

    For example, to clean the TicketID column by removing the TKT prefix from the ticket number:

    <cleandata column="TicketID">^TKT([0‑9]+).*</cleandata>
    

    Assume the sample column value is TKT12345. Then, after processing by the <cleandata> tag, the value of the TicketID column is 12345.

  • <columnaliases>: Replaces column names (if found). Generally, <columnaliases> is used when automatic headers are found, such as with .csv files:

    <columnaliases>comma-separated_list_of_alias_definitions_column=alias_name</columnaliases>
    

    For example, to replace the original column named Bldg with Building and replace the column named Usr with User:

    <columnaliases>Bldg=Building,Usr=User</columnaliases>
    
  • <datecolumns> and <dateformats>: Used in conjunction, a comma-separated list of columns containing dates and the corresponding date format strings:

    <datecolumns>comma-separated_list_of_columns</datecolumns>
    

    <datecolumns> is a comma-separated list of columns containing date or time stamp data.

    <dateformats>yyyy‑MM‑dd HH:mm:ss,yyyy‑MM‑dd HH:mm:ss</dateformats>
    

    <dateformats> is a comma-separated list of date formats that describes how to convert the date string into a real date for processing. It uses the Java SimpleDateFormat syntax.

    The contents of the columns are parsed using the corresponding <dateformat> entry. For all datecolumns successfully parsed, the column value is replaced by the long representation of the date/time and a new column is added using the name columnname.original with the original content.

    It is assumed that the <datecolumns> and <dateformats> references are listed in the same order. If a <dateformats> entry is not defined for a given column, the default local date/time parsing format is used.

    For example, two columns named Timestamp and DateField are available in a row. List Timestamp first to indicate that its value is the one to use as the resultant alarms date/time.

    <datecolumns>Timestamp,DateField</datecolumns>
    

    You also must supply the corresponding format definitions in a <dateformats> tag. If the sample data is:

    2006‑08‑31 13:32:02
    

    This corresponds to a date format string of:

    yyyy‑MM‑dd HH:mm:ss

  • <lifetime>: Specifies the TTL (time to live) for an alarm when it arrives in the Operations Center server:

    <lifetime type="mins|hours|DAYS|weeks|months" relative="create|DATE">amount</lifetime>
    

    Type defines the time measurement. Amount defines the number of time units. The default type is DAYS.

    relative specifies whether the alarm lifetime should be relative to the time the alarm was received and created by the adapter (CREATE) or to the date stamp of the alarm (DATE).

    For more details on the implementation of alarm lifetime, see Section 5.0, Alarm Lifetime.

    An example of defining a 5‑day TTL for alarms:

    <lifetime>5</lifetime>
    

    A 2‑hour lifetime based on when the alarms arrive at the adapter:

    <lifetime type="hours">2</lifetime>
    
  • <multiplex>: Columns containing comma-separated values can be identified by the <multiplex> tag:

    <multiplex [delimiter="delimiter_string"]>comma-separated_list_of_columns</multiplex>
    

    An alarm is generated for each value in the multiplex column.

    Mulitplexing can make hierarchy processing easier when columns are known to contain multiple values, which should be treated distinctly.

    For example, to identify the Departments column as a multiplex column:

    <multiplex>Departments</multiplex>
    

    During content processing, if the value Finance,HR is found for the Departments column, the F/X Monitor generates two alarms for the row. The first alarm contains Department=Finance. The second alarm contains Department=HR.

    If the content of the multiplex columns is not comma-separated the alternative delimiter to be used can be specified using the delimiter argument. This delimiter applies to the *column data* and not the list of multiplex columns themselves, which are always comma-separated!

    For example, to specify that columns Service Groups and Support Centers are multiplex columns and their content is separated by the # character, the following definition should be used:

    <multiplex delimiter="#">Service Groups,Support Centers</multiplex>
    
  • <replacements>: Selectively copies column values based on content matching.:

    <replacements>
    
    <copycol fromcolumn="column_name" tocolumn="column_name">matcher</copycol>
    
    </replacements>
    

    The <replacements> tag must contain the <copycol> tag.

    Matching can be empty, which implies that the copy should always occur.

    For example, to copy the TicketID column to a new column named CheckedTicketID if the content begins with the text TKT and is followed by at least one number:

    <replacements>
    
    <copycol fromcolumn="TicketID" tocolumn="CheckedTicketID">^TKT[0‑9]+$</copycol>
    
    </replacements>
    
  • <severities>: The column values can be matched against severity definitions to provide the generated alarms with severity status:

    <severities>
    
    <default>UNKNOWN</default>
    
    <severity column="Result" condition="OK">pass</severity>
    
    <severity column="Result" condition="CRITICAL">fail</severity>
    
    <severity column="Business_Risk" condition="MINOR">PRODUCTION
    
    <sevRule column="Estimated_Down_Time" condition="MINOR" equation="gt">0</sevRule>
    
    <sevRule column="Estimated_Down_Time" condition="MAJOR" equation="gt">2</sevRule>
    
    </severity>
    
    <severity column="Business_Risk" condition="INFO">Medium
    
    <sevRule equation="script">
    
    var returnType=true;
    
    return returnType;
    
    </sevRule>
    
    </severity>
    
    </severities>
    

    The <severity> tag can contain the <default> tag and multiple embedded <severity> tags, as well as <severity> tags with embedded <severity rules> tags.

    Valid severities are OK, UNKNOWN, MINOR, MAJOR, and CRITICAL.

    Any number of severities can be defined and an optional default severity can be set.

    In the case where more than one severity definition matches, the highest severity match is set as the alarm severity.

    CLEAR is a special delete severity, usually used to close deduplicated alarms generated by use of the uniquecolumns parameter.

    For an additional explanation and usage examples, see More about Severity Rule Tags.

  • <skipcolumns>: (Optional) Skips columns based on the column content:

    <skipcolumns>
    
    <skip column="column_name">matcher</skip>
    
    </skipcolumns>
    

    The <skipcolumns> tag must contain at least one <skip> tag.

    Skipping columns is optionally qualified with a regular expression match. If no regular expression match is defined, then the column is always removed, regardless of content.

    If a column’s content is matched, the line is skipped. It is legal to have multiple skip definitions for the same column.

    For example, to skip the ID column when it contains a number:

    <skipcolumns>
    
    <skip column="ID">[0‑9]+</skip>
    
    </skipcolumns>
    
  • <skiprows>: Skips rows based on matching column values:

    <skiprows>
    
    <skip column="column_name" [invert="yes"|"no"]>matcher</skip>
    
    </skiprows>
    

    The <skiprows> tag must contain at least one <skip> tag.

    Column values can also be marked as inverted, to skip a column if the column value does not match a specified value. This can be useful for enforcing data structure or content.

    For example, to define a policy where rows are skipped if the Status field contains Deleted, or if the Hostname field contains the word Test:

    <skiprows>
    
    <skip column="Status">Deleted</skip>
    
    <skip column="Hostname">.*Test.*</skip>
    
    </skiprows>
    
  • <splits>: Split definitions enable separating complex content in one column into subcomponents in new columns:

    <splits>
    
    <split fromcolumn="column_containing_data_to_split" tocolumns="comma-separated_list_of_columns_to_create" name="name_of_split">regular_expression_group_pattern</split>
    
    </splits>
    

    <splits> must contain <split>.

    The <splits> tag can contain any number of split definitions, but must contain at least one. Split definitions consist of the originating column, the list of columns to split into and the regular expression that defines the splits.

    For example, to split a name into firstname and lastname components:

    <splits>
    
    <split fromcolumn="name" tocolumns="firstname,lastname" name="Name Splitter">(.*) (.*)</split>
    
    </splits>
    

    If data contained in the name column contains the value Joe Bloggs, the split definition creates two new columns: firstname = Joe and lastname = Bloggs.

    Splits can contain any number of components (elements extracted, which are encapsulated in parentheses).

  • <uniquecolumns>: Identifies data from one or more columns that collectively constitutes a unique identifier for the data:

    <uniquecolumns>list_of_columns_that_determine_uniqueness</unique>
    

    This enables differentiating alarm content between being a new alarm and an update to a previous alarm at the adapter level.

    For example, if the column’s hostname and IP address collectively identify a unique entity, then the adapter treats alarms that have identical hostnames and IP address properties as updates to the same alarm in Operations Center:

    <uniquecolumns>Hostname,IPAddress</uniquecolumns>
    

    For more detailed information on alarm uniqueness and deduplication, see Section 6.0, Alarm Unique Key / Deduplication Capability.

    It is possible to list all alarm columns, which can cause an overloaded list of columns. The adapter then generates the key for the alarm based on columns available per alarm.

More about Severity Rule Tags

There are two levels of capability for using the severity rules tags within the F/X parser. The first level is the simple operator tag, allows you to compare an alarm column value with a fixed value (called the pattern). There are four operators:

  • gt: greater than

  • lt: less than

  • eq: equals

  • gt: greater than

If the result of the equation is True, then the severity is set to a specific value. If the value is False, then the normal severity value from the parent tag is used.

In the following example, if the column named Business_Risk contains the text PRODUCTION, then set the alarm condition to MINOR, unless the column Estimated_Down_Time has a numeric value greater than 0, in which case, set the alarm condition to CRITICAL.

  <severities>
    <severity column="Business_Risk" condition="MINOR">PRODUCTION
       <sevRule column="Estimated_Down_Time" condition="CRITICAL" equation="gt">0</sevRule>
    </severity>
 </severities>

These operators can only evaluate against numeric values only. A warning message displays if the column value used for comparison is not numeric. An error occurs if you set the value text of this tag to a nonnumeric character, unless you use the script equation described in the following example.

The tag does not operate against strings, nor does it perform any sorting of string comparisons. This is already performed throughout the adapter’s configuration when building the alarm stream and processing raw data.

However, complex string comparisons still can be performed at this point. Set the “equation” attribute equal to “script”. This enables a developer to create a script fragment (in NOC Script) to process any column within the alarm in any way and make a call to the setSeverity(severity) method. An example:

<severities>
   <severity column="Business_Risk" condition="MINOR">PRODUCTION
      <sevRule equation="script">
         if (alarm.get("Node_Center_Name") == “Dallas”) {
            fxAdapter.setSeverity("MAJOR");
         }
      </sevRule>
   </severity>
</severities>

Multiple Severity Rules

It is possible to use more than one severity rule with a severity tag. In this case, the last severity rule in the list of rules that returns True is used to set the severity. Here is an example:

<severities>
    <severity column="Business_Risk" condition="OK">PRODUCTION
       <sevRule column="Estimated_Down_Time" condition="INFO" equation="lt">3.5</sevRule>
       <sevRule column="Estimated_Up_Time" condition="MINOR" equation="gt">100</sevRule>
       <sevRule column="Standard_Mean_Time" condition="MAJOR" equation="eq">333</sevRule>
       <sevRule column="emergency" condition="CRITICAL" equation="neq">‑1</sevRule>
    </severity>
 </severities>

In this case, if the first, second, and third rules are all triggered, the third rule is used and the severity is set to MAJOR.

Script type rules can be combined with equation style rules.

Delimited Parsers (DelimitedFileParser, TabDelimiterParser, and CSVParser)

All the delimited type parsers share a common configuration parameter set in addition to the standard configuration parameters already described:

  • <columns>: List of column names to use for the delimited content:

    <columns>comma-separated_list_of_columns</columns>
    

    If the main content has more columns than are defined in this tag, the remaining columns are omitted from generated alarms.

    If a <headerrow> tag is defined and found by the parser, the results are used in preference to the values of the <columns> tag. As such, the <columns> tag can be considered as a fail-safe.

    For example, to identify 4 column names:

    <columns>Date,Building,Name,Value</columns>
    
  • <delimiter>: (Required) Character or string used to delimit content:

    <delimiter>delimiter_marker</delimiter>
    

    For TabDelimiterParser and CSVParser, this is already defined.

    For example, use <delimiter>|</delimiter> for content delimited by the | (pipe) character.

  • <headerline>: Identifies the line that contains column names:

    <headerline>line_number</headerline>
    

    The Parser expects the column delimiter to be the same as the delimiter used for the main content.

    For example, to define the first line of a file as the <headerline>:

    <headerline>0</headerline>
    
  • <startline>: The line from which the parser should start to generate alarm content:

    <startline>line_number</startline>
    

    If a <headerline> is defined, the default is set to line 1.

    For example, to set the start line to 2 (the third line of a file):

    <startline>2</startline>
    

XML Parser

(Required) The XML Parser works by identifying target elements within an XML document to reference for the overall parsing requirements. The target elements are identified using XMLPath notation (see http://www.w3.org/TR/xpath for more details and examples of use). XPath syntax enables target elements to be identified by referencing the XML document structure and, optionally, conditional matching of element values and attributes.

Associated processing is encapsulated within containers known as filters. The XMLParser automatically adds all attributes of the matched element as alarm properties. The configuration for each filter can contain any number of XMLParser parameters that enable the incorporation of additional tag values and attribute values into the alarm.

The following lists the required and optional XML Parser tags:

  • <filter>: (Required) Filter definition matches XML content and generates an alarm from each match:

    <filters>
    
    <filter name="filter_name" path="xpath_definition">
    
    </filter>
    
    </filters>
    

    where:

    • Name: A free text name given to the filter that is bound to any generated alarms using the ALARM_XMLFILTERNAME property.

    • Path: An XPath definition.

    The <filters> tag must contain <filter> tags.

    The alarm is automatically populated with the matching node’s value and attributes. Additional configuration parameters determine other properties to add to the alarm.

    The following example illustrates the structure of an XMLParser tag, in this case containing two filters. Each filter can match more than one element in the XML document. Each match generates a single alarm. The (optional) parser parameters add properties to the alarm.

    <parsers>
    <parser>
    <name>PSO Parser</name>
        <class>com.mosol.integration.fx.parsers.XMLParser</class>
        <settings>
          <filters>
            <filter name="PSO Node Numeric Info" path="PerformanceMonitor/Node//PerfNumericInfo">
            <parentname levelsup="1" tofield="Metric" />
            <parentname levelsup="2" tofield="MetricGroup" />
            <parentattribute node="Server" attribute="name" tofield="ServerName" />
              <parentattribute node="Node" attribute="name" tofield="NodeName" />
              <convert field="time" type="DATE" inputformat="long" outputformat="ddMMyyyy hh:mm:ss" tofield="recdate"/>
              <convert field="val" type="NUMERIC" />
            </filter>
          </filters>
        </settings>
    </parser>
    </parsers>
    

    To omit data in long XML strings, such as those generated by RSS feeds, use the double-backslash to exclude tags. For example, assume there are four tags that an XPATH_definition can parse: country, state, city and address. To include only the state, city and address in the Operations Center events/alarms, use the syntax:

    path="//state/city/address"
    

    where the double backslash represents a parent tag that should not be included in the parsed data.

  • <childtag>: Identifies which child tags to incorporate into the alarm:

    <childtag>matcher</childtag>
    

    Stores both the child tag value and all attributes.

    The matcher is a regular expression identifying which child tag (names) to match.

    For example, to match all child tags:

    <childtag>.*</childtag>
    

    To match only the Person or Resource child tags:

    <childtag>(Person|Resource)</childtag>
    
  • <convert>: Enables the conversion of date or numeric fields from the XML String format into a native format and conversion to an alternative format, if required:

    <convert field="field_to_convert" type="DATE|NUMERIC" [inputformat="input_format_type" outputformat="output_format_type"] [tofield="field_to_output_result_to"] />
    

    The converted value can be stored in the original field (default) or in an alternative field using the tofield argument.

    If the type is NUMERIC, then only the field must be identified.

    For example, if the source data is:

    <myval>1.242E+08</myval>
    

    Use the following to convert myval to a numeric and store in the new field numeric_myval:

    <convert type="NUMERIC" field="myval" tofield="numeric_myval" />
    

    If type is DATE, then define the input and output formats using the SimpleDateFormat syntax.

    As a special case, if type is DATE and the source data is a long representation of a date (such as the number of seconds since 1 January 1970), the value long in the inputformat argument instructs the F/X Monitor to convert from a long format.

    For example, assume the sample data is:

    <adate>2006‑08‑31 13:32:02</adate>
    

    This corresponds to the date format string:

    yyyy‑MM‑dd HH:mm:ss

    Assume the requirement is to just show the date in MM/dd/yyyy form. The corresponding <convert ...> tag is:

    <convert field="a_date" type="DATE" inputformat="yyyy‑MM‑dd HH:mm:ss" outputformat="MM/dd/yyyy" tofield="the_day" />
    
  • <parentattribute>: Stores the value of the specified attribute for the named parent node in the specified field name:

    <parentattribute node="name_of_the_parent_tag" attribute="attribute_name_to_collect" tofield="field_name" />
    

    For example, consider the same source XML data again, but this time collect the value of the state attribute of the <Location> tag:

    <Location state="LA">
    
    <Office name="myoffice"/>
    
    </Location>
    

    Use this attribute:

    <parentattribute node="Location" tofield="State" />
    

    This results in adding the following name/value pair to the alarm:

    State=LA
    
  • <parentname>: Stores the name of the parent tag in the specified field name:

    <parentname levelsup="number_of_levels_up" tofield="field_name" />
    

    The number of levels ‘up’ is relative to the matched element.

    For example, consider the following source XML data:

    <Location state="LA">
    
    <Office name="myoffice"/>
    
    </Location>
    

    Assume the filter matched the Office element. To store the tag name of its immediate parent in a field named Folder Type:

    <parentname levelsup="1" tofield="Folder Type" />
    

    This results in adding the following name/value pair to the alarm:

    Folder Type=Location
    
  • <pathattribute>: Enables use of XPath to identify attribute values from the XML document to be added as properties to the alarm:

    <pathattribute path="xpath_definition" [scope="document|FILTER]" attributes="list_of_attributes_to_collect" [tocolumns=l"ist_of_column_names_to_use_for_the attributes"] />
    

    If the provided XPath expression matches multiple nodes, all of the matched attributes are collected. The attributes are distinguished by the .n suffix added to the attribute name in the alarm.

    The XPath expression is processed against the matched element from the filter by default, unless scope is set to document, in which case the scope of the expression is the complete document. Normally the filter (default) scope is required.

    An attribute name or comma-separated list of attributes must be specified.

    (Optional) The tocolumns attribute can be used to provide alternative column names for the attributes (in cases where the original XML attribute names were not desirable).

    Examples:

    • To find the attribute ID from the child tag ticket:

      <pathattribute path="//ticket" attributes="ID" />
      
    • To collect all the user ID attributes of user tags and use the column name username substitution:

      <pathattribute path="some_info/user" attributes="userid" tocolumns="username" />
      
  • <pathvalue>: Enables use of XPath to identify tag values (and the tag’s attribute) from the XML document to be added as properties to the alarm:

    <pathvalue path="xpath_definition" [scope="document|FILTER]" includeattributes="yes|NO" showempty="yes|NO" [tocolumn="column_name"] />
    

    If the provided XPath expression matches multiple nodes, all of the matched tags are collected. The attributes are distinguished by a .n suffix being added to the tag’s column name in the alarm.

    The XPath expression is processed against the matched element from the filter by default, unless scope is set to “Document,” in which case the scope of the expression is the complete document. Normally the filter (default) scope is required.

    The includeattrs attribute specifies whether a matched node’s attributes should be included as alarm properties. The default is no. If attributes are included they are named as:

    <columnname>.<attributename>
    

    The showempty attribute specifies whether to include the node in the alarm is it’s value is null or empty. Default behavior is not to add the property in this case.

    tocolumn identifies the required column name for the tag value, if this is not specified the tag name is used.

    For example, to gather the value of leaf node address:

    <pathvalue path="//address "/>
    

Regular Expression Parser

(Required) The Regular Expression parser is intended for use with nondelimited file content where regular expression matching provides a convenient way to filter content into alarms. The syntax for the <expressions> tag is:

<expressions>
<expression name=" columns="comma-separated_list_of_column_names">matcher</expression>
</expressions>

The <expressions> tag must contain <expression> tags.

Each expression component is matched against a line of the file. If the match passes, then columns are synthesized and the values of the columns are set to the corresponding groups in the match definition.

The first matching expression is applied to a given line. Subsequent expressions are ignored for that line.

For convenience, the name of the matching expression is also added to the alarm as a property named ALARM_MATCHERNAME.

For example, consider a line that has the following format:

140502 SERVER Failed The Server Failed

This can be broken into columns by the following expression definition:

<expressions>
<expression name="My Expression 1" columns="time,component,status,message"> ([0‑9]{6}) (.*) (.*) (.*)</expression>
</expressions>

As with any regular expression, be sure to escape special characters with a backslash. In addition, the following special characters in XML must be replaced with the HTML character entity as follows:

  • For ampersands (&), use &amp;

  • For less-than signs (<), use &lt;

  • For greater-than signs (>), use &gt;

  • For quotation marks (“), use &quot;

  • For apostrophes (‘), use &apos;

Simple Text Parser

The Simple Text parser, as the name implies, has a simple approach to parsing content. Line by line files are processed by comparing the content, to filter in and out expressions that determine which lines constitute alarm content:

  • <severities>: Lines can be matched against severity definitions to provide the generated alarms with severity status:

    <severities>
    
    [<default>condition_as_default</default>]
    
    <severity condition="condition_to_set">matcher</severity>
    
    </severities>
    

    The <severities> tag can contain the <default> and <severity> tags.

    Any number of severities can be defined and an optional default severity can also be set.

    In the case where more than one severity definition matches, the highest severity match is set as the alarm severity.

    Valid severities are OK, UNKNOWN, MINOR, MAJOR, and CRITICAL.

  • <filterin>: Lines are candidates for alarms only if they are filtered in by one or more of the in filter definitions:

    <filterin>
    
    <in>matcher</in>
    
    </filterin>
    

    The <filterin> tag must contain the <in> tag.

    For example, to filter in all lines:

    <filterin>
    
    <in>.*</in>
    
    </filterin>
    
  • <filterout>: Lines can be filtered out (or, not promoted as alarms) by matching an out filter definition:

    <filterout>
    
    <out>matcher</out>
    
    </filterout>
    

    The <filterout> tag must contain the <out> tag.

    Only alarms previously filtered by filterin statements are candidates for filtering out.

    For example, to filter out lines beginning with the word TEST:

    <filterout>
    
    <out>TEST.*</out>
    
    </filterout>