6.6 Configuring Data Synchronization

Sentinel provides the ability to synchronize data to an external database, so that you can use third-party or custom reporting systems to search the data in the external database with more advanced tools than what are provided in Sentinel.

6.6.1 Overview

Sentinel can store the data in an external database by synchronizing a subset of the data that Sentinel gathers.

Sentinel uses the following process to synchronize data:

  1. Sentinel gathers the events from the Event Sources through the Connectors.

  2. Sentinel uses Collectors to normalize the event data.

  3. The normalized event data is then sent to the Sentinel message bus.

  4. The event data is then stored and indexed in the file system in the primary storage.

  5. The data synchronization policies allow events in the primary storage to be copied and stored in PostgreSQL and external SQL databases.

    1. User-defined data synchronization policies synchronize the filtered event data to an external SQL database. For information about the certified databases, see the NetIQ Sentinel Technical Info Website.

    2. Report Data Definitions (RDD) generate system data synchronization policies that are used to copy event data into tables in the internal PostgreSQL database. These data synchronization policies cannot be edited or deleted. Reports that rely on an RDD will search internal database tables for events instead of the primary storage. These kinds of reports search internal tables instead of the event store because they utilize more complex SQL SELECT statements that need to join event data to the data in other tables in the internal database.

Figure 6-1 Data Synchronization

Sentinel allows you to partition tables if they are in the internal PostgreSQL database. When you choose to partition a table in the internal PostgreSQL database, a new table partition is created for each days worth of data.

Partitions are only used with RDD data sync policies. Partitioning has advantages and disadvantages:

Advantages

  • If a retention period is in force, old data can be deleted quickly. When data has aged, it is much quicker to drop a partition than it is to delete individual table records.

  • Reports that query on the event time field might be quicker, because it is only necessary to search the partitions that have the specified event times.

Disadvantages

  • Reports that do not query on event time might be slower where there are multiple of partitions, because every partition must be searched.

  • Each partition causes one or more schema items to be created and managed by the database system. If there is no retention period, the number of partitions just keeps growing.

6.6.2 Creating Data Synchronization Policies

When Sentinel syncs data to an external database, it is not as fast when it writes to the file system. Therefore, you need to ensure that you use filters on the data sync policies to synchronize only the most important data. Consider the following factors based on your business needs for data sync policies:

  • The CPU, RAM, and disk capacity of the Sentinel system

  • Number of EPS scaled per system

  • Number of searches and reports running on a Sentinel system

  • Filters added to the data sync policies

Populating IP addresses in Human Readable Format

By default, Sentinel populates IP address fields in hexadecimal format for efficiency reasons. You can choose to populate the IP address fields in human readable format automatically, by performing the following steps:

  1. Log on to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file and set the datasync.saveIPinDottedNotation property to true.

  3. Restart the Sentinel server.

Enabling SSL Communication for Data Synchronization

You can establish an SSL connection to synchronize data with external databases. Sentinel does not perform certificate validation or authentication.

To enable SSL communication, performing the following steps:

  1. Log in to the Sentinel server as the novell user.

  2. Open the /etc/opt/novell/sentinel/config/configuration.properties file.

  3. If the jsse.enableCBCProtection property is not listed, add this property and set it to false as follows:

    jsse.enableCBCProtection=false
    
  4. Open the /etc/opt/novell/sentinel/config/databasePlatforms.xml file.

  5. Identify the database platform for which you need to enable SSL connection.

  6. Set the JDBC property as follows:

    For MSSQL: Set the SSL property to require as follows:

    <JDBCProperties>
    
    	<Property name="ssl" value="require"/>
    
    </JDBCProperties>
    

    For PostgreSQL: Set the SSLOFF property to false as follows

    <JDBCProperties>
    
    			<Property name="ssloff" value="false"/>
    
    </JDBCProperties>
    

    For Oracle: Set the SSLOFF property to false as follows:

    <JDBCProperties>
    
    			<Property name="ssloff" value="false"/>
    
    </JDBCProperties>
    
  7. Restart the Sentinel server.

Creating a Data Synchronization Policy

To create a data synchronization policy:

  1. Log in to the Sentinel Web interface as a user in the administrator role.

  2. Click Storage > Data Synchronization.

  3. Click Create to create a new data synchronization policy.

  4. Use the following information to create the data synchronization policy:

    Filter query: Select a saved filter to use in the data synchronization policy.

    This filter determines which events are stored in the external database. For more information, see Configuring Filters in the NetIQ Sentinel User Guide.

    Policy name: Specify a name for the data synchronization policy.

    Retention period: Specify how many days to retain the events in the external database.

    Start data synchronization time: Specify when to start synchronizing events to the external database.

    Batch size: Specify how many events are sent to the external database at once.

    Sleep period: Specify the length of time that the data synchronization process sleeps before checking to see if there are more events to process.

    Schedule: Select when the data is synchronized to the external database.

    • All the time: Synchronizes events to the external database constantly.

    • Custom: Allows you to configure specific time periods to perform data synchronization so that it does not occur when the system is busy.

      If you select Custom, specify the following information to set the custom synchronization time:

      • Day of the Week: Select the day of the week, or select Everyday.

      • Start time: Specify the time to start the synchronization process. You can enter 24:00 hour time and it is converted to 12:00 hour time.

      • Duration: Specify the synchronization period in minutes.

    If you do not see the data in the database tables immediately, you need to wait for the next synchronization cycle.

  5. Use the following information to define the connection to the external database:

    Database type: Select the type of external database.

    Host name: Specify the host name of the server where the external database is installed.

    Port: Specify the port used to connect to the external database.

    User name: Specify the name of the user that authenticates to the external database.

    Password: Specify the password of the database user.

    Database: Specify a unique name for the external database.

    Field Mapping: Allows you to map fields in the event to fields in the external database.

  6. Click Save to create the data synchronization policy.

Creating a Table for Event Data Synchronization

  1. Complete Step 1 through Step 5 in Section 6.6.2, Creating Data Synchronization Policies.

  2. Click Field Mapping.

  3. Select Create table.

  4. Use the following information to create the table:

    Table name: Specify a name for the table.

    Table Space (Optional): Specify a tablespace for the table.

    Index Space (Optional): Specify a tablespace for the index.

    Summarize Events: Select this option if you want a summary of events during a specific period.

    Summary Period (Minutes): If you selected Summarize Events, you must specify the amount of time in minutes to summarize events.

  5. Map the fields in the table to the desired fields.

  6. Click Create Table.

  7. Click Save.

Using an Existing Table for Event Data Synchronization

NOTE:Starting from Sentinel 7.0.1 version, the InitiatiorServiceName (sp) and TargetServiceName (dp) fields size is increased from 32 to 256 characters to accommodate more characters in these fields. If you have created a Data Sync policy in Sentinel 7.0 that synchronizes either or both of the event fields, you need to modify the target column size in the external database table to reflect the increased size of the fields.

  1. Complete Step 1 through Step 5 in Section 6.6.2, Creating Data Synchronization Policies.

  2. Click Field Mapping.

  3. Select Select existing table.

  4. Browse to a select an existing table you want to use, then click OK.

  5. (Optional) Select the Summarize Events option if you want a summary of events during a specific period.

  6. (Optional) If you selected Summarize Events, specify the amount of time in minutes to summarize events.

  7. Change the field mappings for the desired fields.

  8. Click Save.

6.6.3 Data Synchronization

You can edit, delete, and view the status of each data synchronization policy you create on the Data Synchronization page. If your policy is a custom synchronization policy and you perform a resynchronization, the data synchronizes during the next synchronization cycle.