You can migrate event data and raw data from the source server to the scalable messaging system by using the scalablestorage_data_uploader.sh script. Run the script individually for event data and raw data. The script uploads the data to the Kafka topics. The script transfers data for the date range (from and to) you specify. When you run the script, it displays the mandatory and optional parameters you should specify to initiate the data upload.
You can specify customizations such as compressing data during the upload, sending data in batches, and so on. To specify these customizations, create a properties file and add the required properties in key-value format. For example, you can add properties as follows:
compression.type=lz4
batch.size=20000
NOTE:For information about Kafka properties, see Kafka documentation. Set the properties and their values at your discretion because the script does not validate these properties.
Ensure that the Sentinel server is able to resolve all Kafka broker hostnames to valid IP addresses for the entire Kafka cluster. If DNS is not setup to enable this, add the Kafka broker hostnames to the /etc/hosts file of the Sentinel server. The script must be run as novell user. Therefore, ensure that the data directories and any files you specify have appropriate permissions for novell user. By default, the script uploads data from primary storage. If you want to upload data from secondary storage, specify the appropriate path for secondary storage when running the script.
To upload event data and raw data:
Log in to the Sentinel server as the novell user.
Run the following script:
/opt/novell/sentinel/bin/scalablestorage_data_uploader.sh
Follow the on-screen instructions and run the script again with the required parameters.
The uploaded event and raw data will have the retention period as set in the target server.
Once the data upload is done, the script records the status such as partitions uploaded successfully, partitions failed to upload, number of events uploaded, and so on. For partitions with previous day and current day’s date, the data transfer status will show IN_PROGRESS considering events that may come in late.
Run the script again in scenarios where the data upload did not complete successfully or where the data upload status for partitions still indicate IN_PROGRESS. When you re-run the script, it first checks the status file to understand the partitions that were already uploaded and then continues to upload only the remaining ones. The script maintains the logs in the /var/opt/novell/sentinel/log/scalablestorage_data_uploader.log directory for troubleshooting purposes.