For scalable and distributed indexing of events, you must install Elasticsearch in cluster mode. The Elasticsearch cluster you install for Sentinel must be used to index only Sentinel data.
NOTE:This section provides information about installing and configuring both Elasticsearch 2.3.2 and Elasticsearch 5.0. Depending on your Sentinel version, you must install the appropriate, certified version of Elasticsearch. For more information about the certified version of Elasticsearch for your Sentinel, see the Technical Information for Sentinel page.
Complete the following prerequisites before you install Elasticsearch:
Set the virtual memory by adding the following property in the /etc/sysctl.conf file:
vm.max_map_count=262144
Set the file descriptors by adding the following properties in the /etc/security/limits.conf file:
elasticsearch hard nofile 65536
elasticsearch soft nofile 65536
You must install Elasticsearch and the required plug-ins on each node in the Elasticsearch cluster.
To install and configure Elasticsearch:
Install the JDK version supported by Elasticsearch.
Download the certified version of Elasticsearch RPM. For information about the certified version of Elasticsearch and the download URL, see the Technical Information for Sentinel page.
Install Elasticsearch:
rpm -i elasticsearch-<version>.rpm
Complete the tasks as mentioned in the RPM post-installation instructions.
Ensure that the Elasticsearch user has access to Java by running the following command:
sudo –u elasticsearch java –version
Configure the /etc/elasticsearch/elasticsearch.yml file by updating or adding the following information:
|
Property and Value |
Notes |
|---|---|
|
cluster.name: <Elasticsearch _cluster_name> |
The cluster name that you specify must be same for all the nodes. |
|
node.name: <node_name> |
The node name must be unique for each node. |
|
network.host: _<networkInterface>:ipv4_ |
|
|
discovery.zen.ping.unicast.hosts: ["<FQDN of elasticsearch node1>", "<FQDN of elasticsearch node2>", and so on"] |
|
|
thread_pool.bulk.queue_size: 300 |
|
|
thread_pool.search.queue_size: 10000 |
Once the search queue size reaches its limit, Elasticsearch discards any pending search requests in queue. You can increase the search queue size based on the below calculation:threadpool.search.queue_size = Average number of widget queries per user for a dashboard x number of shards (per day index) x number of days (search duration) |
|
index.codec: best_compression |
|
|
path.data: ["/<es1>", "/<es2>"] |
Spread data across multiple independent disks or locations to reduce the disk I/O. Configure multiple paths for storing Elasticsearch data. For example /es1, /es2, and so on. For best performance and manageability, mount each path to a separate physical disk (JBOD). |
Update the default Elasticsearch heap size in the /etc/elasticsearch/jvm.options file.
The heap size must be 50% of the server memory. For example, on a 24 GB Elasticsearch node, allocate 12 GB as the heap size for optimal performance.
Start Elasticsearch:
/etc/init.d/elasticsearch start
Repeat all of the above steps on each node of the Elasticsearch cluster.
You must install Elasticsearch and the required plug-ins on each node in the Elasticsearch cluster.
To install and configure Elasticsearch:
Install the JDK version supported by Elasticsearch.
Download the certified version of Elasticsearch RPM. For information about the certified version of Elasticsearch and the download URL, see the Technical Information for Sentinel page.
Install Elasticsearch:
rpm -i elasticsearch-<version>.rpm
Complete the tasks as mentioned in the RPM post-installation instructions.
Ensure that the Elasticsearch user has access to Java by running the following command:
sudo –u elasticsearch java –version
Configure the /etc/elasticsearch/elasticsearch.yml file by updating or adding the following information:
|
Property and Value |
Notes |
|---|---|
|
cluster.name: <Elasticsearch _cluster_name> |
The cluster name that you specify must be same for all the nodes. |
|
node.name: <node_name> |
The node name must be unique for each node. |
|
network.host: _<networkInterface>:ipv4_ |
|
|
discovery.zen.ping.unicast.hosts: ["<FQDN of elasticsearch node1>", "<FQDN of elasticsearch node2>", and so on"] |
|
|
bootstrap.mlockall: true |
|
|
threadpool.bulk.queue_size: 300 |
|
|
threadpool.search.queue_size: 10000 |
Once the search queue size reaches its limit, Elasticsearch discards any pending search requests in queue. You can increase the search queue size based on the below calculation:threadpool.search.queue_size = Average number of widget queries per user for a dashboard x number of shards (per day index) x number of days (search duration) |
|
index.codec: best_compression |
|
|
path.data: ["/<es1>", "/<es2>"] |
Spread data across multiple independent disks or locations to reduce the disk I/O. Configure multiple paths for storing Elasticsearch data. For example /es1, /es2, and so on. For best performance and manageability, mount each path to a separate physical disk (JBOD). |
|
index.merge.scheduler.max_thread_count: 3 |
Merges running on separate threads in parallel optimize the writing speed on Elasticsearch. |
Update the default Elasticsearch heap size by modifying the ES_HEAP_SIZE property in the /etc/sysconfig/elasticsearch file.
The heap size must be 50% of the server memory. For example, on a 24 GB Elasticsearch node, allocate 12 GB to the ES_HEAP_SIZE property for optimal performance.
Restart Elasticsearch:
/etc/init.d/elasticsearch restart
Download and install the Delete-By-Query plug-in for the data retention policies to effectively delete indexed data when the retention policies are applied.
In highly secured environments where you cannot download files directly to the server, you must manually download the plug-in on a computer that has access to internet, copy the file to Elasticsearch nodes, and then install the plug-in.
For information about installing the Delete-By-Query plug-in, see the Elasticsearch documentation.
(Optional) Install the Elasticsearch Head plug-in to do basic monitoring of the Elasticsearch cluster.
For information about installing the Elasticsearch head plug-in, see the Elasticsearch Head plug-in documentation.
Repeat all of the above steps on each node of the Elasticsearch cluster.