A.3.1 Initial Setup

Configure the machine hardware, network hardware, storage hardware, operating systems, user accounts, and other basic system resources per the documented requirements for Sentinel and local customer requirements. Test the systems to ensure proper function and stability.

A.3.2 Shared Storage Setup

Set up your shared storage and make sure that you can mount it on each cluster node. If you are using FibreChannel and a SAN, this may involve physical connections and other configuration. The shared storage will be used to hold Sentinel's databases and event data, so must be sized accordingly for the customer environment based on the expected event rate and data retention policies.

A typical implementation might use a fast SAN attached using FibreChannel to all the cluster nodes, with a large RAID array to store the local event data. A separate NAS or iSCSI node might be used for the slower secondary storage. As long as the cluster node can mount the primary storage as a normal block device, it can be used by the solution. The secondary storage can also be mounted as a block device, or could be an NFS or CIFS volume.

The exemplary solution uses iSCSI Targets configured on a SUSE Linux VM. The VM is storage03 as listed in Initial Setup. iSCSI devices can be created using any file or block device, but for simplicity here we will use a file that we create for this purpose.

Connect to storage03 and start a console session. Use the dd command to create a blank file of any desired size for Sentinel primary storage:

dd if=/dev/zero of=/localdata count=10240000 bs=1024

In this case, we create a 10GB file filled with zeros (copied from the /dev/zero pseudo-device). See the info or man page for dd for details on the command-line options. For example, to create different-sized "disks". The iSCSI Target treats this file as if it were a disk; you could of course use an actual disk if you prefer.

Repeat this procedure to create a file for secondary storage:

dd if=/dev/zero of=/networkdata count=10240000 bs=1024

For this example we use two files ("disks") of the same size and performance characteristics. For a production deployment, you might put the primary storage on a fast SAN and the secondary storage on a slower iSCSI, NFS, or CIFS volume.

A.3.3 Sentinel Installation

There are two options to install Sentinel: install every part of Sentinel onto the shared storage (using the --location option to redirect the Sentinel install to wherever you have mounted the shared storage) or only the variable application data on the shared storage.

In this exemplary solution, we will follow the latter approach and install Sentinel to each cluster node that can host it. The first time Sentinel is installed we will do a complete installation including the application binaries, configuration, and all the data stores. Subsequent installations on the other cluster nodes will only install the application, and will assume that the actual Sentinel data will be available some time later (e.g. once the shared storage is mounted).

In this exemplary solution we install (for traditional HA installations) or configure (for HA appliance installations) Sentinel to each cluster node, storing only the variable application data on shared storage. This keeps the application binaries and configuration in standard locations, allows us to verify the RPMs, and also allows us to support warm patching in certain scenarios.

A.3.4 Cluster Installation

You need to install the cluster software only for traditional HA installations. The Sentinel HA appliance includes the cluster software and does not require manual installation.

Install the cluster software on each node, and register each cluster node with the cluster manager. Procedures to do so will vary depending on the cluster implementation, but at the end of the process each cluster node should show up in the cluster management console.

For our exemplary solution, we will set up SUSE Linux High Availability Extension and overlay that with Sentinel-specific Resource Agents:

If you do not use the OCF Resource Agent to monitor Sentinel, you will likely have to develop a similar monitoring solution for the local cluster environment. The OCF Resource Agent for Sentinel is a simple shell script that runs a variety of checks to verify if Sentinel is functional. If you wish to develop your own, you should examine the existing Resource Agent for examples (the Resource Agent is stored in the Sentinelha.rpm in the Sentinel download package.)

There are many different ways in which a SLE HAE cluster can be configured, but we will select options that keep it fairly simple. The first step is to install the core SLE HAE software; the process is fully detailed in the SLE HAE Documentation. For information about installing SLES add-ons, see the Deployment Guide.

You must install the SLE HAE on all cluster nodes, node01 and node02 in our example. The add-on will install the core cluster management and communications software, as well as many Resource Agents that are used to monitor cluster resources.

Once the cluster software has been installed, an additional RPM should be installed to provide the additional Sentinel-specific cluster Resource Agents. The HA RPM can be found in the novell-Sentinelha-<Sentinel_version>*.rpm stored in the normal Sentinel download, which you unpacked to install the product.

On each cluster node, copy the novell-Sentinelha-<Sentinel_version>*.rpm into the /tmp directory, then:

cd /tmp

rpm -i novell-Sentinelha-<Sentinel_version>*.rpm

A.3.5 Cluster Configuration

You must configure the cluster software to register each cluster node as a member of the cluster. As part of this configuration, you can also set up fencing and STONITH resources to ensure cluster consistency.

In our exemplary solution, we basically use the simplest configuration without additional redundancy or other advanced features. We also use a unicast address (instead of the preferred multicast address) because it requires less interaction with network administrators, and is sufficient for testing purposes. We also set up a simple SBD-based fencing resource.

The exemplary solution will use private IP addresses for internal cluster communications, and will use unicast to minimize the need to request a multicast address from a network administrator. The solution will also use an iSCSI Target configured on the same SUSE Linux VM that hosts the shared storage to serve as an SBD device for fencing purposes. As before, iSCSI devices can be created using any file or block device, but for simplicity here we will use a file that we create for this purpose.

The following configuration steps are very similar to those in Shared Storage Setup:

SBD Setup

Connect to storage03 and start a console session. Use the dd command to create a blank file of any desired size:

dd if=/dev/zero of=/sbd count=1024 bs=1024

In this case, we create a 1MB file filled with zeros (copied from the /dev/zero pseudo-device).

Configure that file as an iSCSI Target:

This procedure should expose an iSCSI Target for the SBD device on the server at IP address 10.0.0.3 (storage03).

Node Configuration

Connect to a cluster node (node01) and open a console:

Connect to each additional cluster node (node02) and open a console:

In some circumstances the cluster communications do not initialize correctly. If the cluster does not start (the openais service fails to start):

In some cases, the script might fail because the xinetd service does not properly add the new csync2 service. This service is required so that the other node can sync the cluster configuration files down to this node. If you see errors like csync2 run failed, you may have this problem. To fix this, execute: kill -HUP `cat /var/run/xinetd.init.pid and then re-run the sleha-join script.

At this point you should be able to run crm_mon on each cluster node and see that the cluster is running properly. Alternatively you can use 'hawk', the web console - the default login credentials are 'hacluster / linux'.

There are two additional parameters we need to tweak for this example; whether these will apply to a customer's production cluster will depend on its configuration:

A.3.6 Resource Configuration

As mentioned in the Cluster Installation, this solution provides an OCF Resource Agent to monitor the core services under SLE HAE, and you can create alternatives if desired. The software also depends on several other resources, for which Resource Agents are provided by default with SLE HAE. If you do not want to use SLE HAE, you need to monitor these additional resources using some other technology:

There are additional resources, such as MongoDB used for Security Intelligence and the ActiveMQ message bus; for now at least these are monitored as part of the core services.

The exemplary solution uses simple versions of the required resources, such as the simple Filesystem Resource Agent. You can choose to use more sophisticated cluster resources like cLVM (a logical-volume version of the filesystem) if required.

The exemplary solution provides a crm script to aid in cluster configuration. The script pulls relevant configuration variables from the unattended setup file generated as part of the Sentinel installation. If you did not generate the setup file, or you wish to change the configuration of the resources, you can edit the script accordingly.

Connect to the original node on which you installed Sentinel (this must be the node on which you ran the full Sentinel install) and do the following (<SHARED1> is the shared volume you created above):

mount /dev/<SHARED1> /var/opt/novell

cd /usr/lib/ocf/resource.d/novell

./install-resources.sh

There might be issues with the new resources coming up in the cluster; run /etc/rc.d/openais restart on node02 if you experience this issue.

The install-resources.sh script will prompt you for a couple values, namely the virtual IP that you would like people to use to access Sentinel and the device name of the shared storage, and then will auto-create the required cluster resources. Note that the script requires the shared volume to already be mounted, and also requires the unattended installation file which was created during Sentinel install to be present (/tmp/install.props). You do not need to run this script on any but the first installed node; all relevant config files will be automatically synced to the other nodes.

If the customer environment varies from this exemplary solution, you can edit the resources.cli file (in the same directory) and modify the primitives definitions from there. For example, the exemplary solution uses a simple Filesystem resource; you may wish to use a more cluster-aware cLVM resource.

After running the shell script, you can issue a crm status command and the output should look like this:

crm status

Last updated: Thu Jul 26 16:34:34 2012
Last change: Thu Jul 26 16:28:52 2012 by hacluster via crmd on node01
Stack: openais
Current DC: node01 - partition with quorum
Version: 1.1.6-b988976485d15cb702c9307df55512d323831a5e
2 Nodes configured, 2 expected votes
5 Resources configured.

Online: [ node01, node02 ]
stonith-sbd    (stonith:external/sbd):    Started node01
 Resource Group: sentinelgrp
     sentinelip    (ocf::heartbeat:IPaddr2):    Started node01
     sentinelfs    (ocf::heartbeat:Filesystem):    Started node01
     sentineldb    (ocf::novell:pgsql):    Started node01
     sentinelserver    (ocf::novell:sentinel):    Started node01

At this point the relevant Sentinel resources should be configured in the cluster. You can examine how they are configured and grouped in the cluster management tool, for example by running crm status.

A.3.7 Secondary Storage Configuration

As the final step in this process, configure secondary storage so that Sentinel can migrate event partitions to less-expensive storage. This is optional, and in fact the secondary storage need not be made highly-available in the same way that the rest of the system has been - you can use any directory (mounted from a SAN or not), or NFS or CIFS volume.

In the Sentinel web console, in the top menu bar, click Storage, then select Configuration, then select one of the radio buttons under Secondary storage not configured to set this up.

The exemplary solution will use a simple iSCSI Target as a network shared storage location, in much the same configuration as the primary storage. In production implementations, these would likely be different storage technologies.

Use the following procedure to configure the secondary storage for use by Sentinel:

The exemplary solution uses simple versions of the required resources, such as the simple Filesystem Resource Agent - customers can choose to use more sophisticated cluster resources like cLVM (a logical-volume version of the filesystem) if they wish.