21.2 Upgrading CDF

You can upgrade CDF in the following ways:

21.2.1 Manual Upgrade

This section provides information about manually upgrading CDF.

Prerequisites

  • Ensure that you have downloaded the Identity Intelligence package on all the CDF nodes. You need the following files in the Identity Intelligence package to upgrade CDF:

    cdf-xxxx.xx.xxxx

  • Ensure that you have minimum 50 GB free space in master node and 30 GB free space in worker node.

  • Create a backup directory with minimum 30 GB of space on every node of your cluster:

    mkdir /tmp/upgrade-backup

    If you do not create a backup directory, the backup files will be stored in the default location (\tmp).

  • Install socat and container-selinux packages on all nodes in the cluster by using the command:

    yum install <package_name>

  • Ensure that you have appropriate permission to restart nodes. You might need to restart nodes if there is an issue during upgrade.

  • Ensure that all nodes are currently running:

    kubectl get nodes

  • Ensure that all pods are currently running:

    <K8S_HOME>/bin/kube-status.sh

Upgrading CDF

  1. Run the following commands on each node:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-xxxx.xx.xxxx

    ./upgrade.sh -t <path_to_backup_directory> -i

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.05.x.x.x.x

    ./upgrade.sh -t /tmp/upgrade-backup -i

    NOTE:If you do not specify -t <path>, the backup files will be stored in the default location (\tmp).

  2. Run the following commands on one of the master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-xxxx.xx.xxxx

    ./upgrade.sh -u

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.05.xxxx

    ./upgrade.sh -u

  3. (Optional) Clean unused docker daemon images by executing the following command on all the worker and master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-xxxx.xx.x.x.x.x

    ./upgrade.sh -c

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.05.x.x.x.x

    ./upgrade.sh -c

  4. Ensure that upgrade is successful by verifying the following on all the nodes:

    • Check the CDF version by executing the command:

      cat <K8S_HOME>/version.txt

    • Check the current status of CDF pods by executing the command:

      <K8S_HOME>/bin/kube-status.sh

      NOTE:If the pods are not in running state, execute the following command to recreate main cluster services:

      <K8S_HOME>/bin/kube-restart.sh

Troubleshooting

This section provides workaround for the following problems during CDF upgrade:

  • If any of the upgrade process fails:

    1. Ensure that kubelet is running by executing the command:

      kubectl get pod -all-namespaces

    2. Rerun the upgrade command.

  • If upgrade process times out, perform the following:

    1. Restart the node. For more information, see Restarting Nodes in the Cluster.

    2. Ensure that all pods are in the running state:

      <K8S_HOME>/bin/kube-status.sh

    3. Rerun the upgrade command.

21.2.2 Automated Upgrade

Automatic upgrade allows you to upgrade CDF from any host (known as the upgrade manager). The upgrade manager can be one of the following:

  • One of the cluster nodes

  • A host outside the cluster in a secure network location

The CDF automated upgrade is run with a single command and requires no interaction until completion of each phase. Typically, the upgrade process takes around 1 hour for a 3x3 cluster.

During the auto-upgrade process:

  • An auto-upgrade directory /tmp/autoUpgrade will be auto generated on the upgrade manager. It will store the upgrade process steps and logs.

  • A backup directory /tmp/CDF_xxxx_upgrade will be auto generated on every node. (approximate size 1.7 GB)

  • A working directory will be auto generated on the upgrade manager and every node at the location provided by the -d parameter. The upgrade package will be copied to this directory. (approximate size 9 GB). The directory will be automatically deleted after the upgrade.

    NOTE:The working directory can be created manually on upgrade manager and every node and then passed as -d parameter to the auto-upgrade script. If you are a non-root user on the nodes inside the cluster, make sure you have permission to this directory.

Prerequisite

  • Ensure that you have downloaded the Identity Intelligence installer package to the download directory (<download_directory>) on the upgrade manager. Verify that you have CDF xxxx.xx upgrade packages in the following location:

    • {download-directory}/identityintelligence-x.x.x.x/installers/cdf-xxxx.xx-x.x.x.x/autoUpgrade.sh

  • Install socat and container-selinux packages on all nodes in the cluster by using the command:

    yum install <package_name>

  • Configure passwordless SSH communication between the upgrade manager and all the nodes in the cluster as follows:

    1. Run the following command on the upgrade manager to generate key pair:

      ssh-keygen -t rsa

    2. Run the following command on the upgrade manager to copy the generated public key to every node in your cluster:

      ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node_fqdn_or_ip>

Upgrading CDF

  1. Change to the directory where you have downloaded the latest CDF package:

    cd /{download-directory}/identityintelligence-x.x.x.x/installers/cdf-xxxx.xx.xxxx/

  2. Run the following command for automatic upgrade:

    ./autoUpgrade.sh -d /path/to/workinig_directory -n {any_cluster_node_adress_or_ip}

    Example:

    ./autoUpgrade.sh -d /tmp/upgrade -n pueas-ansi-node1.swinfra.net

  3. To ensure that the upgrade is successful, verify the following on all nodes:

    • Check the CDF version by executing the command:

      cat <K8S_HOME>/version.txt

    • Check the current status of CDF pods by executing the command:

      <K8S_HOME>/bin/kube-status.sh

      NOTE:If the pods are not in the running state, execute the following command to recreate main cluster services:

      <K8S_HOME>/bin/kube-restart.sh

  4. Delete the auto-upgrade temporary directory and backup directory from the upgrade manager.

    The auto-upgrade temporary directory contains the upgrade steps and logs. If you want to upgrade another cluster from the same upgrade manager, remove that directory.

    rm -rf /tmp/autoUpgrade

    rm -rf /tmp/CDF_xxxx_upgrade

Troubleshooting

  • If the automatic upgrade fails, run autoUpgrade.sh again. The process might take several attempts to succeed.

  • In some cases, the automatic upgrade might return an error message about the upgrade process still running and the existence of a *.lock file which prevents autoupgrade.sh to continue. This file is automatically deleted in a few minutes. Alternatively, you can manually delete this file. Once the file is deleted either automatically or manually, run autoUpgrade.sh again.

  • If the automated upgrade process is still unsuccessful, continue the process on the failed node by using the Manual Upgrade procedure.