21.3 Upgrading CDF

Upgrading CDF has 2 phases: upgrading from CDF 2019.05 to CDF 2019.08 and upgrading from CDF 2019.08 to 2020.02.

You can upgrade CDF in the following ways:

21.3.1 Manual Upgrade

This section provides information about upgrading CDF.

Prerequisites

  • Ensure that you have downloaded the Identity Intelligence package on all the CDF nodes. You need the following files in Identity Intelligence package for upgrading CDF:

    • cdf-2020.02.xxxx

    • cdf-upgrade-2019.08.x.x.x.x

  • Ensure that you have minimum 50 GB free space in master node and 30 GB free space in worker node.

  • Create a backup directory with minimum 30 GB of space on every node of your cluster:

    mkdir /tmp/upgrade-backup

    If you do not create a backup directory, the backup files will be stores in the default location (\tmp).

  • Install socat and container-selinux packages on all nodes in the cluster by using the command:

    yum install <package_name>

  • Ensure that you have appropriate permission to restart nodes. You may need to restart nodes if there is an issue during upgrade.

  • Ensure that all nodes are currently running:

    kubectl get nodes

  • Ensure that all pods are currently running:

    <K8S_HOME>/bin/kube-status.sh

Upgrading CDF from CDF 2019.05 to CDF 2019.08

  1. Run the following commands on each node:

    cd <download_directory>/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -t <path_to_backup_directory> -i

    Example:

    cd /opt/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -t /tmp/upgrade-backup -i

    NOTE:If you do not specify -t <path>, the backup files will be stored in the default location (\tmp).

  2. Run the following commands on one of the master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -u

    Example:

    cd /opt/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -u

  3. (Optional) Clean unused docker daemon images by executing the following command on all the worker and master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -c

    Example:

    cd /opt/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.x.x.x.x

    ./upgrade.sh -c

  4. Ensure that upgrade is successful by verifying the following on all the nodes:

    • Check the CDF version by executing the command:

      cat <K8S_HOME>/version.txt

    • Check the current status of CDF pods by executing the command:

      <K8S_HOME>/bin/kube-status.sh

      NOTE:If the pods are not in running state, execute the following command to recreate main cluster services:

      <K8s_HOME>/bin/kube-restart.sh

Upgrading CDF from CDF 2019.08 to CDF 2020.02

  1. Run the following commands on each node:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -t <path_to_backup_directory> -i

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -t /tmp/upgrade-backup -i

    NOTE:If you do not specify -t <path>, the backup files will be stored in the default location (\tmp).

  2. Run the following commands on one of the master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -u

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -u

  3. (Optional) Clean unused docker daemon images by executing the following command on all the worker and master nodes:

    cd <download_directory>/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -c

    Example:

    cd /opt/identityintelligence-x.x.x.x/installers/cdf-2020.02.x.x.x.x

    ./upgrade.sh -c

  4. Ensure that upgrade is successful by verifying the following on all the nodes:

    • Check the CDF version by executing the command:

      cat <K8S_HOME>/version.txt

    • Check the current status of CDF pods by executing the command:

      <K8S_HOME>/bin/kube-status.sh

      NOTE:If the pods are not in running state, execute the following command to recreate main cluster services:

      <K8s_HOME>/bin/kube-restart.sh

Troubleshooting

This section provides workaround for the following problems during CDF upgrade:

  • If any of the upgrade process fails:

    1. Ensure that kubelet is running by executing the command:

      kubectl get pod -all-namespaces

    2. Rerun the upgrade command.

  • If upgrade process timeouts, perform the following:

    1. Restart the node. For steps to restart the node, see Restarting Nodes in the Cluster.

    2. Ensure that all the pods are in running state:

      <K8S_HOME>/bin/kube-status.sh

    3. Rerun the upgrade command.

21.3.2 Automated Upgrade

The CDF automated upgrade is run with a single command and requires no interaction until completion of each phase. Typically, the upgrade process takes around 1 hour for a 3x3 cluster.

Automatic upgrade allows you to upgrade CDF from any host (known as the upgrade manager). The upgrade manager can be one of the following:

  • One of the cluster nodes

  • A host outside the cluster in a secure network location

There are 4 directories involved in the auto-upgrade process:

  • An auto-upgrade directory /tmp/autoUpgrade will be auto generated on the upgrade manager. It will store the upgrade process steps and logs.

  • A backup directory /tmp/CDF_201905_upgrade will be auto generated on every node. (approximate size 1.5 GB)

  • A backup directory /tmp/CDF_201908_upgrade will be auto generated on every node. (approximate size 1.7 GB)

  • A working directory will be auto generated on the upgrade manager and every node at the location provided by the -d parameter The upgrade package will be copied to this directory. (approximate size 9 GB). The directory will be automatically deleted after the upgrade.

    NOTE:The working directory can be created manually on upgrade manager and every node and then passed as -d parameter to the auto-upgrade script. If you are a non-root user on the nodes inside the cluster, make sure you have permission to this directory.

Prerequisite

  • Ensure that you have downloaded the Identity Intelligence installer package to the download directory (<download_directory>) on the upgrade manager. Verify that you have CDF 2019.08 and CDF 2020.02 upgrade packages in the following location:

    • {download-directory}/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.00134-2.2.0.2/autoUpgrade.sh

    • {download-directory}/identityintelligence-x.x.x.x/installers/cdf-2020.02.00120-2.2.0.2/autoUpgrade.sh

  • Install socat and container-selinux packages on all nodes in the cluster by using the command:

    yum install <package_name>

  • Configure passwordless SSH communication between the upgrade manager and all the nodes in the cluster as follows:

    1. Run the following command on the upgrade manager to generate key pair:

      ssh-keygen -t rsa

    2. Run the following command on the upgrade manager to copy the generated public key to every node of your cluster:

      ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node_fqdn_or_ip>

Upgrading from CDF 2019.05 to CDF 2019.08

  1. Change to the directory where you have downloaded the CDF 2019.08 package:

    cd /{download-directory}/identityintelligence-x.x.x.x/upgrade/cdf-upgrade-2019.08.00134-2.2.0.2/

  2. Run the following command for automatic upgrade:

    ./autoUpgrade.sh -d /path/to/workinig_directory -n {any_cluster_node_adress_or_ip}

    Example:

    ./autoUpgrade.sh -d /tmp/upgrade -n pueas-ansi-node1.swinfra.net

Upgrading from CDF 2019.08 to CDF 2020.02

  1. Delete 2019.08 cdf upgrade directory (cdf-upgrade-2019.08.xxxx):

    rm -rf {download-directory}/cdf-upgrade-2019.08.xxxx

  2. Run a kubectl patch command to configure IDM pod affinity and wait until IDM pods are up and running:

    kubectl patch deployment idm -n core --patch '{ "spec": { "template": { "spec": { "affinity": { "podAffinity": { "preferedDuringSchedulingIgnoredDuringExecution": [ { "labelSelector": { "matchExpressions": [ { "key": "app", "operator": "In", "values": [ "idm-app" ] } ] }, "topologyKey": "kubernetes.io/hostname" } ] } } } } } }'

  3. Change to the directory where you have downloaded the CDF 2020.02 package:

    cd /{download-directory}/identityintelligence-x.x.x.x/installers/cdf-2020.02.00120-2.2.0.2/

  4. Run the following command for automatic upgrade:

    ./autoUpgrade.sh -d /path/to/workinig_directory -n {any_cluster_node_adress_or_ip}

    Example:

    ./autoUpgrade.sh -d /tmp/upgrade -n pueas-ansi-node1.swinfra.net

  5. Delete the auto-upgrade temporary directory from the upgrade manager.

    The auto-upgrade temporary directory contains the upgrade steps and logs. If you want to upgrade another cluster from the same upgrade manager, remove that directory.

    rm -rf /tmp/autoUpgrade

Troubleshooting

  • If the automatic upgrade fails, run autoUpgrade.sh again as outlined above. The process may take several attempts to succeed.

  • In some cases, the automatic upgrade may return an error message about the upgrade process still running and the existence of a *.lock file which prevents autoupgrade.sh to continue. This file is automatically deleted in a few minutes. Alternatively, you can manually delete this file. Once the file is deleted either automatically or manually, run autoUpgrade.sh again.

  • If the automated upgrade process is still unsuccessful, continue the process on the failed node using the procedure outlined in Manual Upgrade.