New Command Control rules inconsistently processed, intermittently failing

  • 7022994
  • 22-May-2018
  • 22-May-2018

Environment

Privileged Account Manager

Situation

New Command Control rules work on primary node but not secondary node.
Recently created rules are inconsistently processed by Primary and Backup Manager(s) - intermittently failing.
Failed attempts being processed by the backup manager; services stopped on backup manager will result in successful requests processed by the online primary manager.
Existing rules remain unaffected and are processed correctly by both managers.


Resolution

The simplest solution is to promote the existing Primary modules so the replication thread pushes the latest configuration to all the Secondaries:

  1. Please verify the Backup Manager's 29120 port is reachable from the Primary Manager:
    telnet <backup> 29120

  2. Re-promote primary package modules:
    Note: This should force replication to happen from the primary manager to all backup managers.
    • Navigate to the Primary Manager's packages in the Hosts Console.
    • Select all the packages that display 'Primary' status
    • Click 'Promote Manager' from the left pane.

  3. If the issue persists, please restart PAM service on both primary and backup manager(s) and wait a few minutes.

Cause

Replication issue of Command Control and Auth Modules caused by network issues from Primary to Secondary (backups) on port 29120. A very rare issue which may occur when there is no connectivity between Primary and Secondary servers at the time the replication thread runs.