This guide explains how to troubleshoot the Cmd Control agent if one or more servers do not appear in the web app after you follow the steps to install the agent on a server or deploy it to a group of servers.

Preliminary checks

First, check the basics:

Ensure the target servers are running on supported OSes.
If you're using a firewall, make sure Cmd's domains are on your allowlist.
Check that the target servers have consistent internet connections.
If you received a terminal message reporting an issue with your project key while attempting to install, contact Cmd support.

Troubleshooting instructions

1. Check if ‘Disable server’ is on in the Server management page

Go to the ‘Sessions’ page, and click Manage servers.
Disabled servers say “Paused” in the Data capture column, as shown here:

If it was disabled, troubleshooting may be surprisingly easy! Click the three-dot menu icon on the right side of the server's row, and select Enable server.
If it was already enabled, go to step 2.

2. Check if the agent is installed:

On CentOS or RHEL, use: rpm -q ccf

On Debian or Ubuntu, use: dpkg -s ccf

These commands should show the agent package version, e.g.:

$ dpkg -s ccf
Package: ccf
Status: install ok installed
Priority: extra
Section: misc
Installed-Size: 3269
Maintainer: dev@cmd.com
Architecture: amd64
Version: 1.3.2-57
Depends: libc6 (>= 2.12), lipcap2 (>= 2.16), zlib1g (>= 1.2.3)
Conffiles: 
  /etc/cmd/cmd.init 8af81799a70cd4c78279f116bfc82c1d
  /etc/cmd/cmd.service b54dc4b078304215ab55ef5ef599bf2a
  /etc/cmd/cmd.upstart 93e108dlf502327ae9d52e725dfd937a
  /etc/cmd/pkd 38db7ce1861ee11b6a231c764662b68a
Description: CCF - Cmd Client Framework
License: unknown
Vendor: Cmd Inc.
Homepage: https://www.cmd.com
$

If you don’t get similar output, go and install the agent.

If you do, go to step 3.

3. Check if the daemon is running:

The command pgrep cmd_daemon should output a PID. If so, the daemon is running. Go to step 4.

If not, restart the daemon:
- For servers with systemd based init (Centos7/Ubuntu16/Amzn2 and newer),
use the following:
- systemctl stop cmd
- systemctl start cmd

- For servers with upstart based init (Centos6/Ubuntu14/Amzn1-2018 and older),
use the following:
- initctl stop cmd
- initctl start cmd

Again check if pgrep cmd_daemon outputs a PID.

If not, contact Cmd support.
If so, log in to the Cmd web app and check whether data from the server appears. If it doesn't appear after a minute, go to step 4.

4. Check the PAM configuration:

Make sure that /etc/ssh/sshd_config has a line that says “UsePAM yes”, and does not have a line that says “UsePAM no”.

Check the output of the following command: grep -Ri cmd /etc/pam.d .
For each of the following operating systems, if the output you get differs from the output shown below, you should uninstall the agent from the server and reinstall it. If the problem is affecting a fleet of servers, contact us.

- Ubuntu/Debian:

/etc/pam.d/common-session:session required pam_cmd.so

- CentOS/RHEL:

/etc/pam.d/system-auth-ac:session required pam_cmd.so
/etc/pam.d/system-auth:session required pam_cmd.so
/etc/pam.d/password-auth-ac:session required pam_cmd.so
/etc/pam.d/password-auth:session required pam_cmd.so
/etc/pam.d/fingerprint-auth-ac:session required pam_cmd.so
/etc/pam.d/fingerprint-auth:session required pam_cmd.so
/etc/pam.d/smartcard-auth-ac:session required pam_cmd.so
/etc/pam.d/smartcard-auth:session required pam_cmd.so

If problems persist, send the following log files to Cmd support, and go to step 5:

For Ubuntu/Debian:
- /etc/pam.d/common-session
- /etc/pam.d/sshd
- /etc/ssh/sshd_config
- /etc/security/pam_env.conf
For RHEL/CentOS:
- /etc/security/pam_env.conf
- /etc/pam.d
- /etc/pam.conf
- Also send a list of the files in /lib64/security (not the actual files).

How does Cmd use PAM?

Linux Pluggable Authentication Modules (PAM) help applications avoid reimplementing user authentication. When you install the Cmd Control agent, it adds a PAM module (and a reference to it in the PAM configuration at /etc/pam.d/ ). After this, any process that invokes PAM (e.g. sshd, sudo, login) to authenticate a new session will invoke the Cmd PAM module. This module sets the environment variable LD_PRELOAD for all new sessions, which ensures a Cmd library is attached to the program and all of its descendant processes. The library that is attached by LD_PRELOAD communicates with the Cmd agent daemon process over a Unix Domain Socket, (UDS) and puts processes under Cmd control. In summary, Cmd can control all processes launched using Linux PAM, and their descendants, but not system services. If you have heavily customized PAM configurations, there may be exceptions to this behavior.

5. check if the session is hooked:

The command grep libinjector /proc/self/maps should produce output similar to:

7fdd98747000-7fdd9874d000 r--p 00000000 ca:01 272761           /lib64/libinjector.so
7fdd9874d000-7fdd98768000 r-xp 00006000 ca:01 272761           /lib64/libinjector.so
7fdd98768000-7fdd98772000 r--p 00021000 ca:01 272761           /lib64/libinjector.so
7fdd98772000-7fdd98773000 ---p 0002b000 ca:01 272761           /lib64/libinjector.so
7fdd98773000-7fdd98775000 r--p 0002b000 ca:01 272761           /lib64/libinjector.so
7fdd98775000-7fdd98776000 rw-p 0002d000 ca:01 272761           /lib64/libinjector.so

If the output looks okay, continue to step 7. Otherwise, the Pam config file might have a missing newline character between config variables. If so, remove it and try again. If problems persist, go to step 6.

6. Check the agent status API:

Check the agent's status using the agent status API.

Contact Cmd support for further assistance.