This guide explains how to troubleshoot the Cmd Control agent if one or more servers do not appear in the web app after you follow the steps to install the agent on a server or deploy it to a group of servers.
First, check the basics:
- Ensure the target servers are running on supported OSes.
- If you're using a firewall, make sure Cmd's domains are on your allowlist.
- Check that the target servers have consistent internet connections.
- If you received a terminal message reporting an issue with your project key while attempting to install, contact Cmd support.
1. Check if ‘Disable server’ is on in the Server management page
- Go to the ‘Sessions’ page, and click Manage servers.
- Disabled servers say “Paused” in the
Data capturecolumn, as shown here:
- If it was disabled, troubleshooting may be surprisingly easy! Click the three-dot menu icon on the right side of the server's row, and select Enable server.
- If it was already enabled, go to step 2.
2. Check if the agent is installed:
- On CentOS or RHEL, use:
rpm -q ccf
- On Debian or Ubuntu, use:
dpkg -s ccf
These commands should show the agent package version, e.g.:
$ dpkg -s ccf
Status: install ok installed
Depends: libc6 (>= 2.12), lipcap2 (>= 2.16), zlib1g (>= 1.2.3)
Description: CCF - Cmd Client Framework
Vendor: Cmd Inc.
- If you don’t get similar output, go and install the agent.
- If you do, go to step 3.
3. Check if the daemon is running:
- The command
pgrep cmd_daemonshould output a PID. If so, the daemon is running. Go to step 4.
- If not, restart the daemon:
- For servers with systemd based init (Centos7/Ubuntu16/Amzn2 and newer),
use the following:
systemctl stop cmd
systemctl start cmd
- For servers with upstart based init (Centos6/Ubuntu14/Amzn1-2018 and older),
use the following:
initctl stop cmd
initctl start cmd
Again check if
pgrep cmd_daemon outputs a PID.
- If not, contact Cmd support.
- If so, log in to the Cmd web app and check whether data from the server appears. If it doesn't appear after a minute, go to step 4.
4. Check the PAM configuration:
- Make sure that
/etc/ssh/sshd_confighas a line that says “UsePAM yes”, and does not have a line that says “UsePAM no”.
- Check the output of the following command:
grep -Ri cmd /etc/pam.d.
For each of the following operating systems, if the output you get differs from the output shown below, you should uninstall the agent from the server and reinstall it. If the problem is affecting a fleet of servers, contact us.
/etc/pam.d/common-session:session required pam_cmd.so
/etc/pam.d/system-auth-ac:session required pam_cmd.so
/etc/pam.d/system-auth:session required pam_cmd.so
/etc/pam.d/password-auth-ac:session required pam_cmd.so
/etc/pam.d/password-auth:session required pam_cmd.so
/etc/pam.d/fingerprint-auth-ac:session required pam_cmd.so
/etc/pam.d/fingerprint-auth:session required pam_cmd.so
/etc/pam.d/smartcard-auth-ac:session required pam_cmd.so
/etc/pam.d/smartcard-auth:session required pam_cmd.so
If problems persist, send the following log files to Cmd support, and go to step 5:
- For Ubuntu/Debian:
- For RHEL/CentOS:
- Also send a list of the files in /lib64/security (not the actual files).
How does Cmd use PAM?
Linux Pluggable Authentication Modules (PAM) help applications avoid reimplementing user authentication. When you install the Cmd Control agent, it adds a PAM module (and a reference to it in the PAM configuration at
/etc/pam.d/ ). After this, any process that invokes PAM (e.g. sshd, sudo, login) to authenticate a new session will invoke the Cmd PAM module. This module sets the environment variable LD_PRELOAD for all new sessions, which ensures a Cmd library is attached to the program and all of its descendant processes. The library that is attached by LD_PRELOAD communicates with the Cmd agent daemon process over a Unix Domain Socket, (UDS) and puts processes under Cmd control. In summary, Cmd can control all processes launched using Linux PAM, and their descendants, but not system services. If you have heavily customized PAM configurations, there may be exceptions to this behavior.
5. check if the session is hooked:
- The command
grep libinjector /proc/self/mapsshould produce output similar to:
7fdd98747000-7fdd9874d000 r--p 00000000 ca:01 272761 /lib64/libinjector.so
7fdd9874d000-7fdd98768000 r-xp 00006000 ca:01 272761 /lib64/libinjector.so
7fdd98768000-7fdd98772000 r--p 00021000 ca:01 272761 /lib64/libinjector.so
7fdd98772000-7fdd98773000 ---p 0002b000 ca:01 272761 /lib64/libinjector.so
7fdd98773000-7fdd98775000 r--p 0002b000 ca:01 272761 /lib64/libinjector.so
7fdd98775000-7fdd98776000 rw-p 0002d000 ca:01 272761 /lib64/libinjector.so
If the output looks okay, continue to step 7. Otherwise, the Pam config file might have a missing newline character between config variables. If so, remove it and try again. If problems persist, go to step 6.
6. Check the agent status API:
Check the agent's status using the agent status API.
Contact Cmd support for further assistance.