Originally published April 25, 2019 @ 8:12 pm

Sooner or later it will happen: you type something after which you expect a password prompt then, as a reflex, you type the password. However, you fat-fingered the first command, and your password ended up in clear text in your shell history, likely in the system log, and who knows on what remote syslog server.

Logs live forever. I’ve seen servers with logs that were older than the server (probably copied from the old server for some reason). With the help from remote syslog and various real-time indexers like Splunk or Graylog, as well as NetBackup and such, logs are almost indestructible. Access to log data is generally poorly controlled with read permission granted to everyone more often than not.

All these reasons, combined with the fact that people make mistakes, turn log files into a valuable source of information for the hackers. It may seem that extracting strings that might be passwords from log data would be very challenging and time-consuming. Not so. Standard requirements for password strength make this task much more manageable.

Here’s a simple example to illustrate this point. Is this string a password: password? What about this one: P@ssw0rd1? See my point? Good, let’s continue.

Below is the script (also on GitHub) you can run on just about any modern Linux server to check users’ .bash_history and /var/log/messages for possible passwords. Naturally, you need to be root to do this. You can modify the script to only look at log files you can access with your credentials.

The script
#!/bin/bash
# Up to how many time do you think a user might enter his password in plain text in shell?
# This is to exclude some password-like strings in various log files that appear on a regular basis.
stupid_limit=5

# This is the regex that matches what may very well be a password
include_string="(?=[a-zA-Z0-9\!#@$?]{8,}$)(?=.*?[a-z])(?=.*?[A-Z])(?=.*?[0-9]).*"

# This regex excludes certain strings that look like passwords but you already know they aren't.
# Feel free to modify this to better match your environment.
exclude_string="11gSoftwareR4|Xms1536m|(Jan(uary)?|Feb(ruary)?|Mar(ch)?|Apr(il)?|May|Jun(e)?|\
Jul(y)?|Aug(ust)?|Sep(tember)?|Oct(ober)?|Nov(ember)?|Dec(ember)?)([0-9]{1,})?20[0-9]{2}"

# Here we search a few directories commonly containing logs and shell history files. Search depth is limited,
# so not to go down some NFS-mounted rabbit hole. Here we're looking specifically for users' .bash_history
# and the primary system log. You can add application and system daemon  logs as you need.
find / /home /var/log -mindepth 1 -maxdepth 3 -mount -type f -name "\.bash_history" -o -name "messages" | while read i
do
   # If something is matched
   if [ $(egrep -vi "${exclude_string}" "${i}" | grep -cP "${include_string}") -gt 0 ]
   then
     # Take a closer look
     egrep -vi "${exclude_string}" "${i}" | grep -oP "${include_string}" | sort -u | while read p
     do
       # Does it appear too often to be a user mistake?
       if [ $(grep -c "${p}" "${i}") -lt ${stupid_limit} ]
       then
         # If it looks just right, print the suspected password along with some other details
         echo "$(stat -c %U" "%y" "%n "${i}") ${p}"
         # This is an optional section commented out by default to reduce clutter. This command will
         # grep for an example of the password and a few lines before and after to put things in context
         #echo "-------------------------------------------------"
         #grep -m1 -B4 -A4 "${p}" "${i}"
         #echo "-------------------------------------------------"
         #echo ""
       fi
     done
   fi
done

If you’re a sysadmin and have access to configuration management tools like Salt or Ansible, you can run this script on multiple systems in parallel. Here’s an example of running the script via Salt CLI:

salt "prod-tomcat*" cmd.script "salt://scripts/bash_history_password_find.sh"

And below is sample output. The first three lines look like the user accidentally copy-pasted into command line an encryption key or somesuch. But the other four lines clearly contain two very stupid passwords for which jjames will receive a beating.

Sample output
prod-tomcat-node02.domain.local      tomcat    2019-03-20  05:14:16.000000000  -0400  /home/tomcat/.bash_history    nXhvR1sCAwEBBaMhMB8wHQYCDD0OBBYE
prod-tomcat-node02.domain.local      tomcat    2019-03-20  05:14:16.000000000  -0400  /home/tomcat/.bash_history    y1ynn3Ln76k0isKBwjzsEFmmFHunDjHmHVhINwq
prod-tomcat-node02.domain.local      tomcat    2019-03-20  05:14:16.000000000  -0400  /home/tomcat/.bash_history    yXkQ1kk9MkIg40HfB191vlZARsGgnWCjFfv8niYfVFkyPV
prod-tomcat-node01.domain.local      jjames    2019-01-28  11:59:58.000000000  -0500  /home/jjames/.bash_history    Hot@123
prod-tomcat-node01.domain.local      jjames    2019-01-28  11:59:58.000000000  -0500  /home/jjames/.bash_history    Hot@123
prod-tomcat-node03.domain.local      tomcat    2019-04-08  13:21:55.000000000  -0400  /home/tomcat/.bash_history    Mypass@123
prod-tomcat-node03.domain.local      tomcat    2019-04-08  13:21:55.000000000  -0400  /home/tomcat/.bash_history    Mypass@123

What to do

Here’s what to do if you realize you typed your password in CLI in clear text. It is as simple as one-two-three.

Step 1

Do not pretend that nothing happened. You made an honest mistake, so don’t exacerbate the situation by making a dishonest one.

Step 2

Ask yourself these two questions: can I change the password right now and will doing so cause an outage? If the answers most definitely are “yes” and “no”, correspondingly, then change the password and proceed to Step 3

If the answer is anything else, then just proceed to Step 3.

Step 3

Do not ignore the problem: it will come out sooner rather than later and you’ll surely get in trouble then. Notify your computer security team as soon as possible, or sooner. Notify your supervisor in an urgent manner. Do not email screenshots, log entries, or the actual password.

You may need to submit a trouble ticket, but, once again, do not include screenshots, log entries, or the actual password. In fact, do not commit to electronic record any specifics of the incident: when folks in charge of handling such incidents require more information – they’ll just have to ask you.

Related ArticlesMore From Author