Originally published August 28, 2019 @ 2:10 pm
To make long story short, had to fix someone’s chmod -R 777 /. A late-night copy-paste fail, it would seem. Needless to say, console access is required, as SSH will not work with permissions on keys wide open. A prerequisite for the process below is to have a similar server (at least same OS flavor and version) with permissions still intact.
The first step would be to log in via console and at the very least fix permissions for these three things: /etc/ssh/, /etc/sudo*, and /root/.ssh. For /etc/ssh permissions should be 644 for public keys and known_hosts, and 600 for everything else. For /etc/sudo*, permissions should be 640 for sudo.conf; 440 for sudoers; and 750 for sudoers.d. Finally, for /root/.ssh permissions should be similar to what you have for /etc/ssh.
With that out of the way, find a similar working server – server_b – to be used as a source of correct file ownership/permission data. On both servers – the model and the broken one (server_a) – edit /etc/ssh/sshd_conf to allow root login and disable any dual-factor authentication. You may also want to temporarily change the root password to something that is easy to type.
We will be using sshfs (yum install sshfs) to mount source (server_b) and target (server_a) root filesystems on the same server. If your two servers have port 22 open between them, then you can mount server_a root on server_b. In the examples below, however, I am using a jumpbox that has SSH access to both server_a and server_b.
Now I am going to mount root filesystems from both servers on the jumpbox:
mkdir /mnt/server_a /mnt/server_b sshfs -o allow_other root@server_a:/ /mnt/server_a sshfs -o allow_other root@server_b:/ /mnt/server_b
The following command will search /etc on the working server and change ownership and permissions of the same files on the broken server, if those folders or files exist:
cd /mnt/server_b/etc && find . -mount -exec chmod -v --reference='{}' /mnt/server_a/etc/'{}' \; -exec chown -v --reference='{}' /mnt/server_a/etc/'{}' \;
Note the -mount option to prevent the find command from traversing mountpoints.
If the affected folders were /usr/bin or /usr/sbin/, the you would also need to copy SUID permissions. Here’s an example that assumes server_b is the unbroken one and server_a is the one that needs fixing:
find /mnt/server_b/usr/{bin,sbin} -user root -perm -4000 | sed 's/server_b/server_a/' | while read line; do
chmod u+s $line
done
When you’re done fixing permissions, don’t forget to restore the /etc/ssh/sshd_config to its original state; bounce sshd, and probably change the root password to whatever it needs to be.
Experienced Unix/Linux System Administrator with 20-year background in Systems Analysis, Problem Resolution and Engineering Application Support in a large distributed Unix and Windows server environment. Strong problem determination skills. Good knowledge of networking, remote diagnostic techniques, firewalls and network security. Extensive experience with engineering application and database servers, high-availability systems, high-performance computing clusters, and process automation.
