Copying Data in a Restricted Environment

Originally published August 9, 2019 @ 8:02 pm

Consider this not-so-hypothetical scenario: you have some data on server_a that you would like to copy to server_b. Unfortunately, these two servers cannot communicate with each other. Nor do they have access to any common network-mounted storage. Bummer. I do have a jumpbox from which I can SSH to either server using a service account with passwordless root sudo privileges.

Such accounts often exists in environments that utilize CMS tools and other types of centralized automation. Normally I would SSH to server_a; make a tarball of whatever I need; logout back to the jumpbox; SCP the tarball to the jumpbox; SCP it further to server_b; SSH there and untar; finally, delete the tarballs from all three systems.

This multi-step process is tedious, not very secure, and sometimes requires considerable additional filesystem space. Now imagine a few files changed on the source and you need to transfer just those changes. Ugh…

A much better option is to use sshfs (yum install sshfs). The little script I have below was written for RHEL/CentOS 5-7 and has a couple of prerequisites: a) you have a server that can connect to both server_a and server_b via SSH with key authentication only using a common service account; b) this service account on both server_a and server_b can sudo - root without being prompted for a password.

The basic command syntax is quite simple:

g source:/path/[file] target:/path/

The script will SSH to source and to target and will run sftp-server on both as root. This would allow you to mount filesystems from both machines with root privileges. The script will then create target:/path/ if it doesn’t already exist. Finally, the script will run a basic rsync archive operation to transfer data. The temporary mounts on your jumpbox will then be unmounted and removed.

You can also download this script from my GitHub page. Copy it to a folder of your choice and make a convenient link, say, /sbin/g.

#!/bin/bash
#
#                                      |
#                                  ___/"\___
#                          __________/ o \__________
#                            (I) (G) \___/ (O) (R)
#                                   Igor Os
#                           igor@comradegeneral.com
#                                 2019-08-09
# ----------------------------------------------------------------------------
# Just a quick way of copying files/folders from any to any server using your
# automation service account. This acconut would need passwordless sudo
#
# Syntax:
# g <source_host>:/<path>/[filename] <target_host>:/<path>/
# if "<target_host>:/<path>/" does not exist, it will be created.
#
# Example:
# root@saltmaster# g dev-tomcat01:/etc/ntp/ prod-tomcat02:/etc/ntp_from_dev-tomcat01/
#
# CHANGE CONTROL
# ----------------------------------------------------------------------------
# 2019-08-09  igor  wrote this script
# ----------------------------------------------------------------------------

rnd="${RANDOM}"
u="service_account"
rsa_id="$(find /home/${u}/.ssh -mindepth 1 -maxdepth 1 -type f -name "id*rsa" | head -1)"
if [ ! -r "${rsa_id}" ]; then exit 3; fi
ssh_opt="-qtT -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes -i ${rsa_id}"
sshfs_opt="allow_other,IdentityFile=${rsa_id},UserKnownHostsFile=/dev/null,StrictHostKeyChecking=no"

if [ ! -z "" ] && [ ! -z "" ]
then
  s=""
  s_host="$(awk -F':' '{print $1}' <<<"${s}")"
  s_node="$(awk -F':' '{for(i=2;i<=NF;++i)print $i}' <<<"${s}")"
  s_dir="$(awk -F'/' -v OFS='/' '{$NF=""; print $0}' <<<"${s_node}")"
  s_file="$(awk -F'/' '{print $NF}' <<<"${s_node}")"
  t=""
  t_host="$(awk -F':' '{print $1}' <<<"${t}")"
  t_dir="$(awk -F':' '{for(i=2;i<=NF;++i)print $i}' <<<"${t}")"
else
  exit 1
fi

s_mnt="/mnt/source/${s_host}_${rnd}"
t_mnt="/mnt/target/${t_host}_${rnd}"
mkdir -p "${s_mnt}" "${t_mnt}"
/usr/bin/ssh ${ssh_opt} ${u}@${t_host} "sudo su - root -c 'mkdir -p \"${t_dir}\"'" 2>/dev/null 1>&2
sshfs -o ${sshfs_opt} ${u}@${s_host}:${s_dir} ${s_mnt} -o sftp_server="/usr/bin/sudo /usr/libexec/openssh/sftp-server"
sshfs -o ${sshfs_opt} ${u}@${t_host}:${t_dir} ${t_mnt} -o sftp_server="/usr/bin/sudo /usr/libexec/openssh/sftp-server"
if [ $(mountpoint "${s_mnt}" 2>/dev/null 1>&2; echo $?) -eq 0 ] && [ $(mountpoint "${t_mnt}" 2>/dev/null 1>&2; echo $?) -eq 0 ]
then
  if [ ! -z "${s_file}" ]
  then
    rsync -aqKx "${s_mnt}/${s_file}" "${t_mnt}"/
  else
    rsync -aqKx "${s_mnt}"/ "${t_mnt}"/
  fi
  umount -f ${s_mnt} ${t_mnt}
fi
/bin/rmdir "${s_mnt}" "${t_mnt}"