<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="/blog/feed.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Sushant Nagil · Write-ups</title>
  <subtitle>Notes from running cloud infrastructure: Kubernetes, Kafka and MQTT, AWS and OCI, and cloud cost.</subtitle>
  <link href="https://sushantnagil.com/blog/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://sushantnagil.com/blog/" rel="alternate" type="text/html"/>
  <id>https://sushantnagil.com/blog/</id>
  <updated>2026-10-18T00:00:00Z</updated>
  <author><name>Sushant Nagil</name><uri>https://sushantnagil.com/</uri></author>
  <entry>
    <title>My first security incident: web shells, a Postgres superuser and a backdoor in a startup file</title>
    <link href="https://sushantnagil.com/blog/first-security-incident/" rel="alternate" type="text/html"/>
    <id>https://sushantnagil.com/blog/first-security-incident/</id>
    <published>2026-10-18T00:00:00Z</published>
    <updated>2026-10-18T00:00:00Z</updated>
    <summary>A failing report download led me to web shells, Postgres jobs, and a backdoor in a service&apos;s startup file. What the attacker did, why cleanup kept failing, and the checks I run now.</summary>
    <category term="Security"/>
    <category term="Linux"/>
    <category term="PostgreSQL"/>
    <category term="AWS"/>
    <content type="html">&lt;p&gt;I was asked to look at report downloads that were failing on a service I don&amp;#39;t usually work on, one whose cloud a separate team runs. I thought it would take a few minutes. While I was in there I found a web shell, and it turned into a few days of cleaning up after someone. It was the first security incident I&amp;#39;ve ever worked.&lt;/p&gt;
&lt;p&gt;I&amp;#39;m writing it down because most of what went wrong was basic misconfiguration rather than any clever attack, and that kind of thing is easy to leave behind on a busy week. I&amp;#39;ve kept everything generic on purpose, so there are no names, no addresses, and nothing useful to an attacker.&lt;/p&gt;
&lt;details class=&quot;tldr&quot;&gt;
&lt;summary&gt;TL;DR: the attack in one chain&lt;/summary&gt;
&lt;div class=&quot;tldr-body&quot;&gt;
&lt;ol&gt;
&lt;li&gt;Leaked credentials got an attacker to services that were open to the internet.&lt;/li&gt;
&lt;li&gt;One was a Postgres database, and a superuser there can run shell commands through &lt;code&gt;COPY ... TO PROGRAM&lt;/code&gt;, so reaching the database meant a shell on the host.&lt;/li&gt;
&lt;li&gt;A web directory that anyone could write to, with automatic deployment on, gave a second foothold as web shells.&lt;/li&gt;
&lt;li&gt;A shared NFS drive, writable by everyone and mounted on both hosts, let the two footholds pass files between them.&lt;/li&gt;
&lt;li&gt;The host&apos;s cloud credentials were stolen, but they were low privilege, so the damage to the account stayed small.&lt;/li&gt;
&lt;li&gt;Cleanup kept failing because the backdoor was hidden in four places at once, including a line in the service&apos;s startup file that rebuilt it on every boot.&lt;/li&gt;
&lt;li&gt;The fix was to take the ports off the internet, cut privileges, lock the web directory, squash root on NFS, and rebuild the host instead of trusting the cleanup.&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
&lt;/details&gt;&lt;p&gt;Here is roughly how it went.&lt;/p&gt;
&lt;ol class=&quot;timeline&quot;&gt;
&lt;li class=&quot;bad&quot;&gt;&lt;span class=&quot;t-when&quot;&gt;weeks before&lt;/span&gt;&lt;span class=&quot;t-what&quot;&gt;The cloud provider&apos;s threat detection flags brute force and stolen credentials, over and over, for weeks. Nobody is reading the alerts.&lt;/span&gt;&lt;/li&gt;
&lt;li class=&quot;bad&quot;&gt;&lt;span class=&quot;t-when&quot;&gt;day of&lt;/span&gt;&lt;span class=&quot;t-what&quot;&gt;Web shells land on the app host. By evening there is an outbound port scan, the storage is enumerated, and files are staged on the shared drive. Overnight the host&apos;s cloud credentials are used from an outside address.&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;span class=&quot;t-when&quot;&gt;+1 day&lt;/span&gt;&lt;span class=&quot;t-what&quot;&gt;I pick up the report ticket and notice a web shell sitting next to the thing I came to fix. First cleanup: block the shells, disable the database jobs, kill the processes.&lt;/span&gt;&lt;/li&gt;
&lt;li class=&quot;bad&quot;&gt;&lt;span class=&quot;t-when&quot;&gt;+2 days&lt;/span&gt;&lt;span class=&quot;t-what&quot;&gt;They are still in. New web shells appear in the morning, and a planted symlink sends the service into a restart loop, so staff cannot log in. The long day: close the ports, rotate credentials, strip privileges, sweep every host.&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;span class=&quot;t-when&quot;&gt;+3 days&lt;/span&gt;&lt;span class=&quot;t-what&quot;&gt;A piece I had missed comes back after a reboot. That is when I find the startup file backdoor. A full pass over every host comes back clean.&lt;/span&gt;&lt;/li&gt;
&lt;/ol&gt;&lt;h2 id=&quot;the-report-bug&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#the-report-bug&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;The report bug&lt;/h2&gt;
&lt;p&gt;The application writes its reports to an NFS share that someone had mounted by hand and never added to &lt;code&gt;/etc/fstab&lt;/code&gt;, so after the last restart it simply wasn&amp;#39;t there. I remounted it and the reports came back, and that part really did take about five minutes.&lt;/p&gt;
&lt;p&gt;Before I logged off I looked at what the servlet container was running, and two of the applications meant nothing to me. They were about a kilobyte each and had been deployed late on a weekend. I opened one, and it was a single page that runs shell commands, with a second page next to it that runs SQL. They were web shells, and they had been sitting there for three days. At that point I stopped worrying about the reports.&lt;/p&gt;
&lt;h2 id=&quot;how-they-got-in&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#how-they-got-in&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;How they got in&lt;/h2&gt;
&lt;p&gt;There was no clever exploit anywhere in this. Everything the attacker needed was already open, and closing any one of those holes would have stopped the whole thing.&lt;/p&gt;
&lt;figure class=&quot;diagram&quot;&gt;
&lt;svg viewBox=&quot;0 0 540 430&quot; role=&quot;img&quot; aria-label=&quot;The intrusion in one picture. Several admin ports were open to the internet, which let an attacker reach a web host running Tomcat and nginx and a database host running Postgres. A shared NFS drive, writable by everyone and mounted on both hosts, let the two footholds pass files to each other.&quot;&gt;
&lt;defs&gt;
&lt;marker id=&quot;ab1&quot; viewBox=&quot;0 0 10 10&quot; refX=&quot;8.5&quot; refY=&quot;5&quot; markerWidth=&quot;7&quot; markerHeight=&quot;7&quot; orient=&quot;auto-start-reverse&quot;&gt;&lt;path d=&quot;M0 0 L10 5 L0 10 z&quot; fill=&quot;#7cc4ff&quot;/&gt;&lt;/marker&gt;
&lt;/defs&gt;
&lt;g class=&quot;d-node&quot;&gt;&lt;rect x=&quot;200&quot; y=&quot;12&quot; width=&quot;140&quot; height=&quot;42&quot; rx=&quot;12&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;270&quot; y=&quot;39&quot; text-anchor=&quot;middle&quot;&gt;the internet&lt;/text&gt;
&lt;text class=&quot;d-bad-tag&quot; x=&quot;270&quot; y=&quot;76&quot; text-anchor=&quot;middle&quot;&gt;DB port · admin port · SSH · RDP · NFS · all public&lt;/text&gt;
&lt;path class=&quot;d-flow&quot; d=&quot;M238 54 C 150 92 135 108 135 146&quot; marker-end=&quot;url(#ab1)&quot;/&gt;
&lt;path class=&quot;d-flow&quot; d=&quot;M302 54 C 395 92 405 108 405 146&quot; marker-end=&quot;url(#ab1)&quot;/&gt;
&lt;text class=&quot;d-note&quot; x=&quot;34&quot; y=&quot;120&quot;&gt;in via the load balancer&lt;/text&gt;
&lt;text class=&quot;d-note&quot; x=&quot;506&quot; y=&quot;120&quot; text-anchor=&quot;end&quot;&gt;straight at the DB port&lt;/text&gt;
&lt;g class=&quot;d-bad&quot;&gt;&lt;rect x=&quot;30&quot; y=&quot;150&quot; width=&quot;210&quot; height=&quot;104&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;48&quot; y=&quot;178&quot;&gt;Web host&lt;/text&gt;
&lt;text class=&quot;d-tag&quot; x=&quot;48&quot; y=&quot;198&quot;&gt;Tomcat · nginx&lt;/text&gt;
&lt;line class=&quot;d-div&quot; x1=&quot;48&quot; y1=&quot;210&quot; x2=&quot;222&quot; y2=&quot;210&quot;/&gt;
&lt;text class=&quot;d-text&quot; x=&quot;48&quot; y=&quot;234&quot;&gt;web shell dropped in&lt;/text&gt;
&lt;g class=&quot;d-bad&quot;&gt;&lt;rect x=&quot;300&quot; y=&quot;150&quot; width=&quot;210&quot; height=&quot;104&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;318&quot; y=&quot;178&quot;&gt;Database&lt;/text&gt;
&lt;text class=&quot;d-tag&quot; x=&quot;318&quot; y=&quot;198&quot;&gt;Postgres ×3&lt;/text&gt;
&lt;line class=&quot;d-div&quot; x1=&quot;318&quot; y1=&quot;210&quot; x2=&quot;492&quot; y2=&quot;210&quot;/&gt;
&lt;text class=&quot;d-text&quot; x=&quot;318&quot; y=&quot;234&quot;&gt;superuser → OS shell&lt;/text&gt;
&lt;text class=&quot;d-note&quot; x=&quot;270&quot; y=&quot;302&quot; text-anchor=&quot;middle&quot;&gt;the two footholds passed files here&lt;/text&gt;
&lt;g class=&quot;d-node&quot;&gt;&lt;rect x=&quot;140&quot; y=&quot;326&quot; width=&quot;260&quot; height=&quot;72&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;270&quot; y=&quot;357&quot; text-anchor=&quot;middle&quot;&gt;shared NFS drive&lt;/text&gt;
&lt;text class=&quot;d-bad-tag&quot; x=&quot;270&quot; y=&quot;377&quot; text-anchor=&quot;middle&quot;&gt;writable by all · on both hosts&lt;/text&gt;
&lt;path class=&quot;d-flow&quot; d=&quot;M135 254 C 135 296 160 306 185 326&quot; marker-start=&quot;url(#ab1)&quot; marker-end=&quot;url(#ab1)&quot;/&gt;
&lt;path class=&quot;d-flow&quot; d=&quot;M405 254 C 405 296 380 306 355 326&quot; marker-start=&quot;url(#ab1)&quot; marker-end=&quot;url(#ab1)&quot;/&gt;
&lt;/svg&gt;
&lt;figcaption&gt;Two footholds on two hosts, bridged by a shared drive. Nothing was exploited; the ports were already open.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;The database was the likely way in. It was reachable from the internet, it allowed password logins from anywhere, and it had more than one superuser account. The part I hadn&amp;#39;t really taken in before was that a Postgres superuser can run shell commands, through a feature called &lt;code&gt;COPY ... TO PROGRAM&lt;/code&gt;. That means an open database isn&amp;#39;t only a risk to the data inside it. It is a shell on the host.&lt;/p&gt;
&lt;p&gt;I couldn&amp;#39;t pin down the exact first access, because the logs had rotated by the time I was looking and the credential alerts had started before the footholds showed up. The web tier was just as open. Its application directory was writable by any user and set to deploy whatever landed in it, so once the attacker could run a command, dropping a file there was enough to turn it into a live application. The admin console was being brute forced tens of thousands of times, but none of those attempts got in, because the writable directory was the easier path.&lt;/p&gt;
&lt;p&gt;The two hosts were tied together by the same NFS drive from the report bug. It was writable by everyone, it didn&amp;#39;t squash root, and it was mounted on both machines, so a file written by one foothold was there for the other to pick up. SSH and RDP were open to the internet as well.&lt;/p&gt;
&lt;h2 id=&quot;why-cleanup-kept-failing&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#why-cleanup-kept-failing&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;Why cleanup kept failing&lt;/h2&gt;
&lt;p&gt;I would find a backdoor, remove it, confirm it was gone, and then find it back the next day.&lt;/p&gt;
&lt;figure class=&quot;diagram&quot;&gt;
&lt;svg viewBox=&quot;0 0 540 440&quot; role=&quot;img&quot; aria-label=&quot;Why cleanup kept failing. A loop: I removed the backdoor and killed the processes, the service restarted or a minute passed, a hidden trigger in a startup file or a database job fired, and the backdoor was recreated, so it came back.&quot;&gt;
&lt;defs&gt;
&lt;marker id=&quot;ab2&quot; viewBox=&quot;0 0 10 10&quot; refX=&quot;8.5&quot; refY=&quot;5&quot; markerWidth=&quot;7&quot; markerHeight=&quot;7&quot; orient=&quot;auto&quot;&gt;&lt;path d=&quot;M0 0 L10 5 L0 10 z&quot; fill=&quot;#cdd6e6&quot;/&gt;&lt;/marker&gt;
&lt;/defs&gt;
&lt;circle class=&quot;d-ring&quot; cx=&quot;270&quot; cy=&quot;220&quot; r=&quot;76&quot;/&gt;
&lt;text class=&quot;d-note&quot; x=&quot;270&quot; y=&quot;215&quot; text-anchor=&quot;middle&quot;&gt;I cleared the symptom&lt;/text&gt;
&lt;text class=&quot;d-note&quot; x=&quot;270&quot; y=&quot;233&quot; text-anchor=&quot;middle&quot;&gt;and missed the cause&lt;/text&gt;
&lt;g class=&quot;d-safe&quot;&gt;&lt;rect x=&quot;165&quot; y=&quot;18&quot; width=&quot;210&quot; height=&quot;66&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;270&quot; y=&quot;46&quot; text-anchor=&quot;middle&quot;&gt;I remove the backdoor&lt;/text&gt;
&lt;text class=&quot;d-tag&quot; x=&quot;270&quot; y=&quot;66&quot; text-anchor=&quot;middle&quot;&gt;kill it, delete files&lt;/text&gt;
&lt;g class=&quot;d-node&quot;&gt;&lt;rect x=&quot;360&quot; y=&quot;185&quot; width=&quot;170&quot; height=&quot;70&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;445&quot; y=&quot;213&quot; text-anchor=&quot;middle&quot;&gt;it restarts&lt;/text&gt;
&lt;text class=&quot;d-tag&quot; x=&quot;445&quot; y=&quot;233&quot; text-anchor=&quot;middle&quot;&gt;a boot or a minute&lt;/text&gt;
&lt;g class=&quot;d-bad&quot;&gt;&lt;rect x=&quot;165&quot; y=&quot;356&quot; width=&quot;210&quot; height=&quot;66&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;270&quot; y=&quot;384&quot; text-anchor=&quot;middle&quot;&gt;a trigger fires&lt;/text&gt;
&lt;text class=&quot;d-bad-tag&quot; x=&quot;270&quot; y=&quot;404&quot; text-anchor=&quot;middle&quot;&gt;setenv.sh · a DB job&lt;/text&gt;
&lt;g class=&quot;d-bad&quot;&gt;&lt;rect x=&quot;10&quot; y=&quot;185&quot; width=&quot;170&quot; height=&quot;70&quot; rx=&quot;14&quot;/&gt;&lt;/g&gt;
&lt;text class=&quot;d-title&quot; x=&quot;95&quot; y=&quot;213&quot; text-anchor=&quot;middle&quot;&gt;backdoor is back&lt;/text&gt;
&lt;text class=&quot;d-bad-tag&quot; x=&quot;95&quot; y=&quot;233&quot; text-anchor=&quot;middle&quot;&gt;setuid · superuser&lt;/text&gt;
&lt;path class=&quot;d-flow2&quot; d=&quot;M376 70 C 450 96 470 132 455 183&quot; marker-end=&quot;url(#ab2)&quot;/&gt;
&lt;path class=&quot;d-flow2&quot; d=&quot;M452 257 C 462 302 420 336 377 358&quot; marker-end=&quot;url(#ab2)&quot;/&gt;
&lt;path class=&quot;d-flow2&quot; d=&quot;M165 388 C 95 362 76 330 92 257&quot; marker-end=&quot;url(#ab2)&quot;/&gt;
&lt;path class=&quot;d-flow2&quot; d=&quot;M90 183 C 80 130 110 96 164 72&quot; marker-end=&quot;url(#ab2)&quot;/&gt;
&lt;/svg&gt;
&lt;figcaption&gt;Four separate layers. Removing one left the rest, and a line in the service&apos;s own startup file rebuilt a piece on every boot.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;There were four of these, and none of them depended on the others.&lt;/p&gt;
&lt;p&gt;Inside the database, scheduled jobs ran every minute as the superuser and kept granting superuser back to the application&amp;#39;s own account, in case anyone took it away. They never showed up in the system cron, because the schedule lives in a table inside Postgres.&lt;/p&gt;
&lt;p&gt;There was also a copy of bash in a temp directory with the setuid bit set, so running it handed back the owner&amp;#39;s shell. I kept deleting it, and after every restart it reappeared. It wasn&amp;#39;t in cron and it wasn&amp;#39;t in systemd. It was three lines added to the end of the service&amp;#39;s own startup script, the one it reads every time it boots:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;&lt;span class=&quot;hljs-built_in&quot;&gt;id&lt;/span&gt; &amp;gt; /tmp/&amp;lt;name&amp;gt;.txt
&lt;span class=&quot;hljs-built_in&quot;&gt;cp&lt;/span&gt; -f /bin/bash /tmp/&amp;lt;name&amp;gt;
&lt;span class=&quot;hljs-built_in&quot;&gt;chmod&lt;/span&gt; 4755 /tmp/&amp;lt;name&amp;gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;So every restart quietly rebuilt it, out of a file I never thought to open. When you clean up after an intrusion, the changes aren&amp;#39;t only in the obvious places like cron. They sit in the ordinary startup and config files you skim past, because you assume those belong to you.&lt;/p&gt;
&lt;p&gt;On top of that there was a watched folder that worked as a command channel. You dropped a script in, it ran, and the output appeared in a file next to it, with nothing crossing the network for anyone to notice. They had also left an extra SSH key in a service account, in case the rest got cleaned up. The web shells themselves kept coming back for a simpler reason, which is that the web port was still open and still deploying whatever was dropped into it.&lt;/p&gt;
&lt;h2 id=&quot;the-outage&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#the-outage&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;The outage&lt;/h2&gt;
&lt;p&gt;One morning nobody could log in. The service was stuck in a restart loop, with hundreds of failed starts behind it. Someone had placed a symlink in its log folder that pointed at a protected system file, and the startup step that fixes permissions on that folder followed the link, hit a file it wasn&amp;#39;t allowed to touch, and gave up. Removing the symlink and restarting brought the login back.&lt;/p&gt;
&lt;h2 id=&quot;the-stolen-credentials&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#the-stolen-credentials&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;The stolen credentials&lt;/h2&gt;
&lt;p&gt;The host&amp;#39;s cloud credentials were stolen and used from an outside address, and that was the part I worried about most. It turned out to be contained, because the credentials were low privilege. They could read and write a couple of storage buckets and reach the monitoring service, and that was all. In the logs you can watch the attacker try to move around the account and get denied, one call after another. There was no access to other accounts, no new users, and no new machines.&lt;/p&gt;
&lt;p&gt;That low privilege is the main reason the damage stayed small. Everything the attacker reached was on the hosts themselves.&lt;/p&gt;
&lt;h2 id=&quot;a-mistake-i-made&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#a-mistake-i-made&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;A mistake I made&lt;/h2&gt;
&lt;p&gt;I made two that are worth owning.&lt;/p&gt;
&lt;p&gt;To cut off the stolen credentials, I put a blanket deny policy on the role. It stopped the stolen key, and it also cut the management agent on the five hosts that shared that role, all at the same moment, so I locked myself out of the machines I was in the middle of cleaning. I narrowed the policy down and got back in, but for a while I had made a bad morning worse. What I took from it is that you cut off access one thing at a time, not in a single sweep.&lt;/p&gt;
&lt;p&gt;Earlier in the same push, a load balancer rule I wrote was a little too broad and briefly returned a 403 to everyone before I corrected it. It is the kind of slip you make when you are moving quickly, and it was the service&amp;#39;s own users who felt it.&lt;/p&gt;
&lt;p&gt;Now, before I touch a role or a policy, I check what else is using it:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;aws iam list-instance-profiles-for-role --role-name &amp;lt;role&amp;gt; \
  --query &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;InstanceProfiles[].Arn&amp;#x27;&lt;/span&gt;
aws ec2 describe-instances \
  --filters Name=iam-instance-profile.arn,Values=&amp;lt;profile-arn&amp;gt; \
  --query &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;Reservations[].Instances[].InstanceId&amp;#x27;&lt;/span&gt; --output text&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h2 id=&quot;clean-or-rebuild&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#clean-or-rebuild&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;Clean or rebuild&lt;/h2&gt;
&lt;p&gt;By the end, everything I had found was gone and the ports were closed, and I still wouldn&amp;#39;t call those hosts clean. Someone had been running commands on them for days, and all I can ever remove is what I actually found. The system was being retired anyway, so we pulled the last reports off it and shut it down. If it had needed to stay in service, the right move would have been to rebuild it from a known good image and restore the data, rather than keep running the copy I had scrubbed.&lt;/p&gt;
&lt;h2 id=&quot;what-actually-fixed-it&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what-actually-fixed-it&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;What actually fixed it&lt;/h2&gt;
&lt;p&gt;Each hole, and what closed it:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The admin ports, the database, Tomcat, SSH, RDP and NFS, all came off the public internet and went behind a VPN.&lt;/li&gt;
&lt;li&gt;The database stopped allowing password logins from the internet, the spare superusers were removed, and the application&amp;#39;s own account lost its ability to run programs.&lt;/li&gt;
&lt;li&gt;The application directory was locked down so it was no longer writable, and automatic deployment was turned off.&lt;/li&gt;
&lt;li&gt;The NFS export was restricted to the hosts that needed it and set to squash root.&lt;/li&gt;
&lt;li&gt;The long lived keys were replaced with temporary roles that carry only the access they need and rotate on their own.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;the-checks-i-run-now&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#the-checks-i-run-now&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;The checks I run now&lt;/h2&gt;
&lt;p&gt;These are the checks that found something here, or that would have found it sooner.&lt;/p&gt;
&lt;p&gt;Anything open to the whole internet:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;aws ec2 describe-security-groups \
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 \
  --query &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;SecurityGroups[].[GroupId,GroupName]&amp;#x27;&lt;/span&gt; --output text&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Postgres superusers, and jobs scheduled inside the database:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;sql&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-sql&quot;&gt;&lt;span class=&quot;hljs-keyword&quot;&gt;SELECT&lt;/span&gt; rolname &lt;span class=&quot;hljs-keyword&quot;&gt;FROM&lt;/span&gt; pg_roles &lt;span class=&quot;hljs-keyword&quot;&gt;WHERE&lt;/span&gt; rolsuper;
&lt;span class=&quot;hljs-keyword&quot;&gt;SELECT&lt;/span&gt; jobid, schedule, username, command &lt;span class=&quot;hljs-keyword&quot;&gt;FROM&lt;/span&gt; cron.job;  &lt;span class=&quot;hljs-comment&quot;&gt;-- if pg_cron is present&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Applications you didn&amp;#39;t deploy, and server side pages changed in the last month:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;&lt;span class=&quot;hljs-built_in&quot;&gt;ls&lt;/span&gt; -la --time-style=long-iso &lt;span class=&quot;hljs-string&quot;&gt;&amp;quot;&lt;span class=&quot;hljs-variable&quot;&gt;$CATALINA_HOME&lt;/span&gt;/webapps&amp;quot;&lt;/span&gt;
find &lt;span class=&quot;hljs-string&quot;&gt;&amp;quot;&lt;span class=&quot;hljs-variable&quot;&gt;$CATALINA_HOME&lt;/span&gt;/webapps&amp;quot;&lt;/span&gt; -name &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;*.jsp&amp;#x27;&lt;/span&gt; -mtime -30&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Setuid files, extra SSH keys, and the tail of the scripts your services run at boot:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;find / -xdev -perm -4000 -&lt;span class=&quot;hljs-built_in&quot;&gt;type&lt;/span&gt; f -not -path &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;/usr/*&amp;#x27;&lt;/span&gt; 2&amp;gt;/dev/null
&lt;span class=&quot;hljs-built_in&quot;&gt;cat&lt;/span&gt; ~*/.ssh/authorized_keys /home/*/.ssh/authorized_keys 2&amp;gt;/dev/null
&lt;span class=&quot;hljs-built_in&quot;&gt;tail&lt;/span&gt; -n 20 &lt;span class=&quot;hljs-string&quot;&gt;&amp;quot;&lt;span class=&quot;hljs-variable&quot;&gt;$CATALINA_HOME&lt;/span&gt;/bin/setenv.sh&amp;quot;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;NFS exports shared with everyone, or without root squashing:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;grep -v &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;^#&amp;#x27;&lt;/span&gt; /etc/exports&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;And the findings the provider has probably already raised that nobody has read:&lt;/p&gt;
&lt;div class=&quot;code&quot;&gt;&lt;div class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;bash&lt;/span&gt;&lt;/div&gt;&lt;pre tabindex=&quot;0&quot;&gt;&lt;code class=&quot;hljs language-bash&quot;&gt;aws guardduty list-findings --detector-id &amp;lt;&lt;span class=&quot;hljs-built_in&quot;&gt;id&lt;/span&gt;&amp;gt; \
  --finding-criteria &lt;span class=&quot;hljs-string&quot;&gt;&amp;#x27;{&amp;quot;Criterion&amp;quot;:{&amp;quot;severity&amp;quot;:{&amp;quot;Gte&amp;quot;:7}}}&amp;#x27;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Most of this came down to open ports. The habit that helped most wasn&amp;#39;t technical at all. I kept a running note of every step as I went, and by the second day I couldn&amp;#39;t have reconstructed the order without it, and I stopped trusting that anything was gone until I had found what kept bringing it back.&lt;/p&gt;
&lt;h2 id=&quot;further-reading&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#further-reading&quot; aria-hidden=&quot;true&quot; tabindex=&quot;-1&quot;&gt;#&lt;/a&gt;Further reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://www.postgresql.org/docs/current/sql-copy.html&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;PostgreSQL &lt;code&gt;COPY ... TO PROGRAM&lt;/code&gt;&lt;/a&gt;. How a database login can become command execution.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://attack.mitre.org/techniques/T1505/003/&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;Web shells (MITRE ATT&amp;amp;CK T1505.003)&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://github.com/citusdata/pg_cron&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;pg_cron&lt;/a&gt;. Scheduling jobs inside Postgres.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;EC2 instance metadata and IMDSv2&lt;/a&gt;. Hardening the path the stolen credentials came from.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://man7.org/linux/man-pages/man5/exports.5.html&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;NFS exports and root squashing&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
</content>
  </entry>
</feed>
