I've spent my evenings this week tracking down a website hack. It's not on this server, but on another some friends and I maintain. A message came in on the trouble ticket system, "you have a hack". Someone had noticed pages full of links to "blue pill" sites stuck up on our site. I volunteered to look into it. I've had to do this sort of forensic/detective work a few times in the past; it's pretty fun and challenging.
The first clue was that the files were all owned by the web server process, indicating a high likelihood that a vulnerability in the web site had been exploited (as opposed to some other type of hack masquerading as a web sploit). The second clue was that there was a zip file, plus all the files had the same timestamp, indicating that they were extracted from the zip (versus uploaded individually). That meant that the hacker had been able to execute code on our server. Arbitrary code execution is a severe vulnerability, it means the hacker can pretty much do anything (limited only by the privileges of the web server process).
This all pointed to PHP. There's a ton of PHP code on this system. PHP itself was notoriously crackable early on, but by about age ten they'd rooted out all the major security flaws (it's still a case study in how not to design, or grow, a programming language). But since PHP has such a low barrier to entry, the code that web programmers write with it is often flawed.
We're using third-party software like mediawiki (wiki system), vbulletin (forums), gallery (images) and wordpress (blogging). Some of those have a pretty bad track record, and I'm looking at wordpress when I say that. And there's a variety of WP vintages on this system. But there's also a lot of custom code written by people I don't even know, with a variety of skill and attention to security concerns. But there's also some legacy Perl CGI code, some of it written by me, that could be at fault....
Back in 2006, someone had written a quick-n-dirty (very dirty) CMS hosted on our system, which worked by using the PHP include function. You stuck in a web site address and it included the contents. If the content is actually itself PHP, it all gets executed together. You can turn off this remote include functionality, but it's on by default (I was quite surprised by that). So there are loads of scripts out there that take advantage of this weakness. Very sophisticated web interfaces for commandeering a server. I found this curious double URL in the logs, pasted it into a browser, and all of a sudden there's a full console for everything from mass spam emails to cracking passwords. You could use it for system administration! It's not hard to imagine a script to crawl the net and find these weaknesses, you just find forms, pass URLs into the parameters, and see if anything in the content of the URL is included in the output. I ultimately found one PHP scriptkit on the system (called r57). I learned a lot about PHP programming by examining the source.
.
But we'd hardened our web server a bit since then. So it had to be something a little different. I worked on two fronts. The first was examining the web logs, writing Perl scripts to analyze them for the fingerprints of certain kinds of attacks. The second was scanning the files on the web site, looking for vulnerable code. The latter yielded results more quickly. I started turning up more sploitkit PHP files in upload folders. My old friend, r57, was one of them. Another was madshell. You have to check out their source code to appreciate their evil genius. But how had they gotten there? There was a lot of code on the system for handling file uploads, and it would be impossible to verify all of it. But most of it looked pretty secure, it either validated user input (made sure they couldn't upload PHP), required a logged-in user, or both.
By going back to the logs, I could see who was using the sploitkits. By listing what else they were doing, I soon found something very disturbing: a vbulletin page was being used to run arbitrary system commands (the PHP shell_exec function). I tried it, put in a URL with the command to show the password file, and it worked. We had the latest vbulletin, and it would be surprising that such a huge, gaping, easily exploited security hole would exist in it. A check on securityfocus.com showed that there were no vulnerabilities listed for it, so it had to be one of the plugins. Sure enough, some of the other activity showed the hacker activating vbulletin plugins.
The smoking gun came in finding access from the same IP address to vbulletin plugin activation, then to the exploit it opened up, then to a bunch of commands to fetch PHP sploitkits, then a bunch of activity with those, and seconds after that, to checking out the pwnage, namely the very files that a user had noticed and started this whole debacle.
One mystery remains: how did the hacker gain administrative access to vbulletin? My theory is that it was simply a bad (poorly chosen) password by one of the admins, but I'm still digging....
Sunday, March 22. 2009 at 19:59 (Reply)