I recently discovered that a virtual Windows server I was running had been hacked. I was being paid to run it as part of an arrangement with a long time client of mine. This server ran Windows Server 2008 R2 and though it was rarely used, it was configured to automatically update when required. The only service it exposed to the internet was RDP over TCP/IP port 3392 instead of the default 3389.
Nevertheless it was hacked. Somebody found the RDP service and managed to crack the credentials for a low privilege account on the machine. Now this isn’t a particularly great accomplishment as both the user name and the password were extraordinarily simple. I could’ve made them more complex but opted to keep it simple for the benefit of my client and ultimately because I believed that exposing RDP over a non-standard TCP/IP port was enough to save me from myself. This of course violated the first and most important rule of security: Security achieved through Obscurity is an illusion.
The hacker in question, operating from an IP address out of the United Kingdom, proceeded to install a fresh variant of Cryptomix Ransomware in an attempt to extort money out of me. Well this is where the story gets both more funny and more sad. You see, I really didn’t pay much attention to the server. The compromised user account didn’t have access to much. But it did have write access (yet another oversight on my part) to an off site replica of some of the client’s files which got updated nightly. So when I started noticing that those files were being renamed and encrypted, I simply assumed they were coming from the client. Truth is, I had almost totally forgotten about this server.
The situation got more confusing when I logged into the clients server only to find that the files appeared to be in pristine condition. Naturally I jumped to the first and most obvious conclusion available: Somebody at the client site paid the ransom. So I re-synced their files with mine, created a backup of those files at an alternate location and raised red flags with my client contact. I then spent more than a few hours researching and digging through the data in an attempt to discover who had done what.
A few days later the files became corrupted again. Yet the clients copies appeared to be pristine. At this point I got paranoid and checked my primary Linux systems for signs of compromise, though none was to be found. I raised more red flags at the client site and was asked to come in and attempt to figure out exactly what was going on. I spent six hours on site that day researching the situation. Yet I was unable to come to a definitive conclusion other than the fact that nobody at the client appeared to have actually paid any ransom and a secondary server of theirs had in fact been compromised. In theory the access there would’ve been enough to access and ransom the files in question.
The result of this is that the original conclusion no longer made sense. If the client wasn’t paying the ransom, then why weren’t their files encrypted and renamed by the time I showed up? So I jumped to a new conclusion: The intruder must be using my clients file store as a test bed for their new ransomware variant and the nightly sync process happened to be copying them up while they were having their fun. It seemed plausible enough, though didn’t sit right with me. I knew I was missing something, but given the evidence I had gathered at the time, it was the best working theory I could construct. Yes it was terribly contrived. But solving a mystery is kind of like making an omelet: You gotta break more than a few eggs.
Somewhere in the flurry of activity and conversations with the client, I realized in the back of my mind that I needed to harden the virtual Windows server I was running for the client like I was hardening their server. But by the time I got home, I had forgotten about it. On Monday evening however, I thankfully remembered and logged onto the server in question. After logging on with the Administrator account, I was surprised to discover that the low level account reserved for the client’s use was logged on as well. This was quite odd as they rarely ever use the server. It exists merely to service them in “oh shit” type of scenarios. So after reconfiguring the NAT rules around the RDP port for the server, I hijacked the user’s session.
The files on the desktop had been renamed and encrypted. The files in the public user profile had been renamed and encrypted. Some of the files in the ProgramData folder had been renamed and encrypted. Most importantly, I apparently had caught the attacker in the act as the ransomware was running and it was again renaming and encrypting the off site replica of my clients files. So I logged the user off. I then killed all remaining process running under the context of said user. I then nuked the user profile. I then cleaned up some of the damage. I restored my offsite replica using the secondary backup I had created.
At this point I realized that cleaning up the mess wasn’t enough. There was a bigger problem here. I was being paid to run a Windows server with a single service exposed to the open internet. The reality is that I’m not a Windows guy. I’ve been transitioning to Linux for well over a decade now and my Windows skills have suffered as a result. In addition, I’ve been working solely as a software developer in my primary job for almost the last three years. I haven’t been doing any real server maintenance work in that time. The hard truth is that this was no longer my area of expertise. This is why I updated my resume. I changed my objective statement. For well over a decade it has read:
“My objective is to expand upon my skills as an Information Technology professional in the areas of Web Development, Networked Systems Security and Systems Administration.”
Sounds great, right? Well yeah it sounds great when you are first starting out in your career and have a “jack of all trades” mentality. For a long while it was actually true. I was doing all of those things simultaneously at previous jobs and I was mostly doing them with Windows.
But now? I primarily write software. I run a few Linux servers, but I’m also a dedicated Linux user. I own a single Windows machine and its an old Surface 3 that I only boot up when I need to reprogram my Logitech Harmony One remote control. That is really its sole purpose in life. The only servers I’m fully responsible for are my own. So given all that, I have now updated my objective statement to this:
“My objective is to write the most useful, efficient and secure software possible to solve the problem at hand.”
So what can you learn from this? For starters: When solving a mystery, don’t limit yourself to whatever narrow scope you happen to be currently entrenched in. That was my primary mistake here. Had I thought about that server sooner, I would’ve discovered the real problem sooner and saved myself and my client a lot of heartache. Secondly, if you don’t use it, you’ll lose it. I used to be able to competently administer a Windows server and keep it safe from all the script kiddies and hackers out there. Clearly that is not the case anymore.