It all starts with a text message from my wife...
Of course, all of the DS iOS apps fail to connect to the device. This has happened before. I figured I'll just open up the browser when I get to my Mac and restart the services. Then I get this screen.
My first thought was WTF? Took a screencap and put it out on twitter thinking Synology installed some encryption service on my device through some automatic update. Then the swarms of "oh no..." tweets come replying back. Apparently, I must have been hiding under a rock. And I was. For the past two weeks I've been heads down doing some front-end web development training. I literally saw nothing over the prior week about SynoLocker.
Don't know what SynoLocker is? It's a form of malware/ransomware based on a different strain of the Cryptolocker ransomware. Once you are infected, all of your files are overwritten with an encryption algorithm that can only be decrypted with a key held by the person that created the strain. Essentially, your files are gone unless you pay the person a certain amount for the decryption key. SynoLocker ransomware scans the internet for port 5000 and 5001 and will return a Synology ACK. This means you have to have the management port of 5000 & 5001 exposed on the internet. Once discovered, the ransomware plants itself through an exploit and begins encrypting your files. The files targeted are <100mb because the larger the file, the longer the encryption process takes. Their target were smaller files such as documents and photos.
So began my day of research and frustration. I opened a ticket with Synology and waited for a support response. Time to begin reading elsewhere.
This Synology thread has the most useful information and will pull out some of the best bits from it as we go. SynoLocker Ransomware Affecting Synology DiskStation. The other good read is on F-Secure. Ransomware Race (part 2): Personal media the next frontier?
On that thread there are lots of differing opinions. Who do you blame, the end-user or Synology? FYI, I only have myself to blame... but I'm still pretty pissed all this happened.
Lets look at Synology's POV:
- SynoLocker exploits a security hole that was addressed in a DSM upgrade that was released in December of 2013. It's been confirmed that if you are running DSM 5, then you are not susceptible to the exploit. However, earlier versions of DSM 4.3 are susceptible.
- Synology products have a version checking feature that will see if you are running the latest version within the management UI.
- Synology will send out news and alerts if you have registered with the MyDS service
- Synology will post all updates on their website
- Good sysadmins live in a constant state of fear and constant updates should be a mantra in life.
- Patching should be a regular part of any product. Even if it's the firmware on your TV, it should be ingrained into everyone's DNA.
Ok, Now lets look at the end-user side:
- You know the old saying, "If it ain't broke, don't fix it"? Well, I've had plenty of vSphere issues in the past with DSM upgrades and didn't want to be in a situation where it wouldn't work.
- Performing a DSM upgrade requires me to shutdown my entire vSphere environment if I want to do it correctly. This entails atleast 20-30 minutes of additional time spent that I didn't care for doing. (hindsight, i know).
- Many people on the thread mentioned above, never upgraded to DSM 5 because there were early reports of the upgrade process bricking their NASs
- There is NO possibility of rolling back from a DSM upgrade. Once you upgrade, you better pray everything works. Many people never upgraded because there were community developed packages they were running or services that had broken from previous upgrades.
- Synology targets SOHO and SMB users. They are not targeting enterprise customers where it takes an IT staff to run a NAS. They want to make it dead simple.
- Unless you subscribed to the MyDS service, you were never notified about anything coming from Synology.
- It was confirmed that in a few DS models (including my DS411+) there was a bug that showed running the DSM 4.3 version was the most recent update. No upgrade was necessary. (picture below depicting another users DS).
- It is possible to change the management port to something other than 5000 or 5001, but "obscurity isn't security". In addition, the Synology iOS & Android apps use ports 80, 5000, and 5001 by default to be used for connection (i had this exposed BEFORE DS QuickConnect was a part of DSM, pre 4.3 days). This includes DS Photo+ and DS Audio. The two apps I use all the time. Turn on the UPnP service for a novice user, and you're exposed. In addition, ALL SYNOLOGY DOCUMENTATION demonstrates using ports 5000 and 5001 for internet facing services.
- As an end-user, how often should you be checking the Synology website for updates? Weekly, Monthly? Me personally, I never checked it for updates because everything was working just fine.
Many people suggested to use the decryptcryptolocker.com service. No such luck because it's a different strain of the malware.
Here are some gems from the thread before I go into my recovery process.
It happened yesterday midday. People started noticing that some files on Synology DS414 are not valid anymore. When I noticed, that they have the same date and time I suspected, they are being encrypted, so I disconnected USB backup drive. I could normally log in to server.
There was no other sign that system was invaded until late in the evening, when all the files were encrypted. Then, even login to the server was denied, and finally ransom message was displayed.
Synolocker with bitcoin demand.
Well, I am not going to sponsor bad guys, so I started with system reinstallation.
Was looking to a infected box and found PORT STATE SERVICE VERSION
8080/tcp open http-proxy PWNED Port 8080 is running the page to input the decryption key
PWNED was the service running the Bit/dogcoin miners
https://www.synology.com/en-global/comp ... rticle/437
Are this the same guys? and/or the same vulnerability?
Encrypted files now contain the following text, from offset 00000100 (this can be viewed via a hex editor):
THE_REAL_PWNED_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_1337
A business I consult for just got hacked by Synolocker on Saturday 8/2. They are freaking out! We like many others need a solution to decrypt our files. Not all have been encrypted, so for now the system has been turned off.
Add my DS412+ to the list of affected users. When I saw the first warnings about this, I went to upgrade DSM from the control panel and saw that my unit was already up-to-date. But today, when I saw that this threat was reposted on Facebook, I decided to double check the version.
Sure enough, I now have SynoLocker.
This email address is being protected from spambots. You need JavaScript enabled to view it.
Situation: Lawyer cabinet 10 workers affected. All file lost
Story
1. Unable to login DSM "System is getting ready. Please login later"
2. Files begins to be unusable (encrypted)
3. Call Synology support, follow instruction, reset DSM
4. Try to connect again. See the Synolocker page, shutdown system
5. Try to restart, configuration lost
6. Install DSM 5.0 fine but files are still encrypted
7. Check backup, Time backup folder are also affected and deleted, only some old version 6 month are available. I found also some files in the trash of the external usb, but nothing to help restart production
8. Resign to pay Synolocker. Hard to get 0.6 bitcoin. Finally send the money and receive the key to decrypt but with the new DSM 5.0 it was impossible to start the decryption process
9. On Synolocker support page send message option not working
CVE-2013-6955 was patched following the rather unusual Bitcoin mining malware for Synology devices back at the end of 2013. I work in information security and I cannot express strongly enough that applying patches is one of the most important aspects of internet security, looking at Windows malware from beginning to end it's quite often unpatched vulnerabilities that enable the infection.
The process is often this.... Innocent but unpatched web server is compromised through exploitation of a vulnerability then a connecting workstation (either through google search or phishing email)that has unpatched "high risk" web browser plugins for Java, PDF Reader, Flash etc connects to the site, is redirected to the Exploit Kit which then silently compromises the workstation, I found that often even with AV it came down to a 50/50 of something nasty being installed undetected.
I still find it rather shoddy that unlike later versions of Chrome or Firefox which block out of date plugins that Microsoft have never bothered to try.
I suppose with the job I do, I've used Metasploit occasionally, looked in reasonable depth at all the infections reported and have "luckily" ended up pretty negative about internet facing devices to common services.
According to F-Secure (http://www.f-secure.com/weblog/archives/00002730.html) it would appear that the team behind Synolocker have unfortunately implemented the encryption properly, so unless some security researchers find the keys somewhere, like they did with Cryptolocker, then the data is lost, paying for the decryption keys is not really an option as they are crooks.
It's a tough lesson to learn and I feel as a Synology owner that they have crammed too much in without too much apparent concern for the security implications of unforeseen vulnerabilities, especially if as I've seen suggested the httpd (web server) runs as root, if true this is another attack waiting to happen.
BTW, as for the unsupported models someone mentioned here...The above being a DS-107 with a two-liner synoinfo.conf mod
They only encrypt small files, size less than 100M, so all music, photos, documents etc will very likely be encrypted. But if you have big files such as video, no worry. Everything is fine, after you follow one of the post here and reboot the NAS using a different hard drive, install the latest 5 DSM, them shut down, replace it with ur original disks, do a migration, you will see they are OK, but remember ONCE YOU DO THIS, THE RANSOM PAGE INFO WILL BE LOST UNLESS YOU HAVE A COPY (IN CASE YOU STILL DECIDE TO PAY).
How I know this? I just did this, on second thought that makes sense as it's gonna take a long time to encrypt the big video files, so they just target the small files. For me I'm gonna just put the NAS behind the router for now, I hope these guys will get what they deserve.
I'm not sure how accurate Shodan (http://www.shodan.io) are but...
Searching for title:SynoLocker will give you 198 hits, of these 194 have port 5000 or 5001 open...
Searching for port:5000 ..... 1.336.485 hits (majority of these are Synology and redirects to 5001)
Searching for port:5001 ..... 6.690 hits
Paid the ransom. Decryption key would not work. Reached out to their support via bitmessage and got a nearly instant response. They logged in and enabled the decryption process.
That said, I think there is still some misunderstanding of what Synology failed to do for its user base. The problem isn't that Synology didn't issue a security patch for their software. The problem is that they didn't tell their users about it. This isn't just the 5 day delay problem in not posting any kind of notification when Synolocker first appeared. Where was the email notification last October? (Not December, BTW, when they finally patched all versions – except DSM 5 which was still in beta and left vulnerable for another 2+ months!) Even on the web site (which one should not have to monitor to be kept informed of security updates), the severity of the problem in the listings for multiple updates from last fall is not indicated, nor are the affected ports and services detailed. (And to claim that such info needs to be kept secret after a patch is issued is simply B.S.) The lack of user notification problem isn't just my humble opinion. The folks over at Extreme Tech are asking the same question about Synology's actions:
This is one of the ugly gray areas of modern computing — if a company issues a patch, but customers aren't aware of it, who bears responsibility for a data breach?
http://www.extremetech.com/computing/184803-hacker-hijacks-thousands-of-synology-storage-devices-forces-them-to-mine-500-million-dogecoins
(See
)
A second Extreme Tech article on the recent pair of Synology hacks has this to say:
Synology may have to take a page from Microsoft's book and aggressively push users to update to later OS versions as a way of resolving security concerns.
http://www.extremetech.com/extreme/187518-synology-nas-devices-targeted-by-hackers-demand-bitcoin-ransom-to-decrypt-files
(See
)
I totally agree, but with the poor quality of Synology DSM releases (just take a look at the DSM 5.0 forum), who wants to take a chance on bricking the device, destroying data yourself instead of letting a hacker do it for you, or otherwise creating new problems with mission critical processes that used to work just fine, thank you? Given these circumstances, people will refrain from upgrading, sometimes with disastrous results. Given Synology's poor user notification and hit-or-miss DSM updates, can you blame the user? I think not.
My answer has been to closely investigate what the security issue is and see if the installations I support are vulnerable. Time and again, the answer has been "no" which is how I got off the upgrade-oh-crap-now-I-broke-it hamster wheel with Synology products. In this case, after much research, it appears that the Synolocker exploit came via a port 5000 vulnerability. Not exposing port 5000? Then you don't have a problem and don't need to update. You do, however, need to remind your users not to turn stuff on without asking you first, however. Not the best situation – I'd rather update DSM if it weren't so problematic.
I'm also going to complain about Synology practices one more time. The information about port 5000 didn't come from Synology. Bad form, folks. Many sources were reporting vastly increased scanning on port 5000, and then I found this gem indicating it was aimed at Synology: https://isc.sans.edu/forums/diary/Port+5000+traffic+and+snort+signature/17771
So there you have it, friends. My open letter on page 32 of this thread stands. You may not agree with everything I said, you may think the users in some or all cases bear more responsibility, but anyone who thinks that Synology doesn't bear at least equal responsibility or doesn't need to clean up their act as I described … well, they are just Synology fanbois.
After about 4 hours, Synology support emails me back and they may have well titled the email "you're screwed".
Hi Kendrick,
Thank you for contacting us. We appreciate your continued patience and support during this time.
According to the information you provided, we can confirm that your DiskStation has indeed been infected by the recent ransomware called “SynoLocker.” As we are unable to decrypt files that have already been encrypted, there are two ways to proceed.
#1 Reset your DiskStation and restore backup data
If you happen to possess a full backup copy of your files (or there are no critical files stored on your DiskStation), we recommend following the below steps to reset your DiskStation and re-install DSM:
Follow the steps in this tutorial to reset your DiskStation: http://www.synology.com/support/tutorials/493#t3
The latest version of DSM can be downloaded from our Download Center here: http://www.synology.com/download
Once DSM has been re-installed, log in and restore your backed up data.
#2 Reset your DiskStation, but preserve encryption information
According to our investigation, we can stop the ransomware from continuing to further encrypt files. We cannot decrypt those files which have been encrypted already. As resetting the DiskStation would remove the information required for decryption, we can back up the encryption information on your data volume, just in case you wish to decrypt them yourself. Once your DiskStation has returned to normal status, you can 1) contact us for information about retrieving the encryption information, or 2) delete the encrypted files entirely.
If you would like us to preserve the encryption information, please follow the below steps to provide us with the necessary information to remotely access your DiskStation:
Power off DS and take out all hard drive.
Power on DS without hard drives.
Run Synology Assistant and perform the installation using the following DSM patch below: ftp://on-line:online_user@ftp.synology.com/on-line/Tools/fake_pat/a.pat
Using this method, the installation will appear to fail, but Telnet service shall be open.
Please open port 23 on your router to allow DiskStation NAT IP (ex. 192.168.xx.xx or 10.0.xx.xx) and we could login via Telnet.
Please re-insert the hard drives into your DiskStation without rebooting and provide us the following information:
Standard ports we need you to open: 23 (Please see the following link on how to forward your ports, and then select the Telnet application: http://www.portforward.com/english/routers/port_forwarding/routerindex.htm)
The WAN IP address of your DiskStation (Please go to http://www.canyouseeme.org/ to find your IP address and check whether or not port 23 is open)
According to our investigation, the ransomware only affects outdated versions of DSM. No vulnerability has been found in the latest version of DSM. To keep yourself informed about new DSM updates, we highly recommend registering at MyDS Center and subscribing to Synology eNews.
We sincerely apologize for any problems or inconvenience this has caused you. We shall do our utmost to provide any feasible solution.
<name removed>
Technical Support
Synology America Corp.
www.synology.com
So at this point, the only option I had left was to pay the bastards. Before I did that, I went ahead and turned off port forwarding of 5000 and 5001 on my router so I can make sure I'm "re-infected" once I go through this process. I know what some of you are thinking... What could you possibly have on there? I have over a decade's worth of photos on there that were never backed up to anything else. It's the only memories I have and it's worth a lot of money for me to pay them. I also have about 25,000 MP3s, but I could care less. I stream all my music now, so it's not *that* big of deal. This will teach me to back up my pictures every once in a while to something else.
One thing that I noticed is that NONE of the VMs I had were susceptible to this attack. Even though .vmx are only a few kb, none of them were ever encrypted. This is because only these types of files being targeted. Image courtesy of F-Secure.
There will be no known way of decrypting this process any time soon so lets do this. I downloaded the TOR browser, and went to the web page listed on my synology device.
I logged in with my "ID" and I saw the following screen. (note, when I first went to this screen it said I had 4 days and 14 hours remaining.)
I went ahead and connected my bank account to my bitcoin wallet and began transferring $356.19 to cover the 0.6BTC ransom. After the transaction went through, I figured it would be a few days just like any other bank transfer.
Fast forward 3 days...
It's now Sunday morning and the status of the BTC transfer says that it won't complete until EOD Wednesday!! Well, that's not good. I don't want to pony up 2x the ransom cost so I reached a point of desperation. I had to ask for help from twitter...
I can't thank everyone enough for the re-tweets and positive support. But most of all, I have to thank Alex Beavan (@alexgnbeavan) and Anthony Elizondo (@complex) for stepping up and willing to transfer me money. Alex and I are co-workers, so it was much easier for him to come and take out my kneecaps if I never pay him back. Alex transferred 0.6BTC and after it cleared I sent the bitcoins into la la land. (and don't worry, I'm sending Alex repayment once my bank transfer clears) Now it was time to wait...
One hour later and my RSA Private key is available.
I open up my DSM to http://ip:5000/redirect.html and paste my RSA key
Click Go...and it fails... great. You see the screen below then it goes right back to the redirect.html page.
So lets open up the support link back in the TOR browser.
Looks like the best way to approach this is through BitMessage. I download the application, and contact them. I sent this message...
Looks like I am either impatient or maybe someone backdoor'd me because I checked my DSM and now the files are starting to decrypt. Time to go to bed and wait this out until tomorrow morning...
Fast forward 20 hours... literally 20 hours...I finally was able to click UNINSTALL. No screenshot taken of the uninstall. But it showed a pop-up saying "This action is irreversible".
I waited a few minutes, then went back to my DS home at http:IP and it automatically re-routed to the http://ip:5000/webman/index.cgi just as usual. What a sigh of relief...
Well... not so much... After even a manual reboot, I still can't log in to the NAS. According to Synology Product Security Advisory DSM 4.3-3827, it seems this message is a good indication that malware is still infecting the NAS.
So I opted to SSH into it and see what's up... Well, there are the remnants of being Pwned... but at the same time I can't even run "top" to see the running processes. The box is screwed up badly.
I went over to my Synology and noticed all I had was a blinking blue light. I went to my laptop and noticed I could still mount the Bonjour (afp) and Samba (smb) shares for all my files. I took this opportunity to find all my external hard drives and start moving off files.
Thankfully, my ESXi hosts could still connect to all the iSCSI and NFS shares. At this point, my home lab environment a lost cause or it could possible be salvaged. The biggest PITA will be setting up vSphere, and View all over again. But it's a shame to lose all my work done with vCAC, vCD, and more. Most of these rely on UUIDs of vCenter and I'm not going to waste all my time trying to hack my way back into a working config. I was also able to follow this two part series How to install ipkg on Synology NAS and How to install an FTP client on a Synology NAS to get a FTP client installed to have multi-threaded means of pushing some stuff off.
After spending a full day trying to FTP the VM files off and having the FTP session continually crap out, I had another idea. Even though the disk activity lights aren't blinking on my Synoloy, I can still spin up my entire vSphere environment. I've always been notoriously bad about backing up my VMs in my home lab (big surprise there) so I thought I might be a chance to back them up and re-deploy them.
I was under the assumption that I would have to completely wipe the NAS and start over. After talking to Synology support, they said:
You will need to perform an operating system reset using the method described here. After doing the reset, make sure the status reported in the Synology Assistant says Configuration Lost or Migratable before going through with the installation. Make sure you also use the latest DSM file for the installation which can be downloaded here. This process is designed to format only system partitions and not data partitions so your volumes and shares will still be intact.
In any case, there is always a possibility of it not working out or all the iSCSI targets gets lost and that screws up my vSphere environment. So I talked to @mattliebowitz and @RickVanover and decided that I could use my NFR Veeam licenses to atleast back up my VMs first. The one great thing I get out of Veeam is deduplication and compression. 1000x better than just trying to FTP the files or save them off. Since there is a possibility that I will lose my entire vSphere environment, I loaded up a Windows 8.1 image on my Fusion environment, did a USB Passthrough to an external drive for the backup target. To try and increase the backup speed, I put my laptop on the same VLAN as my Synology NAS which required some small switch configuration, but I was capping out around 32mb/s. So this is going to take atleast a day to finish.
About 1 day later I finally received a message back from the hacker group. It really wasn't much help, but oh well.
A few days later, all the VMs are backed up. I ran into multiple issues trying to back up directly to the USB drive. So I broke the backup into multiple jobs and then FTPd the backup files to another computer. Safety first.
So now it was time. I followed the directions here to get DSM 5 loaded on the NAS. Grabbed a paper clip, opened it up, held down the reset button for 4 seconds and waited for the beep... *BEEP*... removed it and waited for 2 seconds, then held it down for another 4 seconds, *BEEP*... nothing. The article said the light should be blinking orange but it was still blinking blue. Synology DSM Assistant said "Services starting" but still nothing. So I held down the power button, did a hard shut down, then powered it back on. After another minute the Synology DSM Assistant said "Configuration Lost". Perfect! Went to the web address, loaded DSM 5 and now I'm finally back in business. The one thing that DSM 5 does is make you sign up for MyDS during install. So now I should get all the security updates. :)
On the plus side, all of my data is still there in-tact after the upgrade. The bad part is that I did lose most of the configurations such as my iSCSI target, so I had to re-create all those to make my vSphere environment come alive again. Which wasn't that hard. Copy/Paste the original iSCSI target from the ESXi host into the Synology and I was back in business with my iSCSI LUNs.
Well that was definitely more work than anticipated, but that will teach me to keep my Synology up to date. In the mean time, I've changed the default ports, changed all communication over HTTPS, and did my best to lock it down. I even opened up SSH to make sure and all the pwnd files from before are gone from the / directory.
The last thing I had to tackle was trying to restore my vCenter VM from Veeam because somehow that is the only VM that had some fsck errors on initial boot. Whew.