Meteoplug Server temporarily down **solved**

[admin]

Meteoplug Server was victim of a hacker attack and is currently down. We managed to get control back and it seems to be a classical ransomware approach. Interestingly it seems not to be driven by security holes in the OS or applications (the most common approach) but by hacking IPMI side channel access.
It will need us to restore the server completely. Unfortunately, I don't have physical access to it until Monday next week. We are still in the phase to access the complete damage picture, but as far as it looks right now, we can restore all data and applications as they have been on August 21.

Complying to GDPR we store only an absolute minimum of user data on the server. This is just the email address and account name (which is a artificial name you gave during account creation) and account password, which is only stored encrypted. Therefore, your passwords did not leak, just email addresses could in theory be read out of the database but we have no indication that this happened. No payment information is hold on the server, no addresses, no phone numbers or alike.

Bad news is that this downtime will last until next week, good news is that all data can be restored (as fas as we understand the impact right now). I am really sorry for that, but the IPMI attack vector through a side channel was not in our main security focus. I will keep you posted, how things develop the next days. We will compensate for the outage by adding a month onto the users subscription period.

Last but not least, be assured that not a single dollar will be paid to criminals.

[laurentw]

Hi,
Thanks for information and good luck for recovery. Kind regards.

[bubulino]

Danke für die Informationen und viel Erfolg beim Wiederherstellen.
Wir drücken die Daumen!

Bubulino

[gcams]

Many thanks for the update Boris, it's greatly appreciated. That is good news there will be no loss of weather data. Good luck with the restore process and thanks for your efforts in restoring the service next week.

[admin]

Too early to say that we are back, but it looks at least like receiving data from clients and storing that is online again. I will checks graphs etc tomorrow.

[frabey]

Mi.25.08.21, 6Uhr, Ich schau auf den Monitor und Meteoplug scheint wieder zu laufen, jeden falls sind die Grafiken etc. auf meiner Website wieder da. Super.
Vielen,Vielen Dank an Boris das es nun doch so flott ging.

[lorenz1982]

Seit gestern Abend funktioniert wieder alles auch auf meiner Website.

Vielen Dank.

[bubulino]

Super, vielen Dank für den ausserordentlichen Einsatz!

Top Arbeit.

[gcams]

Too early to say that we are back, but it looks at least like receiving data from clients and storing that is online again. I will checks graphs etc tomorrow.

What wonderful news to wake up to! Thank you so much Boris for working so hard on restoring service. I have no doubt you have spent many hours (and probably not a lot of sleep) getting this working again! I'm incredibly grateful!

giphy.gif (119.34 KiB) Viewed 11183 times

[admin]

As far as I can see we are back to normal operation. I am a bit handycapped as I am on vacation with my family until end this week and I had to deal with my laptop only from a hotel room.

I will check next week, if I can refeed a bit more of the data that did not make it into the database. Perhaps the data gap can be made even a bit smaller by that. And I will add one moth to all user subscriptions.

I want to say sorry that this happened, but I did not had a security issue in the IPMI on my radar.

A few takeaways if you also run server in the internet:

1) Have a second account with root access apart from "root". This allowed us to get into the hacked system as they just gave root a new password.

2) If you are using KVM solution IPMI restrict the IP range that is allowed for access to your provider's IP range or the IP of another server of yours in the Internet and use this as a proxy. IPMI is insecure, just doing password protection on it is not sufficient. It is a shame what supermicro offers here.

3) Junglesec does not have an appetite to encrypt large tgz or zst files. When you have tars of your complete system in that format chances are good that you can restore from there. Using LVM snaphots gives you the chance to tar the root volume during operation without risking inconsistencies.

4) To have a good friend that is a super pro on all Linux, security and networking helps a lot. 1000 thanks to my buddy Uwe who kicked junglesec and their hidden installs from the server.

[Rutishauser]

Vielen herzlichen Dank für diesen ausserordentlichen und zeitintensiven Einsatz! Ein dickes Lob für die schnelle Behebung.

Grüsse Bruno

[Stormedy]

Dear Boris!

many thanks for your effort and support for Meteoplug. Yesterday night all of a sudden my complete weather page was showing a perfect picture! Including graphs Many of my supporters in Switzerland are more than happy - thanks again for your help and restoring everything!

best regards and hope that everything is in good shape again!

Edward Siber

[admin]

As a final step I applied 31 additional days to all users with valid licences to compensate for the outage.

[gcams]

As a final step I applied 31 additional days to all users with valid licences to compensate for the outage.

Thank you very much Boris, I think this is very fair compensation. It was kind of you to do this!