Service Disruption April 5-10, 2025 - RESOLVED

Updated: April 11, 2025

Hi, this is a quick blog post about a service interruption that happened on SIMMER.io and sharemygame.com.

This was our first major service interruption in 8 years. I greatly apologize for the inconvenience.

This was caused by an DDoS attack where a malicious user uploaded many terabytes of malicious data and then created over a billion download requests.

During the mitigation, we migrated to a new storage provider for games. All games upload and load up significantly faster than in the past. A silver lining!

Current Status

All games are now playable! Issues? Let us know in #community-support on discord.
- On April 5-9, Games that were uploaded between Feb 15 - April 6 would not load. Resolved April 9.
User uploads are back. We added captchas, rate limiting and admin alerts to deter hackers. We also added an "uploads kill switch" for simmer admins to enable if we encounter a similar DDOS. This would stop all uploads and prevent further damage.
No credit card or personal data leaked.
Unresolved: New uploads are disabled for legacy.simmer.io, sharemygame.com
- sharemygame.com is gamedev.tv's version of simmer.io that is on our insecure legacy infrastructure, and we cannot easily re-enable uploads. We encourage sharemygame users to directly share to simmer.io for now.
- Planning a release of SIMMER 2.0 powered sharemygame.
- legacy.simmer.io (simmer 1.0) will be wind down next week.

Summary of the Attack

Someone uploaded 41TB of data to simmer's Backblaze cloud servers (game storage). They exploited the fact that we allowed "unlimited uploads" This occurred March 15-April 5 at a rate of about 100MB/s. This was almost certainly done with a script. The files were 100MB binary files that appeared to be encrypted. They created over 1 Billion download requests after the files were uploaded.

This resulted in 429 "Too many requests" errors for our legitimate games, which was brought to our attention via support requests on April 5. We shut down uploads on April 5, and made our storage bucket private on April 8, leading to games that were uncached on our CDN not to load.

We migrated the files to a new service on April 9 and all games became playable again.

But that wasn't the end of it.

On April 9, we also noticed skyrocketing usage on our google cloud account, the one that hosted games uploaded to SIMMER 1.0. The attacker had figured out a similar strategy on legacy.simmer.io, and uploaded 125TB of data to that account. There were no download requests, thankfully, so we likely deleted the files before significant financial damage to SIMMER could occur.

On April 10, we finished improvements to SIMMER to rate limit uploads, and add a captcha to prevent automated attacks. We turned back on uploading at 8PM. We have alarms and a kill switch for uploads if another attack occurs.

I'm still working with Backblaze to hopefully get a bill reduction.

Side note: I need a beer.

-Rocco

Timeline

Times are in PDT.

April 5, 2025

5:00PM Checked simmer support email and got two support requests that games weren't loading.
6:00PM Discovered that our cloud hosting provider was sending "429 Too Many Requests" responses to certain game files (rate limiting).
6:30PM Reached out to cloud service provider for details / fix.
7:00PM Put a notice on the site and disabled new uploads.
10:00PM Games back online that were uploaded previous to SIMMER 2.0
- switched to the old cloud service provider.
- new games (post Mar 2025) still not loading because they're exclusively on the new cloud provider.
- uploads still disabled.
- still awaiting info from cloud service provider.
11:30PM Making progress. Looks like it was a DDOS attack by a single user. Deleting the account, making notes about future safeguards.

April 6, 2025

10:00AM:
- Malicious account data deletion continues. Malicious user uploaded an extreme amount of data.
- Cloud provider ticket still open, but finally assigned to a support rep. It appears that "429 Too Many Requests" issue still occurs and games March 2025+ will not load.
4:00PM Malicious data deletion completed.
4:30 PM: Got reply from cloud provider. Helpful info, but it did not resolve the file serving issue. Replied almost immediately--hoping to get a resolution soon.
5:30 PM: I've been working all day on malicious use detection so that I can re-enable uploads, but it still requires that the cloud provider unlocks access to the data store.
8:00PM: Waiting on further reply from host. Implemented Captcha's so that we're ready for uploads once host unblocks us.

April 7, 2025

10:00AM: I added another nudge on my support request, and found someone through my personal network that works at Backblaze, the cloud service company, and emailed them.
- As a side note, they wrote a blog article about simmer awhile back.
- I may need to consider migrating to a new service if they can't get this resolved soon, but my hands are still tied because the files are locked with "429 Too Many Requests" errors, so there's not much I can do on my side. I can't copy the files to a new service.
5:00PM: Unfortunately Backblaze support is not answering my support requests in a reasonable timeframe. Many (but not all) files are still locked.
- At this point I'm beginning Plan B, which is to switch to a new provider for new uploads. I'm starting to investigate other hosting providers, most likely Cloudflare R2. It may take some time to get this up and running :-(.

April 8, 2025

8:00AM: Working on Cloudflare R2 Integration, will migrate from current host once the files get locked.

11:30AM: Sent out email to users about the outage. Have 4:30PM call with someone more senior at Backblaze, the hosting company.

4:00PM: Backblaze support got back to me. At their request, I set the storage bucket containing March 2025+ games to private, so that I can migrate to another service. This will disable ALL games uploaded March 2025+.

Hopefully this will stop the throttling that is occurring on the Backblaze side.
Intention: Stop the DDoS attack, migrate to Cloudflare R2.
My initial test of an upload with Cloudflare R2 worked. I'm thinking I can get that new bucket up and running by mid-day tomorrow.

6:00PM: The files are unlocked after setting the Backblaze bucket to private! Began migrating data from Backblaze to Cloudflare R2

April 9, 2025

12:30PM Discovered a new similar attack on legacy.simmer.io and sharemygame.com (gamedev.tv's version of simmer, which is still on legacy). Disabled uploads there.

My projection of getting everything sorted by tonight at 7:00PM might be delayed due to this.

3:00PM Malicious user uploaded 1.5 million files totaling about 60TB to legacy.simmer.io bucket between 2025-04-08 00:57:34.819000 and 2025-04-09 03:07:06.870000 and . I wrote a script to find the malicious objects and am going to proceed with deletion momentarily.

6:30PM Data fully migrated from Backblaze to Cloudflare. All games should load!

Uploads projected to resume April 10, 7:00PM PST.
sharemygame.com will not accept uploads for some time, perhaps a week.
I will shut down legacy.simmer.io soon. All games will be playable on simmer.io.
Still deleting files from second attack on legacy.simmer.io and sharemygame.com

April 10, 2025

10:00AM: All data has been migrated to new server (Cloudflare R2). Bonus! Games are loading much faster. All malicious data deleted. Monitoring.

11:30AM: I'm working on turning uploads back on. Still hoping for tonight after I get all precautions in.

8:00PM: We're back baby! See the notes in 'Current Status'.

April 11, 2025

7:00AM A user sent a bug report that uploads were not working with the error "upload_lock_bypass cannot be null".

9:30AM That was a bug on our side and should be resolved now.