More Downtime

Posted by Jesse Andrews on Dec 11, 2008

The morning of December 11th, the site went down for a few hours. Back up - read more for info on what happened.

I woke up this morning early to head down to Add-on-con. Unfortunately I got an SMS from pingdom that userscripts.org was down at 7:16AM PDT.

I was on the phone with ServerBeach at 7:20 after the automated reboot didn't work. They said they weren't able to talk to the server either and dispatched someone to the data center. A few minutes ago I got the note that it was up. I sshed in and started up the app.

Worst case scenerio: as I blogged on S3 Saves the Day, I had a backup of the database less than 12 hours old. So if the server was completely dead we would have lost a few hours of data but the site would have survived.

Sorry the frustration - it has been a frazzled morning for me as well :-/

You could comment on this post if you were logged in.

2 points
login to vote
Jesse Andrews article's author
Posted Dec 11, 2008

From ServerBeach:

Your servers primary hard drive was suffering from filesystem errors.
Our data center techs went ahead and booted your server into remotely
accessible single user environment (rapid rescue) and ran a file system
check (fsck) on the primary drive to correct your issue. After running
the fsck, your server successfully rebooted and is now back up and
remotely accessible.

1 point
login to vote
jerone Scriptwright
Posted Dec 11, 2008

Hopefully everything stays working. Good service that pingdom.

1 point
login to vote
Technowise Scriptwright
Posted Dec 11, 2008

Thanks for the quick fix.
This had got me worried for a while because I wanted to update a script, of which I did not have another copy apart from the one uploaded here.