User Name
Password
AppleNova Forums » Feedback »

Unplanned downtime, 2010-03-14


Register Members List Calendar Search FAQ Posting Guidelines
Unplanned downtime, 2010-03-14
Thread Tools
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2010-03-14, 13:32

AppleNova went unavailable from 04:30 to 14:30 EST today. It looks like one of our backup scripts got stuck in a bad way and decided to fill up one of the disks to 100% usage. I've cleared out the gunk and hopefully everything should be running smoothly again.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
chucker
β€½
 
Join Date: May 2004
Location: near Bremen, Germany
Send a message via ICQ to chucker Send a message via AIM to chucker Send a message via MSN to chucker Send a message via Yahoo to chucker Send a message via Skype™ to chucker 
2010-03-14, 13:33

I was just gonna hit 'submit', but decided to check the Feedback forum another time and saw your thread.

Glad to see it back.
  quote
Dorian Gray
Veteran Member
 
Join Date: May 2005
Location: Paris, France
 
2010-03-14, 13:33

Thanks, Brad!
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2010-03-14, 13:36

I'm also making another partition now and modifying our backup script to archive to that one instead of the main partition. If it goes nuts again, at least it'll just mess up the backups and hopefully not bring down the site.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Maciej
M AH - ch ain saw
 
Join Date: May 2004
 
2010-03-14, 13:39

Thanks Brad!
  quote
murbot
Hoonigan
 
Join Date: May 2004
Location: Canada
 
2010-03-14, 13:46

I don't realize how often I check in here until it goes down.

Brad is the man, etc.

  quote
Capella
Dark Cat of the Sith
 
Join Date: Jun 2007
Location: Rochester, NY
Send a message via AIM to Capella  
2010-03-14, 13:48

Quote:
Originally Posted by murbot View Post
I don't realize how often I check in here until it goes down.
This.

Quote:
Originally Posted by murbot View Post
Brad is the man, etc.
And this. Thanks, Brad!
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2010-03-14, 13:58

See, this is the reward I get for staying up late and sleeping in on Pi Day. I love this job.

Archived files and the backup script have now been shuffled around so not to impact the main DB again. If you experienced a whole lot of slowness a few minutes ago, that's because the disk was thrashing while copying files around. There may be intermittent periods of slowness over the next hour as I'm poking and prodding a few areas of the server.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Mugge
Thunderbolt, fuck yeah!
 
Join Date: Jan 2005
Location: Denmark
 
2010-03-14, 14:00

Was it a rounding error?

  quote
Mac+
9" monochrome
 
Join Date: May 2004
Location: πŸ‡¦πŸ‡Ί
 
2010-03-14, 14:14

Echoing murbot and Capella here... cheers Brad.
  quote
AsLan^
Not a tame lion...
 
Join Date: May 2004
Location: Narnia
 
2010-03-14, 14:53

I don't post much these days but I'm drunken so what better thread to post in!

Pi day is awesome, in Korea it's actually white day also, which is the campanion to valentines day, it's complicated. Anyway, sweeings as in drunken and on an iPhone, happy pi day everyone ^^
  quote
thegeriatric
geri to my friends
 
Join Date: May 2006
Location: Heaven
 
2010-03-14, 15:43

I missed not getting my fix until now.

Thanks Brad.
  quote
evan
Formerly CoachKrzyzewski
 
Join Date: Jan 2006
Location: Charlottesville, VA
Send a message via AIM to evan  
2010-03-14, 16:07

Quote:
Originally Posted by thegeriatric View Post
I missed not getting my fix until now.

Thanks Brad.
ditto
  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2010-03-14, 17:58

Thanks for getting us running again Brad.

Mind if I ask a more technical side question? How did you get into the server to clear the space and get it running again? I've encountered a similar issue with my server but it was a local server and I was able to reboot it and access it via CLI.

Louis L'Amour, β€œTo make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
Partial
Stallion
 
Join Date: Feb 2006
Location: Milwaukee
 
2010-03-14, 18:08

Fine work Bradley. You're both a gentlemen, scholar and nerd extraordinaire
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2010-03-14, 18:36

Quote:
Originally Posted by turtle2472 View Post
Mind if I ask a more technical side question? How did you get into the server to clear the space and get it running again? I've encountered a similar issue with my server but it was a local server and I was able to reboot it and access it via CLI.
Don't mind at all! For me, it was as simple as SSHing into the server and rsyncing/rming the files. Were you not able to SSH into your box when its disk filled up? I've been able to SSH into machines where the root OS partition literally had zero bytes free. It's usually slow and coincides with lots of errors in log and history files, but it should work.

This server had three partitions (now a fourth specifically for backups and logs): one for the OS, home dirs, software, etc.; one for the running MySQL database files; and one for the web files (vBulletin, html, css, images, etc). What happened this morning was that the script I wrote for backing up and archiving the database had been writing its files to the same partition as the database itself. That was by design because the DB partition was the largest partition, and the script normally cleans up fine after itself. This time, however, it didn't and the DB partition filled up to 100%. Fortunately, with my Linode account I had a few GB of unallocated disk space and was able to use that on a new partition, copy the archives/backups to that partition, delete them from the old partition, and symlink some directories so the backups would automatically be written to the new partition.

The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting.
  quote
Robo
Formerly Roboman, still
awesome
 
Join Date: Jul 2004
Location: Portland, OR
 
2010-03-14, 19:35

Quote:
Originally Posted by Brad View Post
Don't mind at all! For me, it was as simple as SSHing into the server and rsyncing/rming the files...
*head explodes*
  quote
Brad
Selfish Heathen
 
Join Date: May 2004
Location: Zone of Pain
 
2010-03-14, 19:46

  quote
turtle
Lord of the Rant.
Formerly turtle2472
 
Join Date: Mar 2005
Location: Upstate South Carolina
 
2010-03-14, 20:17

Quote:
Originally Posted by Brad View Post
Don't mind at all! For me, it was as simple as SSHing into the server and rsyncing/rming the files. Were you not able to SSH into your box when its disk filled up? I've been able to SSH into machines where the root OS partition literally had zero bytes free. It's usually slow and coincides with lots of errors in log and history files, but it should work.

...
Thanks for sharing. Well, I was local and able to log in via the keyboard so I'm sureI could have SSH'ed in on my network. The biggest issue I had was at the time I didn't know CLI on *nix. It was horrible not having a clue how to delete a file without GUI.

I'm in a Unix class now so I actually know how to do more of the features needed to regain my system back. My Dad was sending big files and I didn't spend much time clearing off the drive. It filled to 100% with all on the same single partition. I'm currently running my server with 1TB RAID but I keep it clear now so I don't have that issue anymore.

Louis L'Amour, β€œTo make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.”
Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it.
  quote
drewprops
Space Pirate
 
Join Date: May 2004
Location: Atlanta
 
2010-03-14, 21:49

I was claiming credit over on Twitter..... I SSH'ed in and SSH'ed all over everything. It was just a script all this time.


...
  quote
scratt
Veteran Member
 
Join Date: Jul 2004
Location: M-F: Thailand Weekends : F1 2010 - Various Tracks!
Send a message via Skype™ to scratt 
2010-03-14, 22:07

Quote:
Originally Posted by drewprops View Post
I was claiming credit over on Twitter..... I SSH'ed in and SSH'ed all over everything. It was just a script all this time.


...
We were just humouring you, assuming you were drunk in control of a Twitter account again.
  quote
Posting Rules Navigation
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Post Reply

Forum Jump
Thread Tools
Similar Threads
Thread Thread Starter Forum Replies Last Post
Unplanned site outage, 2009-12-09 Brad Feedback 5 2009-12-10 11:46
Unplanned Site Outage, 2009-10-27 Brad Feedback 17 2009-10-29 07:29
Unplanned Downtime 2008-09-09 Brad Feedback 5 2008-09-09 18:13
Downtime alcimedes Feedback 8 2007-02-28 22:31
Downtime (22/23 Nov) Brad Feedback 21 2005-11-28 18:54


« Previous Thread | Next Thread »

All times are GMT -5. The time now is 06:48.


Powered by vBulletin®
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - 2024, AppleNova