Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
AppleNova went unavailable from 04:30 to 14:30 EST today. It looks like one of our backup scripts got stuck in a bad way and decided to fill up one of the disks to 100% usage. I've cleared out the gunk and hopefully everything should be running smoothly again.
The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
β½
|
I was just gonna hit 'submit', but decided to check the Feedback forum another time and saw your thread.
Glad to see it back. |
quote |
Veteran Member
Join Date: May 2005
Location: Paris, France
|
Thanks, Brad!
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
I'm also making another partition now and modifying our backup script to archive to that one instead of the main partition. If it goes nuts again, at least it'll just mess up the backups and hopefully not bring down the site.
The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
M AH - ch ain saw
Join Date: May 2004
|
Thanks Brad!
|
quote |
Hoonigan
Join Date: May 2004
Location: Canada
|
I don't realize how often I check in here until it goes down.
Brad is the man, etc. |
quote |
Dark Cat of the Sith
|
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
See, this is the reward I get for staying up late and sleeping in on Pi Day. I love this job.
Archived files and the backup script have now been shuffled around so not to impact the main DB again. If you experienced a whole lot of slowness a few minutes ago, that's because the disk was thrashing while copying files around. There may be intermittent periods of slowness over the next hour as I'm poking and prodding a few areas of the server. The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
Thunderbolt, fuck yeah!
Join Date: Jan 2005
Location: Denmark
|
Was it a rounding error?
|
quote |
9" monochrome
Join Date: May 2004
Location: π¦πΊ
|
Echoing murbot and Capella here... cheers Brad.
|
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
I don't post much these days but I'm drunken so what better thread to post in!
Pi day is awesome, in Korea it's actually white day also, which is the campanion to valentines day, it's complicated. Anyway, sweeings as in drunken and on an iPhone, happy pi day everyone ^^ |
quote |
geri to my friends
Join Date: May 2006
Location: Heaven
|
I missed not getting my fix until now.
Thanks Brad. |
quote |
Formerly CoachKrzyzewski
|
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
Thanks for getting us running again Brad.
Mind if I ask a more technical side question? How did you get into the server to clear the space and get it running again? I've encountered a similar issue with my server but it was a local server and I was able to reboot it and access it via CLI. Louis L'Amour, βTo make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.β Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
quote |
Stallion
Join Date: Feb 2006
Location: Milwaukee
|
Fine work Bradley. You're both a gentlemen, scholar and nerd extraordinaire
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
Quote:
This server had three partitions (now a fourth specifically for backups and logs): one for the OS, home dirs, software, etc.; one for the running MySQL database files; and one for the web files (vBulletin, html, css, images, etc). What happened this morning was that the script I wrote for backing up and archiving the database had been writing its files to the same partition as the database itself. That was by design because the DB partition was the largest partition, and the script normally cleans up fine after itself. This time, however, it didn't and the DB partition filled up to 100%. Fortunately, with my Linode account I had a few GB of unallocated disk space and was able to use that on a new partition, copy the archives/backups to that partition, delete them from the old partition, and symlink some directories so the backups would automatically be written to the new partition. The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
|
quote |
Lord of the Rant.
Formerly turtle2472 Join Date: Mar 2005
Location: Upstate South Carolina
|
Quote:
I'm in a Unix class now so I actually know how to do more of the features needed to regain my system back. My Dad was sending big files and I didn't spend much time clearing off the drive. It filled to 100% with all on the same single partition. I'm currently running my server with 1TB RAID but I keep it clear now so I don't have that issue anymore. Louis L'Amour, βTo make democracy work, we must be a nation of participants, not simply observers. One who does not vote has no right to complain.β Visit our archived Minecraft world! | Maybe someday I'll proof read, until then deal with it. |
|
quote |
Space Pirate
Join Date: May 2004
Location: Atlanta
|
I was claiming credit over on Twitter..... I SSH'ed in and SSH'ed all over everything. It was just a script all this time.
... |
quote |
Veteran Member
|
|
quote |
Posting Rules | Navigation |
|
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Unplanned site outage, 2009-12-09 | Brad | Feedback | 5 | 2009-12-10 11:46 |
Unplanned Site Outage, 2009-10-27 | Brad | Feedback | 17 | 2009-10-29 07:29 |
Unplanned Downtime 2008-09-09 | Brad | Feedback | 5 | 2008-09-09 18:13 |
Downtime | alcimedes | Feedback | 8 | 2007-02-28 22:31 |
Downtime (22/23 Nov) | Brad | Feedback | 21 | 2005-11-28 18:54 |