Big Fish Games site down

[GolfHacker]

Wow, never seen that before.

I noticed on my site this morning that all the game graphics I linked to on BFG were missing. So I went to the BFG site, and there's a mostly blank white page that says "Site Maintenance - Big Fish Games is experiencing technical difficulties. We apologize for the inconvenience, please try back later." They've been down for at least a half-hour. I've never seen that happen to a major portal before. Don't these kind of sites usually have backup servers or something...?

[CasualInsider]

Due to a fire in a data center... I'm sure this is a wakeup call to have secondary servers ready in the future if this ever happens again.

[Roman Budzowski]

They're down for at least 6h now...

And when it comes to fire is there anything you can do? You can have backup server, but can it catch the traffic from burned server? Redirecting domains takes some time...

Roman

[Nexic]

Well if they're able to display a "site down" message then they should be able to do a redirect. However, it'd probably be incredibly expense to run a carbon copy of the main servers - much more expensive than what they'd lose during this downtime.

[CasualInsider]

Steam has content servers mirrored at eleven locations in the US alone. I bet BFG could afford to distribute out a bit... or at least partner with a CDN to do it for them. It would at least mean faster download speeds for everyone.

"it seems internal and early estimates from fisher and seattle city light are 5-6 hours till power is restored." The people who host BFG http://twitter.com/bluegeckodba

[Emmanuel]

We will be out for another few hours. This is only the second outage in two years (since we brought everything in-house and stopped relying on third parties that would throw the switch for hours at a time with no notice), but indeed it still is problematic when it happens. The team is working hard at restoring things.

There was indeed a really bad fire at fisher plaza. I hope nobody was hurt.

Ironically I understand that redundancy was already scheduled to be just around the corner for us.

Best regards,
Emmanuel

[GolfHacker]

Interestingly, this apparently happened before in 2006. I found this story while googling for more info on the fire.

I too hope nobody was hurt.

[GolfHacker]

I found this blurb about it on gamezebo, a quote from Big Fish Games founder Paul Thelen:

Big Fish Games has been down all morning due to a fire that broke out at Fisher Plaza, the largest data center in Seattle. Posting on GameBoomers.com, Big Fish Games Founder Paul Thelen said that power was out to the whole building and urged fans to be patient.

"The building is under several feet of water and they will not tell us when it will restore power," Thelen said. "We are in process of transferring all our data to our backup datacenter, but that is a 12 to 18 hour process as we have thousands of servers and massive databases of info to transfer.

"Fisher plaza is the hosting facility for hundreds of companies, so this is a major Internet disruption."

We'll keep you posted on any new developments.

[Indinera]

Hope everybody is okay.

[zoombapup]

thousands of servers?? why not just use a CDN?

[DFG]

thousands of servers??

I wonder what a game company does with all those servers - starting a search engine?

on the serious side, the accident does sound tragic and I too hope there was nobody injured

[Nexic]

I really don't believe it's 1000s of servers just for BFG. He's probably talking about the data center as a whole.

[Backov]

Ya, and it may be that it's their data center - a side business for them or something.

But you never know - all it takes to make a stupid decision like "Hey, let's start our own data center and get thousands of servers all in one NOC to serve our games" is tons of VC money (check), and one foolish CTO (check? No idea.)

[Mark Fassett]

Fisher Plaza also plays home to at least one TV and radio station, both of which are currently broadcasting from somewhere else. It's not just BFG that's been caught by this.

Story

[GolfHacker]

It took the whole day, but it looks like Big Fish Games is back online!

[Maupin]

Oops, looks like by mistake they added the affiliate royalties for Saturday to everyone's account twice this morning.

[cliffski]

Fisher Plaza also plays home to at least one TV and radio station, both of which are currently broadcasting from somewhere else. It's not just BFG that's been caught by this.

Story

I used to work for a banking IT company that had satellite data and satellite uplinks. They even had a fail-safe plan that had a 'backup satellite', that was used by some obscure TV channels. If Our satellite went down, we automatically bumped the TV channels to make way for our data.
Our data was flipping expensive

[Indiepath]

Rackspace Dallas-Fort-Worth also has issues earlier in the week - although they did not release to us customers the exact cause. Good to have an SLA with downtime penalties.

http://www.techcrunch.com/2009/06/29...avorite-sites/

[Game Producer]

Rackspace Dallas-Fort-Worth also has issues earlier in the week - although they did not release to us customers the exact cause. Good to have an SLA with downtime penalties.

http://www.techcrunch.com/2009/06/29...avorite-sites/

I wonder what sort of penalties the bfg hosts would get...