Geocaching.com down?

[Guanajuato]

Is anyone else having problems connecting to Geocaching.com. I keep getting a DNS 'Cannot find Server' error. Don't know if its this end or elsewhere. But it seems to be the only site I'm having trouble with.

[Stokesy]

Yep, me too

[Watford Wobble]

Me to and I'm on for three FTF this afternoon :mad:

[Stokesy]

The Forums are down too, and they are on a different server.....strange

[adsandco]

yep

[Guanajuato]

I'd guess at a power cut or something similar.

Maybe they've forgotten to feed the Hamsters.

Hope its only temporary - we're hoping to go to Morecambe tomorrow (That's living the high life!) and I've not got the details of the caches there yet.

[gazooks]

Same here - got a DNS error !! :confused:

[Mr.Coffee and the Coffee Clan]

gc.com is down in Australia and New Zealand as well. It appears to be world wide. I have sent a message via the iPhone tester group to see just how wide this is.
Regards Mr.Coffee and the "withdrawl symptons" Coffee Clan.
Melbourne, Australia

[andyh3930]

I think it more than a powercut at their site. The DNS record has gone. This means that when you type in the webaddress it doesn't know what IP Address to convert it too.
Also googling geocaching.com and IP Address returns an IP Address that at some point was used by geocaching.com and that is timing-out and not loading. They may be moving servers or hosting companies.

[The Gray Man]

Everything related to Groundspeak is down - geocaching.com, waymarking.com, whereigo.com, groundspeak.com, forums.geocaching.com... I would imagine they have some catastrophic failure somewhere on their hosting net.

What concerns me is that, as it's July 4th tomorrow - which I believe is some small local holiday for those colonials across the pond :p - they may have radically reduced resources looking at the problem. :wacko:

[Matthew 7:7 Too]

On the other hand, I would imagine it would be a peak time for American geocachers to be out doing their thing? So the pressure must be on to ensure the site is back up to support them. (Or I suppose it could have been crashed by the number of users trying to access it? The GC forums were so slow last night that I gave up trying to access threads.

Chris

[Hellspaw]

Yup no access, just when I can skip out of work early [}:)]

[Mr.Coffee and the Coffee Clan]

Posted: 3 July 09 8:25 pm Post subject: Reply with quote
Message from Groundspeak: We are currently experiencing technical problems. We are working hard to fix it. Please accept our apologies groundspeak.com

[Pharisee]

At times like this, aren't you really glad that you keep your GSAK database of all the UK caches fully updated :applause:

[Stokesy]

At times like this, aren't you really glad that you keep your GSAK database of all the UK caches fully updated :applause:

I sometimes wonder why I have so many PQ's running for areas I seldom visit...now I know why!

Hail GSAK! :applause::applause::applause:

[Mr.Coffee and the Coffee Clan]

At times like this, aren't you really glad that you keep your GSAK database of all the UK caches fully updated :applause:
GSAK No1, iPhone gc app saved offline caches Excellent!.
All this does pose two questions:, Does gc.com backup their databases off shore and does gc.com have a backup plan/system

[Watford Wobble]

Looks like I'll have to cut the grass now :mad:

[gazooks]

Looks like I'll have to cut the grass now :mad:


Its just started raining here - no chance of thet either :D:applause:

[wizardsmum]

Great, just when we had planned to go out after work. Hope it comes back on soon. Does any one have saved the co-ords and hints they could give me for Nos 1 - 7 of the devil dyke ones by 'sad exploration ' ? (DDC 08 Fork left )

[Watford Wobble]

I think the hint was 'ivy'.:p

[Pharisee]

Great, just when we had planned to go out after work. Hope it comes back on soon. Does any one have saved the co-ords and hints they could give me for Nos 1 - 7 of the devil dyke ones by 'sad exploration ' ? (DDC 08 Fork left )
Yes... email sent :)

[Mr.Coffee and the Coffee Clan]

There is a rumor going around a few forums that the gc.com website was hacked...Only a rumor though.

[Bear and Ragged]

Apparently...
Elias had 'tweaked' the forums to 'run faster...'

:eek:

[keehotee]

I can't get on today - and have had problems with the forums for the past week or so :(

[wizardsmum]

Thanks Praisee.

[Brenin Tegeingl]

I've just had it officially confirmed that there is a power outage affecting the whole building where the servers are located. So it's not a Groundspeak equipment issue, but one involving multiple companies. The problem is being worked on at the moment.

It's a case of No Power, means No Servers, which means No Site. :(

Deceangi

[DrDick&Vick]

Yup problems problems & problems. Got a great phone and software (GCz) just got a cr@p website that has problems to often. Wanted to check a cache, got the details on the PDA & the Garmin but wanted to check and what do I get Zilch and I pay to be a premium member for this service.
Oh well just read Deceangi's post but then again it is not the first time I have had problems accessing the site and it won't be the last, maybe it is time to change the hamsters for cheetahs.

[von-horst]

We've got some friends who are going away this weekend;

Does anyone have a PQ covering the M5 from Bristol to Exeter?

Does anyone have a PQ covering Exeter andthe surrounding area?

Many thanks in advance for any help,


Mike

[keehotee]

Yes - I've got an Exeter to Bristol PQ...... pm me and I'll email it on.

[von-horst]

PM sent - many thanks!

Mike

[keehotee]

Email sent - glad to help :)

[The Hornet]

Guess who's feeling quite smug sitting in the car on top of Watership Down having just found and logged a great Terracache (https://www.terracaching.com/viewcache.cgi?ID=TCDB2)!:D

[Dakar4x4]

Just followed that Terrorcaching link...

"An internal server error occurred. Please try again later."

So much for Terrorcaching? :applause::applause:

Jon.

[sTeamTraen]

There is a rumor going around a few forums that the gc.com website was hacked...Only a rumor though.
Well, that could have been quickly eliminated as a cause by noting that neither the geocaching.com nor groundspeak.com domain names are resolving to an address. So the problem is at a level well beyond "a web site being hacked".

My information (as unreliable as anyone else's) is that there was a fire which has affected the electrical supply of the building in which Groundspeak's servers are just one of many, many companies which are "colocated". So the building has no power (but apparently the server rooms were not affected). You can bet that a lot of people who are way more influential than Groundspeak will be "encouraging" the local utility people to get this fixed very quickly.

It would be interesting to see (not that I imagine we will ever find out) whether anybody's insurance pays for resulting financial losses. Groundspeak's will probably be fairly minimal, but if you're selling stuff online, a few hours of downtime can be big bucks in lost revenue.

Not the greatest time for it to happen, though: any Friday is bad enough as people plan their weekend, but also it's the Fourth of July holiday tomorrow. I suppose it could have been even worse if that had fallen on a Monday.

[Mrs Blorenge]

Apparently...
Elias had 'tweaked' the forums to 'run faster...'

:eek:

After Elias's tweaking sometime during our Wednesday night / Thursday wee small hours I found that the whole forum was running much better through yesterday, so I'm glad it wasn't him who broke it by fiddling too much :D

Anyway, if it's just a localised power outage we'll just have to sit back and wait...

Community sing-song anyone?

[Stokesy]

Jeremy Irish is "twittering" if anyone is interested....

http://twitter.com/locuslingua

[jacksomd]

Of course, this is just typical as we are off to Wales early tomorrow for a week's holiday!

I don't suppose anyone has a PQ for the areas around Shell Island?

Thanks,

Marcus

[Stokesy]

Of course, this is just typical as we are off to Wales early tomorrow for a week's holiday!

I don't suppose anyone has a PQ for the areas around Shell Island?

Thanks,

Marcus

Where's Shell Island? :p
I have a PQ for North Wales that's recent

[DrDick&Vick]

We've got some friends who are going away this weekend;

Does anyone have a PQ covering the M5 from Bristol to Exeter?

Does anyone have a PQ covering Exeter andthe surrounding area?

Many thanks in advance for any help,


Mike

I've got one that covers from Exeter to Okehampton PM me

[jacksomd]

It's on the coast (obviously) south of Porthmadog. The nearest town is Llanbedr.



Any help anyone can give would be greatly appreciated!

Marcus

[Simply Paul]

Glad it's not just me. Annoying though, as I have a lot of TBs and coins to log, and caches too. Oh well, I'll find something else for my evening's entertainment and hope years of logs and photos haven't been lost forever... :ph34r:

[Stokesy]

It's on the coast (obviously) south of Porthmadog. The nearest town is Llanbedr.



Any help anyone can give would be greatly appreciated!

Marcus

PM me with an e-mail address and I'll send you what I've got, hopefully it will help

[Saddlesore1000]

So what disaster recovery planning was done by Groundspeak for this eventuality? None it would seem.

We are going to North Wales this weekend, luckily we have PQs for the area we are visiting - we planned ahead - Groundspeak take note, smarten your act up please!

[Family_Moose]

Oh for Frak sake! Just returned fromwork to start planning the weekend caching...... :mad:

[sTeamTraen]

So what disaster recovery planning was done by Groundspeak for this eventuality? None it would seem.

We are going to North Wales this weekend, luckily we have PQs for the area we are visiting - we planned ahead - Groundspeak take note, smarten your act up please!
A lot of companies with a lot more to lose will have done a lot less. Groundspeak's hosting company has off-site backups in case the whole server farm burns down. But duplicating the entire server setup is an awful lot of money - you'd have to mirror the database in real time across a pretty big distance (not much point rolling back 24 hours and telling people to log on that version for a day or two until you get the live version back, because someone's going to lose their logs and recently published caches however you do it).

There are techniques to cope with all this, but they tend to be mostly used by banks, who take more than $30 a year off their customers. I think that the day when geocaching has to be high-availability is some way off yet.

[Brenin Tegeingl]

A quick search has produced details of the extent of the damage, which has knocked out larger companies than Groundspeak

http://www.datacenterknowledge.com/archives/2009/07/03/major-outage-at-seattle-data-center/

Deci

[Saddlesore1000]

A lot of companies with a lot more to lose will have done a lot less. Groundspeak's hosting company has off-site backups in case the whole server farm burns down. But duplicating the entire server setup is an awful lot of money - you'd have to mirror the database in real time across a pretty big distance (not much point rolling back 24 hours and telling people to log on that version for a day or two until you get the live version back, because someone's going to lose their logs and recently published caches however you do it).

There are techniques to cope with all this, but they tend to be mostly used by banks, who take more than $30 a year off their customers. I think that the day when geocaching has to be high-availability is some way off yet.

Sounds like they need to change to a better prepared hosting company. DR sites are set up for this situation, I'd be prepared to pay a little more for Groundspeak to be better prepared in the future.

[Speakers-corner]

Found this report on the net

Major Outage at Seattle Data Center July 3rd, 2009 : Rich Miller

Many sites hosted at Seattles Fisher Plaza are offline amid widespread reports of a significant power event at the facility. The largest impact has been downtime at the payment portal Authorize.net, which has been
offline for hours, meaning that its merchants have been unable to process credit card payments through their web sites. We are down due to a fire at our primary data center in Seattle, Authorize.net reports on its Twitter account. We are working to restore services, but no ETA at this time. AdHost chief technical officer Michael Smith has posted a status report on the home page of the hosting provider, whose data center is at Fisher Plaza. Beginning at approximately 11:18 PM on July 2nd and continuing through the present time Fisher Plaza experienced a significant power event that required all power systems including street power, UPS, and Generator power to be completely shut down in Plaza East, Smith reports. The event is ongoing and, at this point, we do not have a ETA for service restoration. Other tenants at the facility say they now expect their sites to be down all day. NANOG mailing list reports that the incident triggered the sprinkler system in an
electrical equipment room. As noted in the NANOG report, water damage could complicate recovery efforts. This is the second major outage at Fisher Plaza in the past year. Last June many sites at the building were offline after an electrical event at the building. Fisher Plaza previously had a major outage in 2006 (more details here on that incident). The owner of the building, which also houses the operations of Seattle TV and radio stations, has been attempting to sell Fisher Plaza.

[sTeamTraen]

Sounds like they need to change to a better prepared hosting company. DR sites are set up for this situation, I'd be prepared to pay a little more for Groundspeak to be better prepared in the future.
We looked at DR for my work some time ago. Most of the SLAs are expressed in days, not hours, and the prices are expressed in hundreds of thousands. You're paying for them to keep a hot spare of any server which you might have. And even then you are rolling back to a previous version of your data.

It's possible to run a cluster with immediate, redundant failover across multiple cities, but your cost per transaction goes up by a factor considerably greater than 2, and it isn't worth doing unless your business model needs reliability with a lot more 9s after the decimal point than Groundspeak's does.

The reliability of the site's software has always been the major source of downtime, so spending money on making it (literally) bombproof probably isn't the most rational choice.

[SaltercreaseRangers]

you can follow @locuslingua for updates on Groundspeak status regarding the current outage

[scottpa100]

Fail over site recovery costs massive money. This is a project that we are just starting in work. I work for a modest sized county council and the cost with infrastructure, hardware, software licensing, support, maintenance, additional utility costs just to keep 'key' systems up and running (payroll, e-mail, social services, education etc etc) are big numbers. We're looking at over £100,000. And we get preferential Government pricing rates! Remember, this is not just a spare box to switch on again in someone's bedroom. This is BANG, your data centre has gone, I'll have my key systems up and running in the next 5 - 10 minutes at this other data centre.

That's what we're all asking for here, and $30 per annum doesn't go very far at all. Hell, people complain about the $30 (£22 for me this year) as it is!

[Middleleaze Moles]

We are off to Cornwall tomorrow and desperate to plan some caches!!! What a nightmare! Staying at TR20 8YN so anyone who has a PQ for that area would be the bestest friend ever!!!!!:cheers:

[SaltercreaseRangers]

New generators arrived on site at the data centre, maybe we'll be back up by the morning

[Middleleaze Moles]

New generators arrived on site at the data centre, maybe we'll be back up by the morning
Excellent news!!!!:cheers:

[Stokesy]

We are off to Cornwall tomorrow and desperate to plan some caches!!! What a nightmare! Staying at TR20 8YN so anyone who has a PQ for that area would be the bestest friend ever!!!!!:cheers:

I ran a load of cornwall PQ's last week, I'm not sure which one would cover the area you need, but I'm happy to forward on all 6 if you want.

PM me if you want them

[Pharisee]

My GSAK database was updated yesterday. It covers the whole of the UK. I'm not prepared to swap or share PQs but I can generate a search of my database and create a GPX file if anyone is desperate. Not too big an area, though or the file will be too big to email. E-mail me... !

[Mr.Coffee and the Coffee Clan]

For those that want to be kept up to date, just keep checking Jeremy's Twitter Feed. (no...you don't have to have a twitter account)

http://twitter.com/locuslingua

He has also done a blog post about what has been going on and how GC are affected. http://locuslingua.blogspot.com/2009/07/fail-fisher-plaza-outage.html


When I get a phone call at 3am it is one of two things - either a relative has passed away or a server has gone down. Looking at those two options I would always take the latter. Bryan was on Caller ID, so I knew it was the latter. What I didn't realize was that this wasn't the ordinary check, reboot and "all's good" kind of server issue.

I tried to VPN into the machines and nothing was resolving, which is generally a Bad Thing. I dusted off my cardkey (unused for at least a year), grabbed my car keys and headed out the door in that daze that only a mom or dad knows during a 4am feeding. The usual fearful thoughts went through my head about the worst case scenarios.

Once I arrived at the Fisher Plaza building I circled the block and drove to the entrance to the parking garage, passing my keycard and receiving a satisfying beep and green light, but the gate didn't raise to let me enter. Through my dozy haze I realized that the parking garage was a dark maw with only the peek of a car in the shadows. It dawned on me that this was no longer just My Problem but Someone Else's Problem. This is one part frustrating and one part relieving. Although a power outage was a shock to the machines we've had similar problems at Fisher Plaza* and the machines were restarted with some coaching. So I settled into the parking lot across the street and walked over to the entrance.

Arriving at the entrance I validated my initial observation that, yes, the power was out, so I struck up a conversation with some folks loitering near the entrance. "Is the power out?" I inquired, more as a conversation starter to get some additional information about what happened.

"Duh," a woman responded with a linguistic flourish. Apparently this was a Kiro employee Who Doesn't Speak with Tech Guys, but I persisted and got some general information. Yes, the power was out. It was due to a fire and one of the generators was fried. And it had been down since around 11pm. And they were here since 3 and they were sooo bored.

I realized that whatever welcome I had spent had emptied the account, so I noticed a person looking just as dazed and techie as me and struck up a conversation with him. Andrew was from PopCap Games and was also called out to check on his servers. Although most of their sites were up, they're Facebook games were hosted exclusively at Internap in the Fisher Plaza Building.

After a while, more tech folks would appear and congregate, sharing any emails received by Internap which were sparse and devoid of any real content, but we'd get wisps of information from other folks who "know people" - generally the guys who actually have to get the work done. What I heard was this:
An electrical cabinet set itself on fire between 11pm and midnight
There was 6 inches of water in the parking garage where the generators are. This happened from putting out the fire.
No one was injured (thanks for asking)
One generator is fried and the other is sitting in the water so it is unusable
The Fire Marshall wasn't letting anyone in (until around 8am when I got in)
Rumors stated that the sprinklers did turn on but as alarming as it sounds, they didn't turn on in the colocation facility - so no wet machines
Kiro stations like Kiro Channel 4 in the building weren't broadcasting - instead they were broadcasting content from sister stations.
Once I got in I inspected the machines to make sure that they weren't buried in rubble or water. With nothing else I to do I headed home to wait for the "experts" to resolve the issue and update geocachers about the situation.

Realizing I couldn't get the word out on the site outages via the normal channels, like email and the web sites, I resorted to posting to Twitter, which in turn posted to Facebook. Fortunately the geocachers continued to retweet and repost the updates throughout the day, so hopefully the information is getting out there. We even updated the text for the iPhone application but that won't satisfy many folks who are experiencing network errors when trying to look up caches.

Fortunately we're in good company. Authorize.Net and Bing Travel and other larger clients are at the facility so Internap is being very serious about getting this resolved. Sometimes it is good to ride on the coattails of the bigger guys when you have a big problem to solve.

I'll update when I have more information. In the meantime I'm monitoring emails and Twitter. When the machines come back online I'll get a flurry of emails so I can head down to make sure that the machines get back online. And I'll continue to update via Twitter and Facebook until this gets resolved. You can follow the latest developments with the hashtag #fisherfire or search for Fisher Plaza or follow me @locuslingua on Twitter.

*Around 2006 Fisher Plaza lost power allegedly due to a safety switch meant to shut down power if someone was electrocuted. It's a nice safety feature unless some knucklehead hits it by mistake.

[KCRacin]

Just read this and thought I'd pass it on

mashable.com/2009/07/03/geocaching-down-too/

[currykev]

Thanks to all (and Jeremy) for the updated info. I didn't want to get another forum ban before finding out the real story.

[thekennelat79]

As of 10,45 am UK time...........

Power has been restored to the building but GC servers are not yet up

[Middleleaze Moles]

I ran a load of cornwall PQ's last week, I'm not sure which one would cover the area you need, but I'm happy to forward on all 6 if you want.

PM me if you want them

Message sent - thank you so much!:cheers:

[mickyz]

Thanks for the update K at79
That's good news. :popcorn: waiting patiently

[DrDick&Vick]

GC Forums page back up now just waiting for GC.com to get going.

[budge35]

Hi can I ask what is a PQ? Only been caching a few months.

[DrDick&Vick]

Its back up and running

[DrDick&Vick]

Hi can I ask what is a PQ? Only been caching a few months.

Its a file that Premium Members can download and contains up to 500 caches centred on a set location

[Simply Paul]

The other forum is live, which can only be a good sign. :)

[Mr.Coffee and the Coffee Clan]

GC.com back online, happy caching to everyone in the UK.
Regards Mr.Coffee and the "11pm time for coffee" Coffee Clan.

[The Wombles]

Well done to the geocaching.com team. I was following Jeremy's blow by blow account and it sounded like quite a saga. Elias even broke his holiday to get the site back up. Thanks.

[Speakers-corner]

Good morning to you all,
just a quick question. Is GC down again? I have a message that says "the service is not available". Is anyone having the same problem?

[Bear and Ragged]

11.43
Seems OK to me...

[Bill D (wwh)]

Good morning to you all,
just a quick question. Is GC down again? I have a message that says "the service is not available". Is anyone having the same problem?
It's working for me. In a situation like this there are bound to be hiccups - their servers are probably overloaded by all those people who haven't been able to access them. Patience is a virtue... :)

[DrDick&Vick]

worked ok this morning, then couldnt access it from the mobile or GCz and now it has let me log 3 caches and frozen me out for the 4th.
Keep getting 'DNS Error - Server cannot be found'
But I can access the forums 'over there'

[jacobite]

A welcome side effect of the recent gc.com server issue is that on 3rd July 2009, 137 users accessed the GAGB forum..............which was nice :D

[GAZ]

A welcome side effect of the recent gc.com server issue is that on 3rd July 2009, 137 users accessed the GAGB forum..............which was nice :D

I am still in Turkey, come back to Blighty on Tuesday, and up until last night, I couldn't access the GAGB forums because the Turkish servers say they are "forbidden" :( .......well, that was until my daughter Leanne, who lives and works out here, told me about a site called "ktunnel.com", which seems to allow them :) .....and youtube :D