17-Dec Missing Images Update

[winstonmitchell]

Just a few hours ago most of my stuff was pretty much wrecked. Now it's all back to normal. Somebody seems to know what they are doing...

Thanks Slug, Emily, the NetApp guys and anyone else who had a hand in straightening things out.

[reflectionsbyruth]

thanks for your hard work in getting the problem resolved.

[lilith]

Sometimes in cases like this the responsibility must feel 'heavy'. Thanks everyone for updating us and for the good work you did.
Lilith

[adalberto_tiburzi]

As we say in the comments... Bravo!!!

[offtheradar]

Next time, please give us updates every six hours or so, so we know something is being done.

Let's hope there is no next time.

But if there is, I'd hope people extend the benefit of the doubt before concluding nothing is being done simply because we're not receiving hourly updates.

Oh for heaven's sake. ONCE A DAY updates when something like
this happens will prevent squabbling as you were doing with unhappy
campers. It's done by most companies who understand customers
feel better that way and that cuts off the problems you saw piling up.

Next time maybe your arm will be okay and you don't have to sit here
playing St. George and the Dragons.

rofl... me? squabbling? It's just the internet.. I hope nobody takes any of this too seriously. Judging from some of the hate-posts some people were leaving for admin, you'd think pbase announced it was going to put George Bush in charge or start using puppies as an alternative fuel. And before anyone says anything, I'm a Republican and love puppies. It's a friendly jab at how sensitive I know some of u democrats/EU folks are about Bush.

As for pbase committing to a set number of updates when there's a crisis? C'mon... we've both been here long enough to know we shouldn't hold our breath for that. What's wrong with simply having some faith that the same people who created the site almost a decade ago (slug and emily) are going to continue to fix things "as soon as possible"? We're not dealing with some anonymous corporate giant like yahoo (flicker). It's the same crew. Problems arise... problems get fixed.

Why can't people just show a little patience?? Is it really that difficult? I understand people have a right to be upset... but there's really no reason to be nasty or condescending when we know things are currently being fixed.

Memory cards, cameras, flash units, internet servers, airline baggage routing.... sometimes they fail. Let's not forget we're still capable of exhibiting basic human traits such as patience and understanding when it comes to dealing with the mechanical portions of our hobbies or careers.

I'm expecting my arm to be fully fixed by Christmas and I'll be able to return to sparring and taking pics... thanks for asking! Also just in time to return to work and start paying off some of these holiday bills. Maybe we're all better off behind the camera than the keyboard.

Just my .02

And who the heck is St George and the Dragons?

[andrys]

Why can't you just let people thank PBase instead of taking over every
single topic and making it all about you and your disdain for other customers
who had concerns and voiced them.

LEAVE this as a thread of THANK YOU to PBase for its actions and leave
it without yet another JIBE at other customers. I am so sick of it.

[flemmingbo]

Update 18-Dec-2007 01:50 UTC
Finally. After what seemed like an eternity, the filesystem repair process completed successfully. It was tempting to take a chance and try to put things back online immediately, but fortunately the engineers at NetApp insisted that we carefully check and repair everything first to avoid data loss.
Everything should be back online and normal with two kinds of exceptions.

1. Anyone that was uploading anything at the occurence of the problem on Saturday afternoon might have some broken images. If after uploading, you could see your photos, then they should be fine. If they seemed immediately broken, then I can't fix them since they were never even written to the disk here. (If there are any instances of this problem, there will be very few I think)
2. Anyone who used the regenerate thumbnail feature during the last day will have no thumbnails. I should have just disabled the tool during that time. The effect is that you won't have thumbnails right now and you'll see the original where you would expect a thumbnail. I'm running a script now to go through and find all these cases and fix them.

If tomorrow you're still seeing any broken images or gigantic thumbnail that the automated process I'm running now should have fixed, send me an email at slug@pbase.com and I'll take care of them.

I have new equipment here for which I'm finishing up their configuration. These will allow us to immediately serve out all the photos in the event that the main storage needs repair like this again. Some images during the last two days were in fact coming from these machines, but I didn't have them completely ready yet. Soon.

Really sorry for all the trouble. I know it's painful to have visitors to your site find broken images.

-Chuck Neel
slug@pbase.com

Thank you so much Slug - everything is working 100% again!!! The effort is very much appreciated, thanks mate!

regards,

Flemming

[offtheradar]

Why can't you just let people thank PBase instead of taking over every
single topic and making it all about you and your disdain for other customers
who had concerns and voiced them.

LEAVE this as a thread of THANK YOU to PBase for its actions and leave
it without yet another JIBE at other customers. I am so sick of it.

Um, it ceased being a "thank you" thread when someone started complaining about the lack of communication. I hadn't said a word until then. I think we're all entitled to our input... and wasn't being nasty about it. I just don't think you like anyone to disagree with you.

Sorry. It's not a "jibe at other customers", I'm a customer too. It's a thread in a public forum where we're all able to comment. If someone uses this as a platform to announce their "plan for better communication", I think anyone and everyone is well within their rights to respond to the current path of the thread.

Stop crying already. It's fixed.

[andrys]

flemmingbo, thanks a lot for bringing Slug's latest note to us today.

Very interesting about the lost thumbnails for thumbnails that were
regenerated after the problems but really great that PBase can somehow
regenerate them in the next day or so.

Slug/Chuck - and all others who were working on this,

Thanks very much.

Also, thanks for the fine words in this latest update too. It's good to know you understood re many customers and visitors they were expecting.

Am still impressed how fast you fixed it and that you're preparing for
the future already.

Get a good rest, I hope.

[charales]

Update 18-Dec-2007 01:50 UTC
Finally. After what seemed like an eternity, the filesystem repair process completed successfully. It was tempting to take a chance and try to put things back online immediately, but fortunately the engineers at NetApp insisted that we carefully check and repair everything first to avoid data loss.
Everything should be back online and normal with two kinds of exceptions.

1. Anyone that was uploading anything at the occurence of the problem on Saturday afternoon might have some broken images. If after uploading, you could see your photos, then they should be fine. If they seemed immediately broken, then I can't fix them since they were never even written to the disk here. (If there are any instances of this problem, there will be very few I think)
2. Anyone who used the regenerate thumbnail feature during the last day will have no thumbnails. I should have just disabled the tool during that time. The effect is that you won't have thumbnails right now and you'll see the original where you would expect a thumbnail. I'm running a script now to go through and find all these cases and fix them.

If tomorrow you're still seeing any broken images or gigantic thumbnail that the automated process I'm running now should have fixed, send me an email at slug@pbase.com and I'll take care of them.

I have new equipment here for which I'm finishing up their configuration. These will allow us to immediately serve out all the photos in the event that the main storage needs repair like this again. Some images during the last two days were in fact coming from these machines, but I didn't have them completely ready yet. Soon.

Really sorry for all the trouble. I know it's painful to have visitors to your site find broken images.

-Chuck Neel
slug@pbase.com


==========
Original post from earlier before the fix.
==========

Here's an update on the progress of fixing the problem of broken images.
I've been working non-stop with the engineers at NetApp to resolve this problem.
Until now, I haven't had a good idea of how long until resolution since the NetApp guys haven't been able to predict it due to the large size of the system. Now I'm hoping the process will be complete within 8 hours, but I can't be sure since I haven't had to go through this before.

On Saturday, we lost 3 disks simultaneously in our main storage system which runs on NetApp hardware. This caused an 8 Terabyte volume to have some inconsistencies which have to be analyed and repaired before we can put the volume back online. Fortunately, the cause of the problem is something NetApp understands and they've provided updated firmware for the disk shelves to correct the bug responsible.
Right now we're just waiting for the filesystem analysis program to complete. At that time, we'll be able to bring the volume back online and all of your photos will display properly.

While this has been a painful process, and I apologize for the disruption, the NetApp system is designed to recover from such failures. Also, their support team, while not cheap, is amazingly responsive and doesn't hesitate to send parts immediately, or spend hours of time on the phone walking us through all the steps to recover.

Fortunately, with the exception of a couple hours on Saturday, new uploads, direct linking, and the majority of the site have been working as usual.

I wish the recovery process could have gone faster, but after a problem with the filesystem, it's important to analyze it carefully so we can be sure everything is healthy.

Chuck Neel
slug@pbase.com

Thank you for your efforts and hard work. We appreciate it.

[uese]

as a pbase customer for almost 4 years I have to congratulate slug and the other pbase admins for their excellent communication during this small "crisis". very good communications so that I never got only even a bit nervous - I just knew everything will be allright again many thanks for your excellent service over the last years and hoping to stay online with you for the next decades or so

[humbert]

Thanks to pbase-team and NetApp for recovering all!! Great job!!

[duto]

Many thanks for your huge work & Bravo

[troonly]

Many thanks for a superb recovery!

[mclew]

Thanks from me too!

If I'd been gone this weekend and just returned to look at the Pbase galleries I wouldn't know anything had happened at all.

Good work! Much appreciated.
Margaret