Comments on: Enterprise Computing: Barclays Bank Services Down Due to Storage Array Problems

By: Storage Sense

Storage Sense — Thu, 18 Jun 2009 14:58:46 +0000

It is also a reminder that no resilient you design an array to be, the actual physical frame, the firmware in the array, and the datacenter the array is housed in are themselves are single points of failure.

What is needed is a well thought out Business Continuity strategy that includes sound processes, personnel managment, and useful technology such as array mirroring.

http://storage-sense.blogspot.com

By: Storage Sense

Storage Sense — Thu, 18 Jun 2009 14:53:30 +0000

It it also a reminder that no matter how resilient a single array is desgined to be, the frame and the datacenter it is housed in are in themselves single points of failure. The only way to mitigate that is to have a well thought out business continuity strategy that inlcudes sound processes and technology such as mirroring of arrays.

http://storage-sense.blogspot.com

By: Chris Evans

Chris Evans — Thu, 18 Jun 2009 07:30:30 +0000

Sounds like everyone commenting is of a similar mind – don’t blame the technology per se, look at the process. Thanks for all the comments.

Chris

By: Gar

Gar — Wed, 17 Jun 2009 21:28:05 +0000

Yep, from reading the article on Barclays, I agree with you Chris. I’ve witnessed outages caused by H/W but exacerbated by failings in the support system, a lack of true understanding of operating in a DR mode, and poor design. It would be intersting to know if they had exectued a full Disaster Recovery test recently.
Regards,
Gar.

By: Locutus

Locutus — Wed, 17 Jun 2009 15:34:47 +0000

As is often the case, DR (not to be confused with HA) is looked upon as an IT expense with low ROI. Therefore, the DR infrastructure is built not so much as a useable environmnet but rather as a tickeyboo to satify an audit requirement.
You can preach the merits of the golden copy.
You can preach the importance of like for like storage footprints at the source and target to preserve the performance of the applications post DR.
You can preach the necessity of maintaining consistency to guaranty a recoverable copy in the event of a rolling disaster.
You can preach ad nauseum.

In the end, it comes down to money Vs risk and often times customers err on the side of money because no one expects to fall into the .0001% availability hole in enterprise storage. They hope they’ll retire before the perfect storm hitts.

I can’t say that is the case at Barklays. But it has always been a sore point with me when I discuss HA/DR with my customers.

By: Rob

Rob — Wed, 17 Jun 2009 15:01:09 +0000

“We shouldn’t be focusing here on the technology that failed. We should be focusing on the process, design and support of the environment that wasn’t able to manage the hardware failure and “re-route” around the problem.”

Right. Along those lines, did the architects miss a single
point of failure for a very important design?

A design doc surely shows it to be a single point of failure.
Or: “We didn’t know it to be a single point of failure!”
“Who know what when? Who signed off on it?”

Either way, someone is called on the carpet somewhere. Not good.

By: Enrico Signoretti

Enrico Signoretti — Wed, 17 Jun 2009 13:22:44 +0000

Chris I agree with you. The most important thing is not the hardware, it can fail or not, but design the right process and test it every time it’s necessary!
ciao,
Enrico