So, here’s my rash statement from Twitter last night: “If FAST isn’t free, I don’t want it!  All it’s doing is automating process I could script/do manually”.  It’s a bold statement, I know, so is FAST really offering something better than what could be achieved today using EMC’s Symmetrix Optimizer?

Hot Spots

EMC’s Symmetrix architecture (18 years old and counting, I believe) uses the concept of disk hypers to present LUNs.  Each physical disk is carved into a number of slices, which are then recombined to create LUNs to present to a host.  A mirrored (RAID-1) LUN uses two hypers, a RAID-5 (3+1) LUN uses 4.  EMC ensure general performance by setting standards on how LUNs are created from hypers and that’s reflected in a “binfile” layout.  However despite this sensible planning, it is possible (especially as hard drives are now much larger and contain many more hypers) that two hypers on a single physical disk could be highly active and so contend against each other – in other words “hot spots” on disk.

Optimizer helps alleviate the issue of hot spots by exchanging the high I/O hypers with low I/O ones, distributing busy LUNs across more physical spindles.  This is classic load balancing where resources are distributed across the available infrastructure in order to obtain better overall generic performance.  EMC have now rebranded Optimizer as part of Ionix for Storage Resource Managment, but it’s still effectively the same product.  Hyper swaps can be managed automatically, based on historical performance data.  They can also be user-defined – a manual swap at the users request.

Although tedious (and not as well automated as Hitachi’s HiCommand Tiered Storage Manager), in theory Optimizer could be used to manually move workload between storage tiers.  In fact, Optimizer is already aware of a tiered storage infrastructure.  Here’s a quote directly from the ControlCenter 6.1 manual:

“Optimizer is also aware of physical drives that operate at different speeds, as well as location of the data on the physical media, which influences the I/O rate. This information is used when determining which logical devices to move.”

So with a little bit of knowledge on the layout of data on a Symmetrix array, it would be possible today to use Optimizer to perform LUN-based FAST.

Load-Balancing Versus Policy

Unfortunately, simple load-balancing of I/O across a storage array doesn’t offer what should be seen as the next generation of storage tiering.  Where Storage Tiering 1.0 was about offering multiple layers of storage within the same physical infrastructure and manually placing or moving LUNs to the appropriate tier, Storage Tiering 2.0 will be about establishing policies that determine more service-based measurements of the performance and availability customers receive. 

A policy-based approach would allow rules to be established on how data at the application layer moves between tiers.  This is a critical distinction from the load-balancing  methodology earlier described.  As an example, where an application was known to require higher performance at a certain time of day or day of the week, data could be moved proactively to a faster tier of storage, returning later once the high I/O workload had completed.  Whilst achievable using Optimizer, there’s no doubt the process of application migration would be tedious and time consuming.  I expect the v1.0 implementation of FAST will simply package up Optimizer into a tool that automates the migration of related data between tiers.  Don’t forget, other vendors have been offering this feature for some time – for example Hitachi and Tiered Storage Manager.

Increasing Granularity

Now LUN-based migration has its benefits.  Where large numbers of disks exist in an infrastructure, application data can be placed or moved to the most appropriate location as required.  However with the introduction of solid state disks (SSDs), a more granular approach is needed as the number of SSDs deployed in an array is likely to be low due to their excessive cost.  Moving an entire application (or even LUN) to SSD will be undesirable unless that application can take full use of the SSD hardware.  There are very few, if any, applications that require high-intensity read/write activity from every piece of application data all the time.

Block-level tiering offers a higher level of granularity to the placement of data.  A LUN can be split into blocks and placed across multiple layers of storage technology including traditional HDDs and faster SSDs.  Selective placement will ensure the more efficient use of expensive SSD media by placing only the highly active data onto it.

All of a sudden with increased granularity we’re back to Storage Tiering 1.0 where data is being placed on faster technology purely based on increasing overall system performance.  This is a feature Compellent have been offering for some time.  Data is migrated up or down the tier hierarchy on a daily basis, subject to performance figures over a 12-day period.  This level of granular performance management is possible because data is stored in a block-based structure.  Unfortunately for EMC, the hyper design legacy represents a technical challenge in making FAST version 2 a reality. 

Patent Rights

As just mentioned, Compellent already offer block-based data migration in their products.  At a recent dinner in London with the Compellent team, they highlighted their strong position in the market, protected by patents covering block-level data migration between tiers.  You can find the filed patent here.  Compellent use the term “Data Progression” to describe how blocks are moved between tiers based on I/O activity.  As I/O activity is monitored over time, it is possible to determine the most appropriate tier of storage to use when expanding capacity.  Typically these are lower tier SATA drives, as initial performance requirements are usually over-estimated.  This metholodogy is very much Storage Tiering 1.0 discussed earlier.

Compellent aren’t the only people claiming rights to block-level tiering within a storage array.  I’ve also found the following patent application from IBM, filed by Barry Whyte, Steve Legg and others.  If IBM and Compellent both claim to have invented the FAST concept, how does that position EMC?  Do they have an earlier patent which trumps these two?

Summary

Storage Tiering 1.0 provides performance management of storage arrays.  Storage Tiering 2.0 extends this to offer policy-driven optimisation offerings.  Both of these technologies are available today from existing vendors in one format or another.  EMC will simply be playing catchup with these vendors once FAST 1 & FAST 2 are released.  I’d like to be surprised and see EMC offer something the competition currently don’t.  I’m not holding my breath…

Earlier this month, Texas Memory Systems announced they had acquired the intellectual assets of Incipient, a company that produced SAN virtualisation hardware and software.  With Incipient gone, EMC hardly bothering to mention Invista, what is the future of SAN LUN virtualisation? 

I talked about Incipient last year, here and here when discussing the costs of performing migrations.  As I said at the time, I couldn’t see how much of a saving deploying their iNSP would bring to the burdensome migration work we all have to manage on an ongoing basis.  So there’s got to be a more compelling benefit out there for using virtualisation products.  If there is, then what is it?

Excluding the defunct Invista, that leaves Hitachi with Universal Volume Manager (UVM) and IBM with SAN Volume Controller (SVC) still in the market place.    From experience, I know UVM is a great product and surprise, I’ve commented on that recently too especially herewhere I reference the fact that Hitachi are offering UVM for free.  Clearly, the drawback to UVM is that it is integrated into the array itself.  When the NSC55 first came out, I heard rumours that it may be a diskless virtualisation “head” and although it can be deployed in that way, it isn’t sold as that.  If Hitachi decided offer the USP VM or its successor as a diskless virtualisation controller, it would put them squarely in competition with SVC from IBM.

Earlier this year I was fortunate to have an invitation to meet Barry Whyte, “Master Inventor” and Performance Architect on the SVC product.  You can find Barry’s blog hereif you’re already not subscribed to it.  I highly recommend it especially for understanding the in’s and out’s of the SVC itself.  During my trip I got to see some of the hardware used to do interoperability testing of SVC – with storage it virtualises as well as servers it connects to.  It’s by no means a trivial task; there are 80 people in Hursley alone, working on development and testing of the product as well as a further 64 scattered around the globe.  Obviously virtualising storage is a complex business and requires huge amounts of testing.  I’d go as far as suggesting that the testing takes way more cycles than writing the code itself.

What’s all this got to do with the future of virtualisation?  Well, I think it highlights what a complex process it is.  Even though standards for interoperability exist, IBM (and presumably Hitachi, EMC and at one time Incipient) have to deal with complex interoperability issues and interleave that with additional features and functionality whilst guaranteeing data integrity.  The slide taken from an SVC presentation deck gives you an idea of what’s involved.  Thanks to Barry for permission to reproduce this.

Both Hitachi and IBM have been successful with a virtualisation product that doesn’t sit within the SAN fabric itself.  This seems to me to be counter-intuitive as I’ve always thought the fabric was the right place for virtualisation.  After all, every I/O leaving a host hits the fabric first and this naturally becomes the best place to route the I/O to its final destination, whether or not that is a “real” LUN or one created from a virtualisation product. 

Perhaps SAN fabric virtualisation was simply too complex and costly to deploy.  After all, recent history has told us that paying for a fabric-based virtualisation product is a non-starter otherwise we’d see more Invista and iNSP.  Perhaps fabric-based virtualisation didn’t provide the feature set that mature IT organisations required from the technology.  Either way, virtualisation in the fabric needs a rethink.  Maybe FCoE provides/provided that opportunity?

“Innovative – featuring new methods or original ideas – creative in thinking” – Oxford English Dictionary of English, 11th edition.

There have been some interesting comments over the weekend, specifically from EMC in regard to this post which I wrote on Benchmarketing started by Barry Burke and followed by Barry Whyte.

“Mark” from EMC points me to this link regarding EMC’s pedigree on innovation. Now that’s like a red rag to a bull to me and I couldn’t help myself going through every entry and summarising them.

There are 114 entries, out of which, I’ve classified 44 as marketing – for example appointing Joe Tucci (twice) and Mike Ruettgers (twice) and being inducted into the IT Hall of Fame hardly count as innovation! Some 18 entries relate directly Symmetrix, another 18 to acquisition (nnot really innovation if you use the definition above) and another 7 to Clariion (also an acquisition).

From the list, I’ve picked out a handful I’d classify as innovating.

• 1987 – EMC introduce solid state disks – yes, but hang on, haven’t they just claimed to have “invented” Enterprise Flash Drives?
• SRDF & Timefinder – yes I’d agree these are innovative. SRDF still beats the competition today.
• First cached disk array – yes innovation.

Here’s the full list taken from the link above. Decide for yourself whether you think these things are innovative or not. Acquisitions in RED, Marketing in GREEN. Oh and if anyone thinks I’m being biased, I’m happy to do the same analysis for IBM, HP, HDS etc. Just point me at their timelines.

• Clariion CX4 – latest drives, thin provisioning?
• Mozy – acquisition
• Flash Drives – 1980’s technology.
• DMX4 – SATA II drives and 4Gb/s
• Berkeley Systems etc – acquisition
• EMC Documentum – acquisition
• EMC study on storage growth – not innovation
• EMC floats VMware – acquisition
• EMC & RSA – acquisition
• EMC R&D in China
• EMC Clariion -Ultrascale
• EMC Smarts – acquisition
• Symmetrix DMX3
• Smarts, Rainfinity, Captiva – acquisitions
• EMC – CDP – acquisition
• EMC Clariion – Ultrapoint
• EMC DMX3 – 1PB
• EMC Invista – where is it now?
• EMC Documentum – acquisition
• Clariion AX100 – innovative? incremental product
• Clariion Disk Library (2004) – was anyone already doing this?
• DMX-2 Improvements – incremental change
• EMC VMware – acquisition
• EMC R&D India – not innovative to open an office
• EMC Centera – acquisition – FilePool
• EMC Legato & Documentum – acquisitions
• Clariion ATA and FC drives
• EMC DMX (again)
• EMC ILM – dead
• EMC Imaging System? Never heard of it
• IT Hall of Fame – hardly innovation
• Clariion CX
• Information Solutions Consulting Group – where are they now?
• EMC Centera – acquisition
• Replication Manager & StorageScope – still don’t work today.
• Dell/EMC Alliance – marketing not innovation
• ECC/OE – still doesn’t work right today.
• Symmetrix Product of the Year – same product again
• Joe Tucci becomes president – marketing
• SAN & NAS into single network – what is this?
• EMC Berkeley study -marketing
• EMC E-lab
• Symmetrix 8000 & Clariion FC4700 – same products again
• EMC/Microsoft alliance – marketing
• EMC stock of the decade – marketing
• Joe Tucci – president and COO – marketing
• EMC & Data General – acquisition
• ControlCenter SRM
• EMC Connectrix – from acquisition
• Software sales rise – how much can be attributed to Symmetrix licences
• Oracle Global Alliance Partner – marketing
• EMC PowerPath
• Symmetrix capacity record
• EMC in 50 highest performing companies – marketing
• EMC multiplatform FC systems
• Timefinder software introduced
• Company named to business week 50 – marketing
• EMC – 3TB in an array!!
• Celerra NAS Gateway
• Oracle selects Symmetrix – marketing
• SAP selects Symmetrix – marketing
• EMC Customer Support Centre Ireland – marketing
• Symmetrix 1 Quadrillion bytes served – McDonalds of the storage world?
• EMC acquires McDATA – acquisition
• EMC tops IBM mainframe storage (Symmetrix)
• Symmetrix 5100 array
• EMC 3000 array
• EMC BusinessWeek top score – marketing
• Egan named Master Entrepreneur – marketing
• EMC 5500 – 1TB array
• EMC joins Fortune 500 – marketing
• SRDF – innovation – yes.
• Customer Council – marketing
• EMC expands Symmetrix
• EMC acquires Epoch Systems – basis for ECC?
• EMC acquires Magna Computer Corporation (AS/400)
• EMC R&D Israel opens – marketing
• Symmetrix 5500 announced
• Harmonix for AS/400?
• EMC ISO9001 certification – marketing
• Mike Ruettgers named president and CEO – marketing
• Symmetrix arrays for Unisys
• Cache tape system for AS/400
• EMC implements product design and simulation system – marketing
• Product lineup for Unisys statement – marketing
• DASD subsystem for AS/400
• EMC MOSAIC:2000 architecture
• EMC introduces Symmetrix
• First storage system warranty protection – marketing
• EMC Falcon with rapid cache
• First solid state disk system for Prime (1989)
• Reuttgers improvement program – marketing
• First DASD alternative to IBM system
• Allegro Orion disk subsystems – both solid state (1988)
• EMC in top 1000 business – marketing
• EMC joins NYSE – marketing
• First cached disk controller – innovation – yes
• Manufacturing expands to Europe – marketing
• EMC increases presence in Europe and APAC – marketing
• Archeion introduced for data archiving to optical (1987)
• More people working on DASD than IBM – marketing
• EMC introduces solid state disks (1987)
• Storage capacity increases – marketing
• EMC doubl
  es in size – marketing
• Product introductions advance computing power – marketing
• HP memory upgrades
• EMC goes public – marketing
• EMC announces 16MB array for VAX
• Memory, storage products boost minicomputer performance
• EMC offers 24 hour support
• Testing improves quality – marketing
• Onsite spares program – marketing
• EMC delivers first product – marketing
• EMC founded – marketing

It’s not very often I side with one vendor or another however after BarryB’s recent post regarding “Benchmarketing” I feel obliged to comment. Have a read of Barry Whyte’s rebuttal too.

We see technology advancements because “concept” devices are used to drive innovation but don’t necessarily translate directly to end-user products. Look at the fashion industry – some of the most outrageous outfits are paraded down the catwalk but the same dress, coat, hat or whatever isn’t sold in the shops. Instead it influences the next fashion season.

Look at the motor industry – concept cars appear well before actual consumer products. We may laugh at some and marvel at others – take the Bugatti Veyron. It is well known that Volkswagen make a loss on each car produced, however what counters this is the publicity, the research, the kudos of being able to claim Veyron technology (disputably the fastest car in the world) is deployed in the standard VW range. Lexus is another good example of a brand created by Toyota to perform the same function. Much the same can be said for Formula 1.

Now, I’m not endorsing IBM per-se here, however I don’t see the harm with IBM marketing a “concept” piece of technology which could lead to innovation in the future. After all, IBM is well known for research of this kind; the disk drive and the tape drive spring to mind.

Even EMC’s own bloggers question whether EMC is known for innovation and other than Symmetrix, I can’t think of one thing I view as an EMC “idea”.

Anyway, ’nuff said. As previously offered – I would love to take the position of moderator in developing real world benchmarking – bring it on!!

Moshe Yanai of EMC, XIV and now IBM fame (I’m sure I don’t need to fill in the details) is now blogging. You can catch up with him here.

As previously requested, I’m going to start posting my RSS blog feeds, starting with IBM.

• Tony Pearson – Inside System Storage Blog – Homepage – RSS – Rating: *****
• Barry Whyte – Storage Virtualisation – Homepage – RSS – Rating: *****
• Moshe Yanai – Think Storage – Homepage – RSS – Rating: N/A

The Rating figure is an indication of how often and how useful I find this blog – it is *not* any comment on the quality of the writing or author, before I start to get comments on any sort of bias!

I’ve been on holiday for the last week (sunning myself and the family in Cyprus). I had no Internet access – not even TV! Although I had no laptop (or Blackberry this time) I did take my iPod Touch, now configured with the mobile version of NewsGator. As I’ve mentioned previously, I have a 100+ RSS feeds (which I’ll publish once I get around to it) on storage and others. My backlog was about 2500 entries, so I decided to challenge myself to get up to date and read as much as possible. Clearly I didn’t read them all (there were plenty that could be skipped) but I read most and it provides for an interesting cross section…

EMC – blogs are run like a military machine; co-ordinating the news relating to new product releases and mercilessly hammering the competition. EMC have more storage bloggers than any other storage company and there are some good ones out there – one of my favourites is Information Playground by Steve Todd, where he discusses the design of Clariion.

Netapp – follows a close second to EMC with lots of bloggers and lots of competitor bashing. I particularly like Alex McDonald’s postings.

IBM – doing a great job running the “resistance“, fighting back against the continual onslaught of Barry (A Burke). Check out Barry (Whyte) and Tony Pearson. I’d like to see more from IBM though, especially their product developers working on DS arrays and XIV.

HDS – A jolly good bloke, but not really a player in the blogosphere. Only Hu contributes regularly, but doesn’t engage in any serious debate.

Sun – quite literally on another planet with their storage strategy!

Dell – bought some toys, but doesn’t know how to play with them. Unfortunately the older boy who could help them play with them has left…

Now there are more companies out there and I don’t think I have any blog links from Brocade, 3Par, Compellent, Emulex, Qlogic, Pillar and others although I may be wrong (it is getting late). Any RSS link offerings gladly welcome – although I might not get around to reading them before my next holiday!

Following on from BarryW’s comment to my XiV post, I’ve been thinking over how the XiVarchitecture works. When a disk fails and the missing mirrors need to be recreated, then the data is likely to exist across all or most of the configured drives. Logically it would make sense that the target for the new lost mirrors would be all drives. So, as a drive fails, all drives could be copying to all drives in an attempt to ensure the recreated lost mirrors are well distributed across the subsystem. If this is true, all drives would become busy for read/writes for the rebuild time, rather than rebuild overhead being isolated to just one RAID group. Whilst that seems like a good thing for rebuild time, it seems like a bad thing for performance. Perhaps this isn’t the case and in fact the failed device is re-created on a spare drive by copying all the mirrors back in from their other location.

Following the same line, in order to recreate a failed drive and rebuild the lost data across the array, then each drive must have spare capacity; in say a 200 drive system, that would require about 1/200th of all drives to be free at any one time, ready to receive rebuilt mirrors. Obviously the alternative option is just having spare drives, but that sounds less interesting!

What about when the failed disk is replaced? There must be another algorithm which ensures the replaced disk is not a target for all new writes, so presumably, static mirrors are pro-actively moved onto the replaced device.

This architecture throws up some interesting questions, especially when trying to understand performance. I am starting to get excited about messing about with one!

There’s been a few references to Invista over the last couple of weeks, notably from Barry discussing the “stealth announcement”.

I commented on Barry’s blog that I felt Invista had been a failure, due to the number of sales. I’m not quite sure why this is so, as I think that virtualisation in the fabric is utimately the right place for the technology. Virtualisation can be implemented at each point in the I/O path – the host, fabric and array (I’ll exclude application virtualisation as most storage managers don’t manage the application stack). We already see this today; hosts use LVMs to virtualise the LUNs they are presented; Invista virtualises in the fabric; SVC from IBM sits in that middle ground between the fabric and the array and HDS and others enable virtualisation at the array level.

But why do I think fabric is best? Well, host-based virtualisation is dependent on the O/S and LVM version. Issues of support will exist as the HBAs and host software will have supported levels to match the connected arrays. It becomes complex to match multiple O/S, vendor, driver, firmware and fabric levels across many hosts and even more complex when multiple arrays are presented to the same host. For this reason and for issues of manageability, host-based virtualisation is not a scalable option. As an example, migration from an existing to a new array would require work to be completed on every server to add, lay out and migrate data.

Array-based replication provides a convenient stop-gap in the marketplace today. Using HDS’s USP as an example, storage can be virtualised through the USP, appearing just as internal storage within the array would. This provides a number of benefits. Firstly driver levels for the external storage are now irrelevant (only requiring USP support, regardless of the connected host), the USP can be used to improve/smooth performance to the external storage, the USP can be used for migration tasks from older hardware and external storage can be used to store lower tiers of data, such as backups or PIT copies.

Array-based replication does have drawbacks; all externalised storage becomes dependent on the virtualising array. This makes replacement potentially complex. To date, HDS have not provided tools to seamlessly migrate away from one USP to another (as far as I am aware). In addition, there’s the problem of “all your eggs in one basket”; any issue with the array (e.g. physical intervention like fire, loss of power, microcode bug etc) could result in loss of access to all of your data. Consider the upgrade scenario of moving to a higher level of code; if all data was virtualised through one array, you would want to be darn sure that both the upgrade process and the new code are going to work seamlessly…

The final option is to use fabric-based virtualisation and at the moment this means Invista and SVC. SVC is an interesting one as it isn’t an array and it isn’t a fabric switch, but it does effectively provide switching capabilities. Although I think SVC is a good product, there are inevitably going to be some drawbacks, most notably those similar issues to array-based virtualisation (Barry/Tony, feel free to correct me if SVC has a non-disruptive replacement path).

Invista uses a “split path” architecture to implement virtualisation. This means SCSI read/write requests are handled directly by the fabric switch, which performs the necessary changes to the fibre channel headers in order to redirect I/O to the underlying target physical device. This is achieved by the switch creating virtual initiators (for the storage to connect to) and virtual targets for the host to be mapped to. Because the virtual devices are implemented within the fabric, if should be possible to make the virtual devices accessible from any other fabric connected switch. This poses the possibility of placing the virtualised storage anywhere within a storage environment and then using the fabric to replicate data (presumably removing the need for SRDF/TrueCopy).

Other SCSI commands which inquire on the status of LUNs are handled by the Invista controller “out of band” by an IP connection from the switch to the controller. Obviously this is a slower access path but not as important in performance terms as the actual read/write activity.

I found a copy of the release notes for Invista 2.0 on Powerlink. Probably about the most significant improvement was that of clustered controllers. Other than that, the 1.0->2.0 upgrade was disappointing.

So why isn’t Invista selling? Well, I’ve never seen any EMC salespeople mention the product never mind pushing it. Perhaps customers just don’t get the benefits or expect the technology to be too complex, causing issues of support and making DR an absolute minefield.

If EMC are serious about the product you’d have expected them to be shoving it at us all the time. Maybe Barry could do for Invista what he’s been doing in his recent posts for DMX-4?