This is a continuing series on Enterprise Data Migration Strategies.  Previous posts:

Enterprise Computing: Data Migration Strategies – Part I

Enterprise Computing: Data Migration Strategies – Part II

Enterprise Computing: Data Migration Strategies Part III

So far in this series of posts I’ve discussed the pre-work required in order to get to a position to execute on actual data moves.  In this post, I’ll discuss the migration scenarios and how to ensure data integrity is maintained throughout this process.  The final post in this series will talk about some of the migration tools which may be utilised.

Maintaining Integrity

Migrating data is a pretty simple task.  You could simply copy the files from one volume to another; you could use replication tools like SRDF, TDMF or even the host native volume manager.  However during this process, it is essential to ensure that data integrity is maintained.  If something goes awry during the data move stage, the last thing you want is to be unsure of the position of the last valid copy of your data.  Here’s what could go wrong:

• The copy process itself fails.
• The server crashes/reboots.
• The target storage device crashes, or becomes unreachable.
• A minor datacentre issue occurs (array failure for instance).
• A major datacentre issue occurs.
• Replication or access to a remote datacentre is lost or disrupted.

In addition, consider that a more complex migration plan will be required for hosts which have replicated storage and/or point-in-time clones or snapshots.  If these form part of the business processes on the server, then chances are they will all need to be up and synchronised before the server can be accepted as migrated successfully.

Get The Business Involved

At this stage it may be a good time to re-emphasise something I’ve mentioned already, and that’s making sure you have a good relationship with your server owner.  Obviously they will have been in previous discussions regarding migration, however you need to know from them exactly how the storage on their server is used.  You need to be aware of any business processes which rely on snapshot or other point in time copies of data. In terms of the migration process, you should be developing a business-centric view of the migration steps, which shows at each stage of a complex migration, exactly what the currency of each instance of data is.  Obviously this isn’t essential for every single server within a storage environment, but will certainly be important for the most complex configurations.

Some Typical Examples

Here are some typical migration scenarios:

• Synchronously Replicated Data.  Host data is mirrored between a primary and secondary array.  Replication is used for disaster recovery.  Here, the migration process must maintain the DR position for the data.  If it’s possible to take an outage then the mirroring can be split re-established to a new replication pair, perhaps by making the new local copy a snaphot clone.  If no outage is possible and the application/server must be kept running then this won’t be an acceptable solution.  In this case another method is required, for example host-based mirroring.
• Asynchronously Replicated Data. Host data is mirrored between a primary and secondary array.  Replication is used for disaster recovery.  In this scenario, the same issues apply as with synchronous replication, however there’s the added complication of maintaining write order/consistency during the copying or only accepting a checkpoint of the completed copy. 
• Replicated Data With Local Snapshot.  Data is replicated remotely with a local snapshot for instant data recovery.  This option adds the extra complication of retaining a local snapshot.  If the snapshot has to be retained from a specific date/time, then the snapshot itself will also require copying.  Obviously the snapshot can’t then be associated with its source volume until a new copy is required as it would potentially overwrite the contents.  Where possible (for example with daily snapshots), it may be possible to tie the migration to creation of the snapshot and complete the snapshot recreation after data has been moved to the new location.
• Three Site Replication.  Data is replicated through an intermediate site.  Here, specific vendor technology is being used to create a multi-hop configuration, perhaps for enabling a mixture of synchronous and asynchronous replication.  The process in which the intermediate copy is used needs to be examined to ensure integrity can be maintained during the migration.  The same previous comments apply, however consideration needs to be given to where any data recovery would take place – so if the intermediate site it not used for data recovery, then the final location needs to provide an acceptable recovery point.

Summary

Where complex replication techniques have been used then data migration needs a little extra thought to ensure consistency can always be maintained.  Hopefully most servers won’t fall into the complex category, but having a list of migration scenarios to meet each configuration type is essential to making sure migrations run smoothly.

You will remember that I recently posted a comment about migration costs, specifically with relation to Incipient. My view was (and still is) that the majority of migration costs come from preparatory and remedial work rather than execution of the migration. Well, Incipient asked for the right of reply and I had a call last week with Robert Infantino, their Marketing and Alliances Sr VP.

The $5000/TB figure they were quoting was an average they had seen in the industry for certain vendors’ professional services time to come in and perform the migration work on behalf of the customer. Incipient’s take was that they could provide their appliance/software expertise to provide the same service but at a significantly reduced cost (I won’t quote specific numbers here, but the number quoted was much lower than the equivalent cost from “a vendor”). So, I guess with clarification, it is more clear that Incipient were comparing the vendor costs versus their product costs and not including any internal customer costs (project management, preparation work etc) in the calculation. This seems a more appropriate comparison in my opinion.

Getting back to the vendor discussion, there’s a real issue here. If vendor X wants to sell you their latest technology, they need to accept and take the hit on helping with migration to their new array. This should be even more so where the vendor doesn’t change as this should be a “no brainer” and built into the technology.

In a world where hardware is becoming a commodity, one differentiator will be the vendor who can minimise the effort/cost and impact of migrating from one technology to another. Until then, products like SVC and those from Incipient will continue to have a market position – oh and humble consultants like yours truly!

I’ve not paid much attention to Incipient (their news page doesn’t provide an RSS feed, so there’s no chance of me seeing their press releases easily), but my attention was recently drawn to a recent release relating to their iADM and iNSP products (catchy names, those).

Now, if you want to know about their products, have a look at their website for yourself. Rather, my interest was sparked by a claim in their press release, quoted below:

The High Cost of Today’s Data Migration

Industry estimates and field data captured by Incipient indicate that SAN storage is growing at 40 – 60 percent annually and 25 percent of data under management is moved annually at an average cost of $5,000 per terabyte. Based on these estimates, a data center with one petabyte of storage under management today spends $1.25 million annually on data migration operations. Two years later, the data center is likely to grow to nearly two petabytes increasing the annual data migration cost to nearly $2.5 million.

Source: Incipient Press Release 11 June 2008

So the estimate is $5000 per TB of data movement and 25% of data being moved each year. I can understand the latter; it’s simple logic that if you have a 3-4 year lifecycle on technology then on average 25% of your estate will be being refreshed each year (although that figure is slightly distorted by the fact that you’re also deploying an additional 40-60% each year). Now, how to get to a $5000 per TB calculation…

Excluding new storage acquisition, network bandwidth, etc, I’d assume that the majority of migration costs will be people time. That would include planning and execution of migrations. In environments of 1PB or more, I could (almost) bet my house on the fact that there will be a significant amount of the storage infrastructure which is (a) not understood (b) badly deployed (c) backlevel amongst many other issues. $5000/TB would therefore seem quite reasonable, based on the amount of work needed to refresh. The only problem, though, is that a majority of the manpower cannot be solved by software alone. This will include documenting the environment, bringing server O/S, firmware and drivers up to date, negotiating with customers for data migrations, migration schedule planning, clearing up wastage, new server hardware and so on.

It would be an interesting exercise to determine what percentage of the $5000/TB cost is actually attributable to data movement work (i.e. having someone sitting at a screen issuing data replication commands). I suspect it is quite low. From experience, I’ve been able to move large volumes of data in quite short timespans. In fact assuming sensible preparation and planning, most of the time doing migrations is sitting around (previous employers disregard this statement).

So how much money would Incipient save? My bet is not much.