This post was mentioned on Twitter by PariseauTT: Another cracking post by @chrismevans on automated tiering. Last two lines especially interesting (too long to tweet). http://is.gd/4uAsW…

And indeed, as BarryW says, it’s not the movement (or avoidance of movement) that’s so hard…it’s figuring out what needs to be moved when and where.

EMC has been working on FAST for even longer than the announced date, analyzing traces from literally thousands of installed arrays and modelling different prefetch algorithms. And an almost equivalent amount of time and effort has gone into the definition of the management interfaces for FAST, leveraging a half dozen or so customer Technical Advisory Panels where different approaches have been modelled and analyzed by future FAST users.

And it really isn’t a race to see who’s first; I’ll safely predict that FAST-like implementations will form the foundation of virtually every storage platform within a few years. There will be differences of implementations, but the idea of distributing data across multiple different types of storage to optimize both performance and cost is inevitable.

IMHO, anyway

Chris

However I’m more than happy if you *want* to ell me more!

It’s all about minimizing latency on predictable cache misses…

I don’t think you’re really thinking this all the way through, though.

The objective of promotion to a faster type of storage isn’t as simple as to try to stick the most recently requested data onto the fastest storage – heck – that would frequentrly be a waste of good Flash, because lots of data is accessed once and never again.

No, the objective of FAST is to get the data that most often (and repeatedly) results in long response times when it requested over a period of time promoted – and to get that data promoted BEFORE it is even requested. Thus, the secret of FAST is to PREDICT what needs to be in Flash, and to determine what can safely be demoted to slower storage without significantly impacting response times – and then to get that data there at the right time. I can’t get into all the details here (patents and all that), but the simple LRU scheme you describe simply isn’t sufficient for multi-tiered FAST. And in fact, the DRAM cache management algorithms themselves have to be adapted to accomodate the different performance characteristics of Flash, FC and SATA.

That said, the actual implementation of sub-LUN FAST is indeed complex, but it is hardly a grounds-up rewrite. No, in fact, Symm Virtual Provisioning was architected from the very beginning to support FAST. And Symmetrix has a strong legacy of pre-fetch algorithmic experience – it just has to be adapted to the performance characteristics of Flash and SATA and the different temporal space of these slower storage media. From there, it is really just a simple matter of programming (and QA, and use-case validation, and algorithm optimizations, and regression testing, and performance tuning, dot-dot-dot).

Remember, the only reason God ceated the earth and the heavens in just 6 days was because He didn’t have to deal with an installed base.

You have a good point here. What if you didn’t actually “move” the data, but simply made a temporary second copy on SSD.

The problem with DRAM cache is time. A block will typically age out of cache (due to capacity) in seconds. A block moved to SSD will typically live there for days or weeks (again due to order of magnitude more capacity).

But what if you could discard data in the SSD tier as quickly as you can cache data. So not ‘move’ it in the first place… but simply clone it… for as long as it was needed