I love the late evening banter on Twitter, where a conversation between a number of individuals turns into a personal rant from yours truly.  Tonight’s subject – performance management of Microsoft Exchange and overconfiguration of storage for email.

Some 4 years ago, I was working for a large investment bank (which may now be defunct) and I did the storage configuration and testing for the new Exchange deployment.  Having been called in at the last minute, I had to take the storage configuration provided by the previous experts and the vendor.  This consisted of a DMX1000-P2 (performance model) and using only the fastest 50% of the drives. 

As the pre-deployment testing progressed, all MSFT Exchange servers were installed, configured and loaded with the Jetstress software to test performance.  Unsurprisingly, as the setup had been so hideously over-configured, the  testing concluded with flying colours.  As I checked out the configuration of the individual servers, I found wide variations in their setup; HBAs at 1Gb/s rather than 2 (with HBAs on the same servers running at different speeds); drivers and firmware that were inconsistent; differences in the host logical volume layout.  Despite all this, the configuration worked flawlessly, even with all of the intended production servers running stress loading at the same time.

This isn’t the only over-configured Exchange implementation I’ve seen; another springs to mind that used 300GB drives as 146GB models.  I’ve also seen the same attention given to Notes.  In that instance, however, common sense prevailed and it became clear very quickly that each Notes server could be more heavily loaded with data and that there was no need to short-stroke the drives to achieve the desired throughput. Performance/capacity logic was applied and the configuration streamlined.

The moral of this story?  (a) don’t over-configure purely based on what the vendor recommends.  Chances are they’re doing CYA to ensure they can’t be blamed for poor response times and throughput (b) review your configuration regularly and if response times are overly good, tune things down; use that extra disk space; load the servers more heavily.  

Don’t just assume because everything works normally that you can’t squeeze that extra level of performance from the configuration.

A question I get asked occasionally is; “How many IOPS can my RAID group sustain?” in relation to Enterprise class arrays.

Obviously the first question is to determine what the data profile is, however if it isn’t known, then assume the I/O will be 100% random. If all the I/O is random, then each I/O request will require a seek (move the head to the right cylinder on the disk) and the disk to rotate to the start of the area to read (latency) which for 15K drives is 2ms. Taking the latest Seagate Cheetah 15K fibre channel drives, each drive has an identical seek time of 3.4ms for reads. This is a total time of 5.4ms, or 185 IOPS (1000/5.4). The same calculation for a Seagate SATA drive gives a worst case throughput of 104 IOPS, approximately half the capacity of the fibre channel drive.

For a RAID group of RAID-5 3+1 fibre channel drives, data will be spread across all 4 drives, so this RAID group has a potential worst case I/O throughput of 740 IOPS.

Clearly this is a “rule of thumb” as in practical terms, not every I/O will be completely random and incur the seek/latency penalties. Also, enterprise arrays have cache (the drives themselves have cache) and plenty of clever algorithms to mask the issues of the moving technology.

There are also plenty of other points of contention within the host->array stack which makes this whole subject more complicated, however, when comparing different drive speeds, calculating a worst case scenario gives a good indication of how differing drives will perform.

Incidentally, as I just mentioned, the latest Seagate 15K drives (146GB, 300GB and 460GB) all have the same performance characteristics, so tiering based on drive size isn’t that useful. The only exception to this is when a high I/O throughput is required. With smaller drives, data has to be spread across more spindles, increasing the available bandwidth. That’s why I think tiering should be done on drive speed not size…

I was asked again this week whether defragging of hard drives on Windows servers is really necessary. This is quite pertinent as the cost of deploying an enterprise-wide defrag tool can be significant and any opportunity to save money has to be a good one.

I discussed fragmentation last year (here) when looking into a problem which turned out to be the lack of free space consolidation. however I didn’t discuss the potential performance impact.

So, reflecting on whether defragmentation is required or not, I’d say that in most cases the benefits are minimal. Here’s why…

Firstly, hard drives have on-board cache; the larger and more “enterprise” the drive, then the larger the cache. We are also likely to see more and more hybrid drives on the market, which will have large amounts of fast memory fronting access to the old moveable components. Cache will mask the impact of fragmented files during writes as data will be written to cache and confirmed as written by the drive. The data can then be written to disk asynchronously afterwards. Obviously if cache becomes overrun, then the benefit will be negated.

Second, operating systems have built in file system performance techniques, including lazy writing and file prefetch. These features will try and minimise the latency issues of reading and writing to disk.

Third, if the data being accessed is a database, the database itself will also have lazy writer processes to asynchronously write data to disk.

Now all of the above applies directly to “traditional hard drives”. In systems which have RAID controllers with onboard cache, then the issues will be less. Where storage is taken from a SAN with an enterprise or modular array, all reads and writes will occur through cache, giving the best options for performance and masking the underlying hard drive mechanics.

So, can fragmentation actually help? In fact, I have seen one instance when this occurred and it required slightly special circumstances. The file system was made up from a number of concatenated LUNs using a volume manager. As the server had multiple CPUs and the storage was SAN connected, then multiple I/Os could be issued to the volume. With more fragmentation, then the I/Os are spread across all of the LUNs and performance was increased.

According to Wikipedia, lightning can travel at a speed of 100,000 MPH, however I think storage vendors are even faster than lightning when it comes to highlighting or dissing the competition.

Mere microseconds after reading Claus Mikkelsen’s blog on the USP-V SPC figures, there are posts from BarryW and BarryB, doing the highlighting and dissing respectively (I almost wrote respectfully there; that would have been a funny typo). BarryB must have no real work to do other than to write his blog, looking at the size of the posts he does!

Anyway. I’m not going to comment on the results because the others have done that enough already and I don’t think the details are that relevant. I think what’s more relevant is the stance EMC are taking in not providing figures for customers on the performance of their equipment. I can’t decide whether its a case of arrogance and therefore a feeling they don’t need to provide details because as BarryB says, the customer will buy anyway, or is it because the DMX will not match up to the performance of its competitors. I think it is a mixture of both.

EMC aren’t an array vendor any more and haven’t been for a long time. OK, it is the product they’re most remembered for historically, but their reach is now so wide and deep I think Symmetrix isn’t the focus of a lot of their attentions. If it was, DMX4 would not just scale by the GB, it would have more connectivity, more cache and EMC would have been the *leader* in the implementation of technology like thin provisioning, not the also ran.

On reflection, I think EMC should provide SPC figures. If DMX is better than the others and is “Simply the Best” prove it; bragging starts to sound hollow after a while.

HDS announced today a few amendments to the AMS/WMS range. The most interesting is the apparent ability to power down drives which are not in use a-la-Copan.

According to the press release above, the drives can be powered down by the user as necessary, which presents some interesting questions. Firstly, I guess this is going to be handled through a command device (which presumably is not powered down!) which will allow a specific RAID group to be chosen. Imagine choosing to power down a RAID group someone else is using! Presumably all RAID types can be supported with the power down mode.

One of the cardinal rules about hardware I learned years ago was never to power it off unless absolutely necessary; the power down/up sequence produces power fluctuations which can kill equipment. I’m always nervous about powering down hard drives. I’ve seen the Copan blurb on all the additional features they have in their product which ensures the minimum risk of drive loss. I’d like to see what HDS are adding to AMS/WMS to ensure power down doesn’t cause data loss.

Finally, what happens on the host when an I/O request is issued for a powered down drive? Is the I/O simply failed? It would be good to see this explained as I would like to see how consistency is handled, especially in a RAID configuration.

However, any step forward which makes equipment run cooler is always good.

The announcement also indicated that 750GB SATA drives will be supported. More capacity, less cooling….

I did some more work on my NTFS issue on Friday. As previously mentioned, I was seeing NTFS filesystems with large levels of fragmentation even after drives were compressed.

The answer turns out to be quite simple; Windows doesn’t consolidate the free space blocks which accumulate as files are created and deleted. So, as a test I started with a blank 10GB volume and created a large file on it. Sure enough the allocation occurs in a small (2 or 3) number of extents. I then deleted the large file and created 10,000 small (5K) files and deleted those too. I then re-created the large file, which immediately was allocated in 100’s of small fragments and needed defragmentation immediately. The large file was created using the freespace blocks freed up from the small files.

What’s not clear from the standard fragmentation tool provided with Windows is that the free space created by the deletion of files is added to a chain of free space blocks. These free space blocks are never consolidated even if they are contiguous (i.e. as in this instance where I deleted all the files on the disk). This means even if you *delete* everything on a volume, then the free space is still fragmented and files will be created with instant fragmentation. The other thing to note is that the standard Windows defragmenter doesn’t attempt to consolidate those segments when a drive is defragmented, it simply ensures that files are re-allocated contiguously. It also doesn’t report that fact either.

I’m currently downloading Diskeeper, which allegedly does consolidate free space. I’m going to trial this and see how it affects my fragmentation problem.

Incidentally, I used one of Sysinternals’ free tools to look at the map of my test drive. Sysinternals were bought by Microsoft in the summer of 2006, however you can find their free tools here. I used Diskview to give me a map of the drive and understand what was happening as I created and deleted files. What I would like, however is a tool which displays the status of free space fragments. I haven’t found one of those yet.

So, now I have an answer, I just have to determine whether I think fragmentation causes any kind of performance issue on SAN-presented disks!

I’ve just read a couple of Gary O’s postings over at Thoughtput, the blog from Gear6.

In his article “Feeding the Virtual Machines”, he discussed NAS and SAN deployment for a virtual environment and makes the bold claim:

“Most people tend to agree that NAS is easier and more cost effective than SANs for modern data center architectures.”

I have to say that I for one don’t. Anyone who’s had to deploy hardware such as Netapp filers will know there’s a minefield of issues around security, DNS and general configuration, which unless you know the products intimately are likely to catch you out. I’m not saying SAN deployments are easier, simply that both SAN and NAS deployments have their pro’s and con’s.

The second post, Shedding Tiers questions the need to tier storage in the first place and Gary makes the comment:

“If money were no object, people would keep buying fast drives”

Well, of course they would. I’d also be driving a Ferrari to work and living in Cannes with a bevvy of supermodels on each arm but unfortunately like most people (and businesses) I have champagne tastes and beer money…

Tiering is only done to save money as Gary rightly points out, but putting one great honking cache in front of all the storage seems a bit pointless. After all, that cache isn’t free either and what happens if those hosts who are using lower tier storage don’t need the performance in the first place?

I almost feel obliged to use BarryB’s blogketing keyword…. :0)

After the recent EMC announcements on DMX-4, I promised I would look at the question of whether the new DMX-4 is really as green as it claims to be. I did some research and the results are quite interesting.

Firstly we need to set the boundaries. One of the hardest part of comparing hardware from different manufacturers is that they are intrinsically different (if they were too similar, the lawyers would be involved) and that makes it difficult to come up with a fair comparison. So, I’ve divided the comparisons into controller and disk array cabinets. Even this is difficult. The DMX has a central controller cabinet which contains only batteries, power supplies, interface boards and so on. The USP however uses half of the central controller for disks. The DMX has 240 drives per cabinet, however the USP has 256 disks per cabinet. This all needs to be taken into consideration when performing calculations.

Second, I want to explain my sources. I’ve tried to avoid the marketing figures for two reasons; firstly they usually refer to a fully configured system and secondly they don’t provide enough detail in order to break down power usage by cabinet and by component. This level of detail is necessary for a more exact comparison. So, for the USP and USP-V, I’m using HDS’s own power calculation spreadsheet. This is quite detailed, and allows each component in a configuration to be specified in the power calculation. For EMC, I’m using the DMX-3 Physical Planning Guide. I can’t find a DMX-4 Planning Guide yet, however the figures on the EMC website for DMX-4 are almost identical to those for DMX-3 and it’s as close as I can get.

DMX-3/4

The DMX figures are quite simple; the controller cabinet (fully loaded) takes 6.4KVA and a disk cabinet 6.1KVA. A fully configured controller cabinet has 24 controller slots, up to 8 global memory directors and 16 back and front-end director (FED) cards. A typical configuration would have eight 8-port FED cards and 8 BED cards connecting to all 4 disk quadrants. EMC quote the disk cabinet figures based on 10K drives. Looking at Seagate’s website and standard 10K 300GB FC drives, each requires 18W of power in “normal” operation, so 240 drives requires 4.32KVA. The difference between this figure and the EMC value will cover when drives are being driven harder and the power supplies and other components which need powering within a disk cabinet. We can therefore work on an assumption of 25.4W per drive on average.

Now the figures for the controller cabinet are interesting. Remember EMC have no drives in the controller cabinet so all the power is for controllers, charging batteries and cache. So all that 6.4KVA is dedicated to keeping the box running.

USP

The HDS power calculator spreadsheet is quite detailed. It allows specific details of cache, BEDS, FEDs and a mix of 73/144/300GB array groups. A full USP1100 configuration has 1152 drives, 6 FEDs, 4 BEDs and 256GB of cache. This full configuration draws 38.93KVA (slightly more than the quoted figure on the HDS website. Dropping off 64 array groups (an array cabinet) reduces the power requirement to 31.50 KVA or 7.43KVA for the whole cabinet. This means the controller cabinet draws 9.21KVA and in fact the spreadsheet shows that a full configuration minus disks draws 5.4KVA. The controller cabinet has up to 128 drives in it, which should translate to about 3.7KVA; this is consistent with the 9.21KVA drawn by a full controller cabinet. The 7.43KVA in a cabinet translates to 29W per drive, making the HDS “per drive” cost more expensive.

This is a lot of data, probably not well presented but it shows a number of things;

1. There’s an inescapable power draw per drive which can’t be avoided; this equates to about 20W per drive.
2. The controller frame needs about 6KVA and this varies only slightly depending on the number of controllers and cache.
3. The HDS controller is slightly more efficient than the EMC.
4. The HDS disk array is slightly less efficient than the EMC.

Neither vendor can really claim their product to be “green”. EMC are playing the green card by using their higher density drives. There’s no doubting that this does compute to a better capacity to power ratio, however these green power savings come at a cost; SATA drives are not fibre channel and not designed for 24/7 workloads. Whilst these drives provide increased capacity, they don’t provide the same level of performance and DMX systems are priced at a premium so you want to get the best bang for your buck. However, if EMC were to price a SATA-based DMX competitively, then the model is compelling, but surely that would take business away from Clariion. What’s more likely to happen is customers choosing to put some SATA drives into an array and therefore see only modest incremental power savings.

So what’s the future? Well, 2.5″ drives currently offer up to 146GB capacity at 10K and only half the power demands, which also translates into cooling savings. Is anyone using these in building arrays? Hybrid drives with more cache should allow drives to be spun down periodically, also saving power. Either way, these sorts of features shoudn’t come at the cost of the levels of performance and availability we see today.

One final note of interest…HDS are quoting figures for the USP-V. These show a 10% saving over the standard USP, despite the performance improvements…

Here’s the last of the performance measurements for now.

Logical Disk Performance – monitoring of LDEVs. There are three main groups Tuning Manager can monitor; IOPS, throughput (transfer) and response time. The first two are specific to particular environments and the levels for those should be set to local array performance based on historical measurement over a couple of weeks. Normal “acceptable” throughput could be anything from 1-20MB/s or 100-1000 IOPS. It will be necessary to record average responses over time and use these to set preliminary alert figures. What will be more important is response time. I would expect reads and writes to 15K drives in a USP to perform at 5-10ms maximum (on average) and for 10K drives to perform up to 15ms maximum. Obviously synchronous write response will have a dependency on the latency of writing to the remote array and that overhead should be added to the above figures. Write responses will also be skewed by block size and number of IOPS

Reporting every bad LDEV I/O response could generate a serious number of alerts, especially if tens of thousands of IOPS are going through a busy array. It is sensible to set reporting alerts high and reduce them over time until alerts are generated. These can then be investigated (resolved as required) and the thresholds reduced further. LDEV monitoring can also benefit from using Damping. This option on an Alert Definition allows an alert to be generated only if a specific number of occurrences of an alert are received within a number of monitoring intervals. So, for instance, an LDEV alert could be created when 2 alert occurrences are received within 5 intervals. Personally I like the idea of Damping as I’ve seen plenty of host IOSTAT collections where a single bad I/O (or handful of bad I/Os) are highlighted as a problem when 1000s of good fast IOPS are going through the same host.

This is the last performance post for now. I’m going to do some work looking at the agent commands for Tuning Manager, which as has been pointed out here previously, can provide more granular data and alerting (actually I don’t think I should have to run commands on an agent host, I think it should all be part of the server product itself, but that’s another story).

Next under discussion for performance is array groups.

First the background (and as usual, apologies to those who already know all this). HDS enterprise arrays lay their disks out in array groups, either RAID-1/0, RAID-5 and RAID-6. Variable size LUNs are then carved out of the array groups for presentation to hosts. Obviously it makes sense to ensure that every host has their LUNs selected from as many array groups as possible. This means that large volumes of read and write data can be serviced quickly and on writes, the data written into cache can be destaged to disk quickly.

Pick any hour in the production day and use something like Tuning Manager and you’ll likely see (especially on unbalanced systems) the standard exponential curve for IOPS or MB/s written. The 80/20 rule applies; around 20% of the array groups will be doing 80% of the I/O. Ideally it would be best to have workload balanced evenly across all array groups, however the effort of rebalancing the data probably doesn’t justify the returns, unless of course you have some very busy array groups. Personally I’d look to ensure no single array group exceeds 50% active and I’d want 25-50% read hits (remember that you need to make sure you have some spare capacity to cater for disk sparing). Any array groups exceeding these metrics are candidates for data movement. I’ve recently used Cruise Control and I found it a disappointment – EMC’s Optimiser swaps two LUNs using a third temporary LUN to manage the exchange. Cruise Control expects you to provide a free LUN as a target of migration. This may be difficult if a very quiet array group has been fully allocated. Therefore I tend to recommend manual exchanges, especially if hosts have a Logical Volume Manager product installed.

Balancing workload will mean checking array groups and moving any “hot” LUNs away from each other. This is where knowing your data becomes important and if possible knowing how the data is mapped on the host to make sure LUNs aren’t just busy with transient data (for example, multiple LUNs that comprise a single concatentate volume on a host may be busy over time as more data is allocated to the volume). It is also equally possible that a single host may be able to overload one array group, so in that instance, moving the LUN will provide no benefit and the data layout on the host will need to be addressed.

It’s possible to spend hours looking at array group balancing. The key is to make sure the effort is worth the result.