They say that you learn the most when you make mistakes and things go wrong.  Well, last night I certainly must have learned a lot.  What started as a simple physical re-organisation of my hardware turned into a rebuild of my production VMware ESXi server – finishing at 1am.  Here’s what happened.

Failing Disk

I started by shutting down and moving my production ESXi Server out and back into the standard rack it occupies.  On power up, the server failed to reboot, claiming the boot disk was no longer present.  A quick check inside showed that the SAS connector on the boot disk had come loose, so I plugged it back in and tried again (Oh, SAS specification guys – bad design, no retainers on the plugs).  Unfortunately, the boot disk had somehow become corrupted and the server wouldn’t come up.  No problem, I thought, just repair using the installation media.  This is where things started to get complicated.

My ESXi server runs off a Seagate Savvio 2.5″ 15K 73GB drive, one of four Seagate generously loaned me last year for long term testing.  More on that another day.  The server has two disks installed, one of which has VMs on it.  During the repair process I wasn’t sure which disk was the O/S and which was data.  ESXi doesn’t help much, only indicating that both disks contained data in partitions, data that would be lost if I reinstalled. 

Lesson 1 – Make sure you know exactly how your hardware is configured, down to the SAS ports each drive is plugged into.

Actually having multiple drives of the same type is a pain.  So rather than risk data loss, I removed both drives and re-installed the ESXi O/S from a third Savvio drive.  All good.  Now I need to locate and import all my VMs, however some were on the removed Savvio disks.  This meant installing each disk independently and checking the contents to determine which contained VMs and which contained the broken O/S.

Lesson 2 – Wherever possible, place your VMs on disks separate from the server itself.

Yes, I do have most of my VMs on my Iomega ix4-200d, but, rather crucially, not my Windows 2008 AD Server, which needed to be moved from internal disk to the ix4 before I continued (schoolboy error there).  The AD server was rather important for accessing my, ahem, ix4, which is configured to validate logins using AD.  This creates a bit of a circular reference which could have been a disaster.

Lesson 3 – Place your Windows domain controller on a physical server, or have another independent backup elsewhere.

Having a physical server just for AD control isn’t part of my total virtualisation plan, so I’m looking at whether I can host a backup controller with Amazon AWS and use VPN to secure it into my private network.  This way, if I ever have an issue, I can still authenticate.  The issue of course is cost, which may make a dedicated server the cheaper option.

So, by 1am everything was back up and running.  Did I learn anything else?  Well yes…

Lesson 4 – after 22 years in IT, I should remember that adequate documentation and a DR plan are crucial.  In fact, in a virtualised environment, they are essential due to the concentration of risk placing all systems on a single server causes.

So what next for my virtual infrastructure?  I have a few changes planned; I’ll create a backup ESXi server that can import and run the VMs in the event of a future server failure.  I will also be investigating AWS with Windows 2008 and VPN to create a backup domain controller and see if I can continue to work if both server’s hardware failed.

That leaves one Single Point of Failure… my ix4-200d.  Anyone want to donate me a spare one?

This is the second in a series of posts looking at the DroboPro from Data Robotics Inc.  Previous post(s):

Review: DroboPro – Part I

Previously I discussed a few frustrations with connecting my new DroboPro to ESXi, the target environment for my new device.  I’ve now managed to get the ‘Pro connected and visible within ESXi.  See the attached screenshot image taken from my production ESXi server.

So far, I’ve configured eight 2TB LUNs (although the DroboPro itself only has a capacity of 5TB, 2x 2TB and a single 1TB drive) which are numbered 0-7; the LUN column on the screenshot.  Compare this to the LUNs presented from my ix4-200d, which has three LUNs all numbered ‘0′.  The difference is in the way the DroboPro is choosing to present iSCSI devices and the Runtime Name column gives a clue.  In standard Controller/Target/LUN or CxTyLz/CxTyDz methodology, the ix4 disks are using separate targets to map out each iSCSI LUN, with the LUN number always ‘0′.  The Target setting is analagous to a storage port, typically a physical port on large fibre channel arrays.  The LUNs are then numbered on that port.  The DroboPro presents all its iSCSI LUNs on the same target (in this instance number 3) and so the LUNs are numbered from 0.

This method of presentation doesn’t necessarily cause a problem, but is just a different way of presenting the LUNs.  Presumably with the Drobo Elite the multi-user functionality is achieved by enabling multiple targets per Drobo, each mapped to a separate user.  In any case, I now have visibility of my DroboPro within ESXi.  Time, to write some data.

I’m in the process of migrating my existing physical infrastructure into a virtual environment. There’s nothing like “eating your own dog food” (there’s got to be a better expression than that) for testing out your beliefs on how technology should be implemented, so it’s only fitting I virtualise the IT infrastructure I rely on.

In summary, I run a “production” Windows AD domain, Exchange, IIS and file services. This is a typical scenario for many small businesses, with perhaps a few variations thrown in for good measure, such as SharePoint. I’m moving to a virtual environment that is built on ESXi v4.0, a custom design Intel-based server with 2x Quad Core Intel 5420 processors and 16GB of memory. Local hard drives include 2x SAS 7.2K 500GB drives and 2x 73GB SAS 15K drives.

Storage Choices

The first question I posed for my infrastructure is to decide on a storage platform. As this is a SOHO type deployment, I don’t have the luxury of a fibre channel SAN environment. I expect that many SMBs will feel the same and not want to commit to fibre channel deployments when the alternative options out there are low cost and probably more suitable. Therefore here are my choices:

• Internal drives. I have over a terabyte of internal storage but unfortunately its not RAID protected. The disks are deployed on SAS connectors on the motherboard rather than through a dedicated RAID card. Although I could change the configuration and include a card, there are a number of issues; (a) increased cost (b) downtime to replace failed drives (c) physical intervention and downtime on the server to replace drives. My goal is to deploy this infrastructure to run 24×7, so having to take the system down to replace a drive is not an option. I also think that internal drives wouldn’t give me the scalability I might need.
• iSCSI. In this environment, iSCSI is a great solution; there are lots of SMB iSCSI solutions on the marketplace today, offering low cost storage with simple operations. iSCSI is definitely a possibility.
• NAS. Using NAS (more specifically NFS) for storage on VMware provides lots of flexibility. The management of the filesystem itself moves to the NAS hardware and so any VMware host files appear as standard files on the NAS device. When people see this, they usually get the “light bulb moment” as they realise the possibilities this kind of deployment offers. Whole virtual machines can be cloned, backed up or even simply copied as files around the infrastructure. NAS offers easy access to log and configuration files too.

So, for me the choice is NAS. I have iSCSI deployed, but for my production environments, NAS will be used to store my VMware hosts. This is for a variety of reasons, however there’s one additional benefit I haven’t discussed and that’s portability. Placing my vmdk’s on NAS means I can move them around at will. I can replicate them, I can back them up and I can move them easily to another NAS device without using ESXi as the data mover.

Stay tuned for more discussions as I evolve my setup.