VMworld 2012: Software defined Storage Technology aka VMware Distributed Storage

Another fascinating VMworld closed out last week. As always, I met a ton of customers, partners and field folks. To all those that took the time to stop by and say hi, thank you! It was amazing making new friends and touching base with old friends once again. A lot of bloggers have done a thorough job of capturing a summary of VMworld (Duncan Epping captures it here, Josh Coen captures it here, Stu Miniman captures it here). I am going to focus instead on three sessions that piqued my interest, were well delivered and in my opinion, have the potential to be game changing. (By the way, you can e-attend the top 10 VMworld sessions here. None of the sessions I am going to talk about are up there FWIW).

A disclaimer: The descriptions below are a combination of what was presented plus my take. Where it is the latter, I will try my best to call it out. As the official sessions / presentations become available, always use those as your official reference.

The first of these was a Tech Preview titled ‘Software defined Storage Technology’. As the speakers disclosed, the internal VMware name for this is ‘VSAN’. TO BE CLEAR, this is a TECH PREVIEW with no guarantee of if and when this will become available.

In case you have been hiding under a rock, the big focus at this year’s VMworld is around the software-defined data center. The whole idea of looking at all aspects within a data center and offering them within the software stack is a very powerful concept. In my opinion, VSAN seems to be a core part of that strategy.

The basic idea is this – you have a bunch of servers with disks connected to them (or not). You can now create shared storage across these servers using disk attached storage. Pretty neat stuff!

In effect, this will create disk attached virtual storage for your VMware environment. In my opinion, depending on the performance you need, all classic virtual storage and shared storage use cases are fair game. In effect, VSAN is creating another tier of virtual storage.

What are the problems in today’s data center?

There is a basic gap between how customers provision for VMs and how storage is provisioned underneath (By the way, you will see this loss of fidelity in provisioning theme repeated in a subsequent post about VM granular storage or vvols). The storage provisioning is inconsistent and results in waste with storage being provisioned often at a higher tier than is needed by the VM (to guarantee operational SLAs).

A second trend that is becoming compelling is that servers can be loaded with flash storage. In other words, some really fast storage is now available right next to the CPUs.

Lastly with server proliferation, there is a fair bit of storage that is now attached directly to the server (DAS). So, the question becomes ‘What if’ vSphere was able to virtualize the disks in the servers and make them appear as a common datastore to a cluster of hosts.

If you have ever asked yourself that question then VMware Distributed Storage is looking to provide an answer.

What does this help you do?

So what would you be able to do with this that you previously could not and what use-cases is VMware targeting with this technology.

At an abstract level, VSAN is converting a set of servers into a converged platform. The layer that ‘connects’ or more correctly ‘clusters’ the storage is directly built into the vmkernel. From there, stuff gets even cooler.

You can now have hosts with no local storage participating in the Distributed Storage cluster. In other words, you can have thin servers connecting to a pool of storage. And now, you can see why this is referred to as VSANs. It is creating a software defined SAN by making the storage from a cluster of servers appearing to be shared storage (Dare I say federation at the DAS level?).

As you add servers with storage to the cluster, the shared vmdk grows. This allows you to dynamically grow as your needs change over time.

One of the big pushes that VMware is implementing is policy based management. The basic idea is that the VM allocation all the way through should be on the basis of requirements from the infrastructure. From a storage standpoint, examples of policies would be capacity, latency, IOPS, availability, level of disaster recovery and operational recovery. Each of these policies can be rolled into profile templates (the classic gold, silver, bronze). When a VM is now created, you have to specify the profile. Based on the profile itself, storage can now be allocated. The VM and the profile are monitored for compliance with the SLA. If it does not meet the SLA then changes can be made to bring the VM back into compliance. (By the way, in case you are wondering, this is similar to the approach being thought about with vm granular storage).

From this point on, VMware distributed storage behaves identically to regular storage. And that sounds like meh until you realize that this means like things needing shared storage (VMware HA, normal vMotion – without the enhanced bit in vSphere 5.1, DRS, storage I/O control, sDRS) should ‘just work’ (translation: there is tons of engineering work behind the scenes to make this happen). Examples of this: if there is a component server failure, then the VMs on the particular host (and the associated storage) will be subject to restart based on the original policies specified.

Use cases being targeted

Here were the four use-cases VMware said they were targeting.

  1. VDI deployments
  2. Reducing TCO and provisioning time in test and dev environments
  3. To enable high bandwidth / scaleout big data environments
  4. As a target of DR activities to reduce HW requirements at target site using vSphere replication

My take

Simply put, this has huge potential. To a large extent, it removes the barrier between DAS and SAN and creates a uniform way to address storage no matter where it is located. By creating this DAS tier, VMware is giving more customers the ability to use the capabilities provided by shared storage. I also see this as being a stepping stone to allow customers to move seamlessly between DAS storage and SAN storage. For example, a customer could start out using DR at the DAS tier and over time (for various reasons) move to a SAN tier DR or even vice versa. If you now combine this with the promise of VM granular storage, VMware is creating a very fluid end-to-end storage environment independent of where the storage is.

And that brings me back full circle to where we started. Virtualization is changing the bounds of the traditional stack. With VSAN, VMware is blurring the boundary between DAS and SAN. One of the big changes that is happening right in front of our eyes is that the tranditional application, compute, network and storage stack is collapsing, becoming more seamless independent of where the resources are located. Interestingly VCE had a badge that they handed out at this VMworld.

Frankly, nothing says it better. It truly does eventually starts to look like one collapsed stack be it one server, a cluster of servers, converged infrastructure (Vblocks, FlexPods and their ilk) and even datacenters (think VPLEX). And to me, the possibilities with that paradigm are exciting and numerous!

So what do you think? Game changing or not?

[If you are attending VMworld Barcelona, I would definitely recommend attending this session. It is being offered on Oct 10. Here is the link].

UPDATE: 09/29/2012: Here are some other blog posts related to this subject that might be illuminating: