Federation – A new business model for disruptive innovation

[Even if it has been stated elsewhere, for this particular post, it is even more important to state that these are my views. My interest in this topic is to help organize my thoughts on the implications of the Federation structure created at EMC. Equally importantly, I am an EMC employee and an EMC shareholder, so I have my biases. Consider yourself warned!].

What is the Federation?

For the past six months or so, EMC has organized itself as a Federation. For the official announcement and details behind it, please look here. In case you see it on slides, here is the logo that is being used for the federation.

EMC Federation Logo

In a nutshell, the concept of the Federation is the coexistence of independent companies with a shared vision and at the end of the day, to a large extent a common P&L.

Why organize in this way?

Usually, when looking at business organizations, there are two common ways of organizing. Obviously, no one mechanism of organizing is superior to the other. The correct answer for any given organization depends on a bunch of factors – the desired end goal, market dynamics, level of overlap between the businesses and finally, which bets does the organization want to place. In other words, there is no right answer.
The two common ways of organizing are

  1. Monolith
  2. Conglomerate

Monolith

While the word monolith creates all sorts of mental images, this is by far the most common way of organizing. The organization is built around a singular vision. Different businesses within that organization are organized as business units. Typically, such organizations are able to benefit from shared services – HR, IT, Sales etc. The goal of this mode of organization is clarity of purpose, driving more alignment and efficiency in terms of common capabilities that can be optimized. This doesn’t stop these companies from being diverse – just that the degrees of separation between the individual businesses is relatively small and usually, focused on the same or very adjacent markets. The selling motion is usually similar – the end customer tends to be similar. In other words, the channels are more or less consistent. Naturally, as with most things in life, there are shades of gray – the degree of freedom afforded each business unit is a variable, what shared services operate is a variable etc.
So, whats the downside of such an organization? The greatest strength of such organizations is also their weakness. These organizations tend to be pretty set in terms of their mode of business, what channels and markets they can go after. So, in some ways, for such companies, it is difficult to change their core fabric. Not impossible but definitely herculean. Purely from a business perspective, optimization for a particular market implies that you are completely subject to the vagaries of that market. So, any macro or micro economic trends that impact that particular market leave such a company vulnerable.

Conglomerate

To counter some of the issues seen in the monolith, the conglomerate emerged as a way to get diverse selling motions, diverse companies under the same organizational umbrella. The businesses are not required to be related to each other. The conglomerate functions as independent businesses pooling their P&Ls together. Berkshire Hathaway, General Electric and the Tata Group are some of the prominent examples of this structure. This organizational structure gives up on the efficiency of a common set of services in order to be able to diversify what they sell to their customers. Each business may have its own discrete vision, strategy and market. The common binding item across the conglomerate is the set of shared values across the teams. To illustrate this point, transactions across businesses within the conglomerate are identical to transactions with companies that are not within the conglomerate. The businesses are free to optimize to meet their objectives.

Why Federation?

With this perspective, let’s now look at the Federation.

In some ways, EMC has been on this journey since the VMware acquisition in 2003 (yeah – it was that long ago!). Right from the get go, the interactions between EMC and VMware have been independent. I will not say that this was easy at first. My first interaction with VMware was as a developer dealing with a customer escalation where the customer expected that we would behave as one company. There were a few moments of awkward phone silence as we explained to the customer that we were independent entities. As we all grew comfortable operating in this model, folks on both sides understood why this “together but independent” state was important. At EMC, we realized that we had to win VMware’s business on our merits and that VMware had to interface with EMC’s competition in the same way that they interact with EMC. As that message was understood across both companies, EMC mobilized to have the best possible integration with VMware not because of our inherent affinity but because we recognized the business value that VMware brought.

With VMware as a separate entity, this allowed two additional benefits:

  • VMware was able to maintain and develop its go-to-market independent of EMC. This gave it access to different markets and different levers to enable.
  • VMware was free to innovate independent of the impact to EMC. They were not beholden to EMC’s inherent interests and were able to take a different stance than a business internal to EMC might have been able to.

Clayton Christensen has talked about the innovator’s dilemma where disruptive new technology suffers from the hegemony of the current dominant technology. It is far more deeply entrenched than just technology. For a big successful company like EMC, the entire company has been tuned to make the current technology successful. GTMs, incentives, selling motions, profitability structures, purchase cycles, relationships – the list is endless – are all geared to the current technology. Now imagine changing all that after recognizing a market shift while maintaining your current revenues and while people are changing what they do. Despite best intentions, this transformation is a perilous journey that few dare undertake and even fewer complete successfully. To say nothing about going to individuals who care passionately about the products they work on and tell them that the next big thing is going to surpass their pride and joy.

With this backdrop, fast forward to today. The world of IT has been changing dramatically. You can see that there is a shift in how data is generated, how it is consumed and utilized. Data is the new frontier. The implications of this have not been fully realized. However, it is clear that the magnitude of this shift is enormous. So, the same conditions that led to VMware being kept as an independent entity are now in effect for attacking this new market. That independent entity is Pivotal.

So, you have three independent companies (shades of a conglomerate) all working in adjacent markets (shades of a monolith) working with a shared vision (shades of a monolith) with the independence to optimize what they need for their business (shades of a conglomerate). That’s the Federation – not a monolith, not a conglomerate but an amalgamation of both.

It is typical of what I have seen within EMC – we have a big challenge in front of us, we tackle it head on with a very creative solution. I have this mental image of whoever came up with this as waking up one morning with this solution in their heads and feeling like they have solved world hunger – this is brilliant stuff!

Does this mean that everything is perfect? It never is and there are tradeoffs that are needed to make this successful. For one, this strategy does mean that the individual businesses will have to work that much harder to maintain alignment. The businesses have to be judicious about where it is okay to overlap in technology (since they are in adjacent markets) and what to do when that inevitable overlap arises.

If this strategy does work, this may be an interesting way to address the innovator’s dilemma and guide future companies trying to innovate disruptively while continuing to execute on existing businesses. To me, the beauty of this approach is that it takes that challenge out if the realm of organizational discipline (which is frail) and canonizes it within the corporate structure. I, for one, am rooting for its success (and beyond just the ‘I am an EMC employee’ reason). We will see how the story unfolds. As Churchill aptly said, ‘However beautiful the strategy, you should occasionally look at the results.’

Advertisements

Talkin’ about VPLEX and RecoverPoint Part 4

The past three editions of these have been very popular. Our marketing and CSE team has created some new videos in support of the Q2 launches for VPLEX and RecoverPoint. So here are twelve videos for you to dig into.

  1. Why VPLEX for VMware Environments: Don Kirouac does an excellent job explaining how VPLEX integrates with VMware environments.
  2. Why VPLEX for Oracle RAC: Don Kirouac from the Corporate Systems Engineering team talks about the integration between Oracle RAC and VPLEX Metro to deliver continuous availability
  3. VPLEX with XtremIO: Charlie Kraus from the Product Marketing team explains how VPLEX delivers value to XtremIO environments
  4. ViPR with VPLEX and RecoverPoint: Devon Helms from the Product Marketing team explains how provisioning for VPLEX and RecoverPoint can be made simple with the ViPR Controller.
  5. Why VPLEX for SAP: Jim Whalen from the Solutions Marketing Team explains how VPLEX can help deliver SAP Application Availability.
  6. Why VPLEX for Microsoft Hyper-V Environments: Charlie Kraus talks about how VPLEX integrates with Microsoft Hyper-V environments to deliver mobility and availability
  7. VPLEX with Vblock: Charlie Kraus delves into how VPLEX integrates with and provides value to a Vblock environment.
  8. VSPEX Solutions for VPLEX and RecoverPoint: Karl Connolly from the VSPEX Marketing Team

  9. MetroPoint topology: Paul Danahy and I walk through the benefits and value propositions of the MetroPoint topology
  10. VPLEX Virtual Edition: Paul Danahy and I introduce the VPLEX Virtual Edition solution and why we think this is such a game changer
  11. Simplified Provisioning with VPLEX: Paul Danahy and I talk through how VPLEX Integrated Array Services simplifies provisioning with VPLEX
  12. EMC AppSync for RecoverPoint: Parag Pathak from the AppSync Marketing team and Devon Helms talk about the integration between AppSync and RecoverPoint to deliver application consistent protection

2014 Launch Post 3: Management@Scale: VPLEX Integrated Array Services (VIAS)

On April 4th, 2014, as part of the Data Protection and Availability Division (DPAD) launch, there were three VPLEX and RecoverPoint items that were launched or GAd:

  • VPLEX Virtual Edition – Availability late Q2
  • MetroPoint Topology – Joint capability of VPLEX and RecoverPoint – Availability Late Q2
  • VPLEX Integrated Array Services – Available now

This is the third in a series of posts to walk through what was launched / delivered.

The IT World we live in

Customers are experiencing a massive growth in data center scale. This has been the result of a lot of consolidation across data centers, the growth of big data and fast data applications and specialized infrastructure to manage the different application workloads.

In addition, as VPLEX environments have mushroomed, they are subject to the same stresses and strains of the overall IT environment. So, it is a good news / bad news story. The good news is that customers are putting more and more of their IT infrastructure behind VPLEX. The bad news is that what used to be a few steps at smaller scale and therefore barely noticeable, at scale becomes a much bigger problem. Customers have come to us with a simple request – make provisioning simple!

And we listened

We are addressing this problem in multiple ways with the simplest being the Virtual Storage Integrator (VSI) plug in as a mechanism for VMware environments (see here) and the most elaborate being the incredible capabilities of the ViPR controller (see here).

And we are now providing customers with the means to do end-to-end provisioning natively through Unisphere for VPLEX. Basically, from within the VPLEX UI, you can now provision all the way to the backend storage without having to go to the backend UIs. This implies that when you provision storage, you no longer have to navigate between two or three GUIs to provision storage to VPLEX. This capability we are referring to as VPLEX Integrated Array Services (VIAS). The first installment of this GAd in March as a part of GeoSynchrony 5.3 and was launched on April 4th. For the initial release, we support the VMAX and VNX.

So how does it work?

The basic process works by exposing the backend Management interfaces to VPLEX. VPLEX uses SMI-S to talk to the backend arrays. As a result, while we support VMAX and VNX for the initial release, we expect to add more arrays to the VIAS support matrix.

The backend array needs to be configured to some notion of pools (collection of storage) – for the VNX, these are called storage pools, for VMAX, these are called storage groups. These pools are exposed to VPLEX as manageable entities via the management for described earlier.

From there on, the provisioning is managed entirely within VPLEX. You have to select what pool of storage you need to use, what level of protection you want (local mirror, distributed mirror), what consistency group to use and what storage view you want to expose to the host. VPLEX then coordinates the creation of the LUN on the underlying storage, exposing that to VPLEX (on the VPLEX backend), claiming this storage, creating a virtual volume on this new storage, adding into the appropriate consistency group and finally, if so desired putting it into a storage view to expose it to a host on the VPLEX front end. In counting the steps pre-VIAS and post-VIAS, we have been able to reduce the steps by over 90%. The effect of this is nothing short of transformational. Large data center environments suddenly become easy to manage and work with.

Here are some screenshots of how VIAS looks:

image
VIAS: Selecting consistency group
image
VIAS: Selecting protection options
image
VIAS: Selecting storage pool
image
VIAS: Selecting storage view
image
VIAS: Review and finish

Here is a demo put together by Jen Aspesi to show how VIAS works (Her post on VIAS is here).

Once again here is a brief video that Mr. Danahy and I put together to walk through the benefits of VPLEX Integrated Array Services (VIAS).

To us, VIAS represents a quantum leap forward in terms of simplifying the life of the VPLEX administrator and enabling them to address the data center sprawl. Do let us know what you think!

2014 Launch Post 2: MetroPoint: Extending the Availability and Protection Continuum

On April 4th, 2014, as part of the Data Protection and Availability Division (DPAD) launch, there were three VPLEX and RecoverPoint items that were launched or GAd:

  • VPLEX Virtual Edition – Availability late Q2
  • MetroPoint Topology – Joint capability of VPLEX and RecoverPoint – Availability Late Q2
  • VPLEX Integrated Array Services – Available now

This is the second in a series of posts to walk through what was launched / delivered.

VPLEX and RecoverPoint

It has been two years since we introduced the RecoverPoint splitter within VPLEX. The awesomeness of VPLEX was joined with the coolness of RecoverPoint. With this combination, we delivered operational and disaster recovery to VPLEX customers to add to the continuous availability that they already had access to. These were extremely complementary use-cases. While there were a lot of skeptics outside of EMC about this combination, we were quietly confident in our belief that customer wanted an extended continuum between disaster recovery and continuous availability. Suffice it to say, that this combination has exceeded our revenue expectations. Since the launch in May 2012, the organizations have come even closer together within a single business unit further solidifying the bonds between the two teams.

A quick recap of the current integration points between VPLEX and RecoverPoint.

RecoverPoint delivers continuous data protection enabling local and/or remote protection. This is enabled by a RecoverPoint splitter which resides within the VPLEX platform. RecoverPoint has a similar splitter in the VMAX and VNX platform as well. The RP splitter enables WRITES to be sent to a RecoverPoint Appliance (RPA). From there, you can enable local protection (where the writes are journaled locally) or remote protection (where the writes are journaled remotely) or both. The beauty of RecoverPoint is that it can store every single write to give recovery a DVR like capability. The other benefit of RecoverPoint is that the protection is heterogeneous i.e. it can protect between every combination of VPLEX / VMAX and VNX.

The combination of VPLEX and RecoverPoint supports the following topologies:

  1. VPLEX Local with RecoverPoint Local Protection
  2. VPLEX Local with RecoverPoint Remote Protection
  3. VPLEX Metro with RecoverPoint Local Protection
  4. VPLEX Metro with RecoverPoint Remote Protection
  5. The slide below shows the currently supported topologies.

    Currently supported VPLEX and RecoverPoint topologies
    Currently supported VPLEX and RecoverPoint topologies

    Customer topologies are all over the map – we see a lot of traction with the VPLEX Local and RecoverPoint Remote Protection (as we expected). However, the second largest topology is the three sided cascaded topology. And that was a surprise. Upon digging further, a lot of customers have business requirements that need them to have out-of-region disaster recovery site. Yet other customers are deploying VPLEX Metro within one site. So, the usage of RecoverPoint in this case is to provide DR to a Metro deployed within the site. This is the cascaded topology.

    As you can imagine, the downside of the cascaded topology is that if the replicating VPLEX Cluster fails or loses connectivity, DR protection is lost. Since the launch of RecoverPoint on VPLEX quite a few customers have been asking us to add the capability to protect both sides of a VPLEX Metro to a common third site using RecoverPoint. Well, that is exactly what we have done.

    MetroPoint: Operational and Disaster Protection across both sides of a VPLEX Metro

    MetroPoint Topology
    MetroPoint Topology

    The MetroPoint solution launched April 4th will GA at the end of Q2. This is a joint capability between RecoverPoint and VPLEX. Starting with RecoverPoint 4.1 and GeoSynchrony 5.4, customers will now be able to add Disaster Recovery and Operational Recovery protection to both sides of a distributed volume. With MetroPoint, we took the time to do this right – although the protection is on both sides of a distributed volume, only one of the sides is replicating data. The data goes to a single copy of a DR leg. In other words, no additional bandwidth or storage is needed to enable MetroPoint as compared to enabling a standard DR scenario.

    To enable this, we have created a new kind of consistency group called MetroPoint consistency groups. This enables replication on both sides of a distributed volume. Another characteristic of the MetroPoint consistency group is that you can load balance which site is the primary replication site. If there is a failure on the primary replication site, the replication will AUTOMATICALLY switch to the surviving site. In other words, there is no loss of DR protection even if you lose the primary replication site.

    To me, one of the more exciting implications is the extension of the VMware HA and VMware SRM use cases to the MetroPoint topology. Here is what this looks like:

    image
    MetroPoint with VMware HA and SRM

    The VPLEX Metro sites are protected with VMware HA and the remote DR site is protected with VMware SRM. This now gives our customers simultaneous HA and DR.

    One comment here: We talk about MetroPoint as a three site deployment and that is true. However, it is worth remembering that there are a number of customers who deploy VPLEX Metro within a data center either to protect multiple floors or multiple SANs or across a campus type environment. In those scenarios, customers can use MetroPoint to protect to a second site. There is a lot of interest in this deployment model.

    More coolness – along the way, we were able to meet one more request that our customers had requested. With the MetroPoint consistency group, we were able to provide operational recovery on both sides of a VPLEX Metro. And this does not need a third site!!

    Operational Recovery on both sides of a VPLEX Metro
    Operational Recovery on both sides of a VPLEX Metro

    To top this all off, MetroPoint is completely heterogeneous. All these goodies work with both EMC as well as non-EMC arrays. So long as the storage array is supported by VPLEX, you are good to go.

    Here is a short video that Paul Danahy and I put together to give you brief overview of MetroPoint:

    With MetroPoint, we have raised the bar on continuous availability and disaster recovery. This has been the result of collaboration between the VPLEX and RecoverPoint engineering team with a lot of input from some of our lead customers. To all those who helped us get here, a very BIG thank you!

2014 Launch Post 1: Software Defined Coolness: VPLEX Virtual Edition!!

2014-04-08 One Correction below

On April 4th, 2014, as part of the Data Protection and Availability Division (DPAD) launch, there were three VPLEX and RecoverPoint items that were launched or GAd:

  • VPLEX Virtual Edition – Availability late Q2
  • MetroPoint Topology – Joint capability of VPLEX and RecoverPoint – Availability Late Q2
  • VPLEX Integrated Array Services – Available now

This is the first in a series of posts to walk through what was launched / delivered.

The drivers towards a VPLEX Virtual Edition

Data center infrastructure is undergoing a massive shift. Virtualization in the data center has had a profound impact on customer expectations of flexibility and agility. Especially as customers get to 70+% virtualized, they have the potential to realize tremendous operational savings by consolidating management in their virtualization framework. In this state, customers typically do not want to deploy physical appliances and want everything handled from their virtualization context. Similar changes in networking and storage have meant that the basic infrastructure is now completely in software running on generic hardware. This is the software defined data center. VPLEX has been no stranger to this conversation. Especially given the very strong affinity of VPLEX to VMware use-cases, customers have been asking us for a software only version of VPLEX. That is precisely what we have done. This past week, we launched VPLEX Virtual Edition – with a GA towards the end of Q2.

What is the VPLEX Virtual Edition and what does it do?

The VPLEX Virtual Edition (VPLEX/VE) is a vApp version of VPLEX designed to run on an ESX Server Environment to provide continuous availability and mobility within and across data centers. We expect this to be the first in a series of virtual offerings. In comparison to the appliance, all the VPLEX directors are converted into vDirectors. For the first release, the configuration we support is called the ‘4×4’ – this will support four vDirectors on each side of a VPLEX Metro. From a configuration standpoint, that is the equivalent of two VPLEX engines on each side of a VPLEX Metro cluster. Each side of VPLEX/VE can be deployed within or across data centers up to 5 msec apart.

4x4 VPLEX/VE Topology
4×4 VPLEX/VE Topology

VPLEX/VE supports iSCSI for front-end and back-end connectivity. For the initial release, we have decided to support only the VPLEX Metro equivalent use-cases. Most of the VPLEX Local related use-cases can be addressed by a combination of vMotion and storage vMotion. To list the use-cases:

  • The ability to stretch VMware HA / DRS clusters across data centers for automatic restart and protecting VMs across multiple data arrays
  • Load balancing of virtual machines across data centers
  • Instant movement of VMs across distance
VPLEX Virtual Edition Supported Use Cases
VPLEX Virtual Edition Supported Use Cases

From a performance perspective, VPLEX/VE is targeted up to a 100K IOPS workload. Obviously, the true performance will depend on your workload. The deployment is designed to be customer installable from the get go. There is an installation wizard that guides you all the way through the installation. When GAd, please refer to the release notes to determine what kind of ESX Servers are supported for VPLEX/VE. The vDirectors need to be loaded onto separate ESX Servers such that no two vDirectors are deployed on the same ESX server. This is done so as to give the system maximum availability. Running application VMs on the same ESX server as that running the vDirector is supported. This means that you should be able to use your existing ESX servers (subject to the minimum requirement that will be established for the vDirectors).

The way that an I/O will flow is from the application VM (via iSCSI) to the VPLEX/VE vDirector VM and from there to the iSCSI array connected to VPLEX/VE. Speaking of which, right out of the chute, we support VNXe arrays. We will add other iSCSI arrays over time.

One of the more interesting changes that we have made with VPLEX/VE is the way that it is managed. Since VPLEX/VE is tailored for ESX servers only, our management interface to VPLEX/VE is completely through the vSphere Web Client. Here are some screenshots of how VPLEX/VE management looks. The coolest part for me is that you can go from creating your VMs, setting up an HA cluster, all the way to creating a distributed volume all within the vSphere Web Client. _VERY_ nifty! In addition, we have now enabled VPLEX/VE events and alarms to show up in the vCenter Event Viewer. For all practical purposes, this is a seamless vApp designed for your vSphere environments.

Customer installable
Customer installable
VPLEX Virtual Edition in vSphere Web Client
VPLEX Virtual Edition in vSphere Web Client
VPLEX/VE vDirectors in vSphere Web Client
VPLEX/VE vDirectors in vSphere Web Client
VPLEX/VE Operations in vSphere Web Client
VPLEX/VE Operations in vSphere Web Client

When a distributed volume is provisioned for VPLEX/VE, it is configured as a vmfs 5 volume and made available as a resource to vCenter.

With VPLEX/VE, we have had the opportunity to do a lot of things differently. One of our guiding principles was to not think of it as a storage product but rather to think of it as a product designed for VMware environments and targeted to an ESX Administrator. Naturally, I cannot wait to see this get into our customers hands and to see whether we have hit our marks and what adjustments are needed.

Equally importantly, this is a strategic imperative within EMC. You can expect to see a lot more of our product portfolio embarking on the software defined journey. There are a lot of intersects within the portfolio that we have only begun to explore (HINT: Composing software is a lot easier than composing hardware!).

Frequently Asked Questions

Since launch, I have seen a ton of questions on twitter, on internal mailing lists and via people directly or indirectly reaching out to me. So, here are the collated answers:

  • Is VPLEX/VE available right now?
    • A: VPLEX/VE will GA towards the end of Q2.
  • Will VPLEX/VE support non-EMC arrays?
    • A: As with VPLEX, we expect to qualify additional EMC and non-EMC arrays over time based on customer demand. Expect new additions fairly quickly after GA
  • Will I be able to connect VMs from ESX clusters that are not within the same cluster as the one hosting VPLEX/VE?
    • A: Yes No
  • Will I be able to connect non-VMware ESX hosts to VPLEX/VE?
    • A: At this point, we only support VMware iSCSI hosts connecting to VPLEX/VE. This is one of the reasons the management has been designed within the vSphere Web Client
  • Can I connect VPLEX/VE with VPLEX?
    • A: VPLEX/VE is deployed as a Metro equivalent platform (i.e. both sides). Connecting to VPLEX is not supported. If there are interesting use-cases of this ilk, we would love to hear from you. Please use the comments section below and we can get in touch with you.
  • Is RecoverPoint supported with VPLEX/VE>
    • A: Not today. So, I am explicit – the MetroPoint topology which also launched last week is also not supported with VPLEX/VE
  • Is VPLEX/VE supported with ViPR?
    • A: At GA, ViPR will not support VPLEX/VE. Both the ViPR and VPLEX/VE teams are actively looking at this.
  • Does VPLEX/VE support deployment configurations other than a 4×4?
    • A: Currently, 4×4 is the only allowed deployment configuration. Over time, we expect to support additional configurations primarily driven by additional customer demand.
  • Will VPLEX/VE be qualified under vMSC (vSphere Metro Storage Cluster)?
    • A: Yes.

    If you are interested in a Cliff’s note version of this, here is a short video that Paul and I did to walk through the virtual edition: