Category Archives: EMC

Customer Focus: The EMC Way

Every company has some core founding principles / values – some rather overt, some implicit. These are not principles related to product or technology. Nor are they related to vision. More often than not, these values are not written down. You only learn these through oral traditions of stories / myths at bars through people who have been in the organization long enough. Rarely you get to experience these first hand.

So why bring this up now?

After I shipped myself to the west coast, I am in the odd position of being in a minority (a person who was with EMC but moved to Isilon, from Hopkinton to Seattle). I am a conduit of these very myths – some I have learned (The ‘Yes it does snow in New England’), others I have lived. This is one in the latter category.

This story is from many _many_ years ago. All names (except some key principals who I am sure won’t mind) have been kept confidential for obvious reasons.

I had recently taken on a management role within the engineering organization. I was responsible for SW development and customer escalation management.

As most stories go, this one started with a rather innocuous request from a customer (BTW, they have since become one of my favorite customers – visionary, drive technology, take calculated risks and in every way, partner with us to build better products. It also helps that they are a household name – one of the few ways I can help my non techie family members understand what it is I work on). The request was for them to migrate between data centers with a special request to ensure that engineering was involved.

As it came to us, this seemed like a normal request and we assumed that engineering involvement was needed largely for review. Then the oddness started – the product we had was specifically designed for migrations. But the customer was apparently not using this product for migration.

We dug in, contacted the account team who contacted the customer. Turns out it is a migration, except it isnt a copy of the data but rather a physical move of infrastructure. And the customer wanted to keep their operations online through the physical move and were convinced that they could do this with our product.

I have to share this in all honesty – as the person responsible for carrying the quality banner, I was petrified. While the customer, in theory, was RIGHT (aren’t they always?), we had never quite anticipated a customer contemplating using the product in this manner. Oh yeah, the move was going to happen on Thu and we learned about this on the Monday of that week.

Once you get past the seven stages of coming to terms with reality, we got down to brass tacks (and yes, involvement from engineering was going to be more than just review :-)). One of the engineers from my team was going to head down to the customer site (drivable distance from Hopkinton) to perform this ‘move’. He got testing and practicing the move procedure working out any kinks. So far so good and nothing too far out of the ordinary as far as customer escalations go.

One more thing …

So we have worked through the kinks, the engineer going to the customer site was feeling confident. Thursday morning we run through a last check. Lo and behold, we found out that the long distance SFPs that are needed to enable the migration were misplaced at the customer site. For some reason that I cannot remember, these were not SFPs that were just lying around (I seem to remember that the default was to use short distance SFPs). So, here we are at noon on Thu with everything set except the key ingredient to make the move successful.

I remember going to my manager (@MattWaxman) with a complete dead look that basically said, ‘I am out of options’. In an inspired moment, he suggested something off the wall – since then I have learnt that desperation makes you creative – “Let’s email all the people we know at EMC (our PMTs, BMTs, execs, support) with a system wide SOS that said something to the effect of ‘We need long distance SFPs for a customer in the next three hours – here is the model number. Please contact us if you have any of these lying around. We need 24.’.”

We sent that note – expecting this to be a complete Hail Mary with no chance of success.

Not having much else to do, we ran through one more dry run for the move and let the account team know that we may have to cancel since we didn’t have the SFPs. The final go no go was set at 3:00 PM. The engineer was leaving at 4:00 PM for the drive over.

Here is what happened instead.

I returned to my office post that dry run. And on my desk, I had ten to twenty different packages of SFPs – some from people I knew, most of these from people I didn’t know. I had one guy who drove from our factory in Franklin MA with a box of these SFPs. His exact statement to me was – ‘Someone told me that they had heard about you needing SFPs for a customer. I have tested all of these – they work. Make the customer successful’. I had sticky notes that said the same thing.

Needless to say, the engineer was able to take this on their ride to the customer site with them. The move was executed flawlessly. In fact, the customer’s end customers didn’t see a single app bounce. This customer and that account team became one of our biggest advocates.

What did I learn

Customer focus was something Dick Egan intentionally drove into the EMC culture. Many many years removed from his direct involvement with the company, EMC employee #1 continued to cast a large shadow. It tells you how important founders (and the culture they establish) are.

Customer success is everyone’s responsibility whether you work on a product or not. Customers make the world go round. Over time I have been in many situations where my direct responsibility would have caused me to not act. In all those situations, I try to act in the same proactive ownership culture that I was the beneficiary of.

Always focus on what’s right for the customer. The easy answer above would have been to declare the use case as unsupported. The harder answer was to look past the execution risk and focus on what the customer justifiably needed. Many thanks to the customer for pushing us.

Even as I type this blog many years removed from the actual incident, I continue to be touched by the camaraderie and the sheer stick-to-itiveness of the EMC culture that did not allow us and the customer to fail.

It is one of those rare moments where it feels like the entire company stands behind you as an individual helping you succeed. In my day-to-day interactions within EMC, I continue to use this incident as the yardstick by which I measure myself.

2014 Launch Post 1: Software Defined Coolness: VPLEX Virtual Edition!!

2014-04-08 One Correction below

On April 4th, 2014, as part of the Data Protection and Availability Division (DPAD) launch, there were three VPLEX and RecoverPoint items that were launched or GAd:

  • VPLEX Virtual Edition – Availability late Q2
  • MetroPoint Topology – Joint capability of VPLEX and RecoverPoint – Availability Late Q2
  • VPLEX Integrated Array Services – Available now

This is the first in a series of posts to walk through what was launched / delivered.

The drivers towards a VPLEX Virtual Edition

Data center infrastructure is undergoing a massive shift. Virtualization in the data center has had a profound impact on customer expectations of flexibility and agility. Especially as customers get to 70+% virtualized, they have the potential to realize tremendous operational savings by consolidating management in their virtualization framework. In this state, customers typically do not want to deploy physical appliances and want everything handled from their virtualization context. Similar changes in networking and storage have meant that the basic infrastructure is now completely in software running on generic hardware. This is the software defined data center. VPLEX has been no stranger to this conversation. Especially given the very strong affinity of VPLEX to VMware use-cases, customers have been asking us for a software only version of VPLEX. That is precisely what we have done. This past week, we launched VPLEX Virtual Edition – with a GA towards the end of Q2.

What is the VPLEX Virtual Edition and what does it do?

The VPLEX Virtual Edition (VPLEX/VE) is a vApp version of VPLEX designed to run on an ESX Server Environment to provide continuous availability and mobility within and across data centers. We expect this to be the first in a series of virtual offerings. In comparison to the appliance, all the VPLEX directors are converted into vDirectors. For the first release, the configuration we support is called the ‘4×4’ – this will support four vDirectors on each side of a VPLEX Metro. From a configuration standpoint, that is the equivalent of two VPLEX engines on each side of a VPLEX Metro cluster. Each side of VPLEX/VE can be deployed within or across data centers up to 5 msec apart.

4x4 VPLEX/VE Topology
4×4 VPLEX/VE Topology

VPLEX/VE supports iSCSI for front-end and back-end connectivity. For the initial release, we have decided to support only the VPLEX Metro equivalent use-cases. Most of the VPLEX Local related use-cases can be addressed by a combination of vMotion and storage vMotion. To list the use-cases:

  • The ability to stretch VMware HA / DRS clusters across data centers for automatic restart and protecting VMs across multiple data arrays
  • Load balancing of virtual machines across data centers
  • Instant movement of VMs across distance
VPLEX Virtual Edition Supported Use Cases
VPLEX Virtual Edition Supported Use Cases

From a performance perspective, VPLEX/VE is targeted up to a 100K IOPS workload. Obviously, the true performance will depend on your workload. The deployment is designed to be customer installable from the get go. There is an installation wizard that guides you all the way through the installation. When GAd, please refer to the release notes to determine what kind of ESX Servers are supported for VPLEX/VE. The vDirectors need to be loaded onto separate ESX Servers such that no two vDirectors are deployed on the same ESX server. This is done so as to give the system maximum availability. Running application VMs on the same ESX server as that running the vDirector is supported. This means that you should be able to use your existing ESX servers (subject to the minimum requirement that will be established for the vDirectors).

The way that an I/O will flow is from the application VM (via iSCSI) to the VPLEX/VE vDirector VM and from there to the iSCSI array connected to VPLEX/VE. Speaking of which, right out of the chute, we support VNXe arrays. We will add other iSCSI arrays over time.

One of the more interesting changes that we have made with VPLEX/VE is the way that it is managed. Since VPLEX/VE is tailored for ESX servers only, our management interface to VPLEX/VE is completely through the vSphere Web Client. Here are some screenshots of how VPLEX/VE management looks. The coolest part for me is that you can go from creating your VMs, setting up an HA cluster, all the way to creating a distributed volume all within the vSphere Web Client. _VERY_ nifty! In addition, we have now enabled VPLEX/VE events and alarms to show up in the vCenter Event Viewer. For all practical purposes, this is a seamless vApp designed for your vSphere environments.

Customer installable
Customer installable
VPLEX Virtual Edition in vSphere Web Client
VPLEX Virtual Edition in vSphere Web Client
VPLEX/VE vDirectors in vSphere Web Client
VPLEX/VE vDirectors in vSphere Web Client
VPLEX/VE Operations in vSphere Web Client
VPLEX/VE Operations in vSphere Web Client

When a distributed volume is provisioned for VPLEX/VE, it is configured as a vmfs 5 volume and made available as a resource to vCenter.

With VPLEX/VE, we have had the opportunity to do a lot of things differently. One of our guiding principles was to not think of it as a storage product but rather to think of it as a product designed for VMware environments and targeted to an ESX Administrator. Naturally, I cannot wait to see this get into our customers hands and to see whether we have hit our marks and what adjustments are needed.

Equally importantly, this is a strategic imperative within EMC. You can expect to see a lot more of our product portfolio embarking on the software defined journey. There are a lot of intersects within the portfolio that we have only begun to explore (HINT: Composing software is a lot easier than composing hardware!).

Frequently Asked Questions

Since launch, I have seen a ton of questions on twitter, on internal mailing lists and via people directly or indirectly reaching out to me. So, here are the collated answers:

  • Is VPLEX/VE available right now?
    • A: VPLEX/VE will GA towards the end of Q2.
  • Will VPLEX/VE support non-EMC arrays?
    • A: As with VPLEX, we expect to qualify additional EMC and non-EMC arrays over time based on customer demand. Expect new additions fairly quickly after GA
  • Will I be able to connect VMs from ESX clusters that are not within the same cluster as the one hosting VPLEX/VE?
    • A: Yes No
  • Will I be able to connect non-VMware ESX hosts to VPLEX/VE?
    • A: At this point, we only support VMware iSCSI hosts connecting to VPLEX/VE. This is one of the reasons the management has been designed within the vSphere Web Client
  • Can I connect VPLEX/VE with VPLEX?
    • A: VPLEX/VE is deployed as a Metro equivalent platform (i.e. both sides). Connecting to VPLEX is not supported. If there are interesting use-cases of this ilk, we would love to hear from you. Please use the comments section below and we can get in touch with you.
  • Is RecoverPoint supported with VPLEX/VE>
    • A: Not today. So, I am explicit – the MetroPoint topology which also launched last week is also not supported with VPLEX/VE
  • Is VPLEX/VE supported with ViPR?
    • A: At GA, ViPR will not support VPLEX/VE. Both the ViPR and VPLEX/VE teams are actively looking at this.
  • Does VPLEX/VE support deployment configurations other than a 4×4?
    • A: Currently, 4×4 is the only allowed deployment configuration. Over time, we expect to support additional configurations primarily driven by additional customer demand.
  • Will VPLEX/VE be qualified under vMSC (vSphere Metro Storage Cluster)?
    • A: Yes.

    If you are interested in a Cliff’s note version of this, here is a short video that Paul and I did to walk through the virtual edition: