EMCCloud

Openstack Operations
OpenStack Operations: Challenges and Solutions
Azhagarasu A
Dear Experts, May be this is a very basic question but I would like to understand what would happen to instances running on a Nova compute node (KVM based) when it fails and disappears from cluster?
cloud foreign
those instances would be unavailable. Your ability to resurrect them depends wholly on your architecture. But cloud advocates cattle over pets.
Sean Winn
That depends on how you created that instance. Heat has ways to deal with this using Ceilometer and autoscaling. There's also PaaS.
Sean Winn
also application architecture has a tremendous impact on what effect the outage will have
Kenneth Hui @rubrikInc HQ
It's assumed that app owner is responsible 4 handling failures.
Scott Carlson
If you have a single compute node with local storage, they die. If you have shared storage, you can likely resurrect elsewhere.
Sean Winn
most of the hypervisors support some form of vm migration - live or not depends on other architectural choices like shared storage
Shamail Tahir
The loss of a VM != loss of service necessarily
Kenneth Hui @rubrikInc HQ
This brings up an interesting ops question - should OpenStack pivot 2 provide HA 4 traditional enterprise apps?
Azhagarasu A
@craig_tracey Agreed. But instead of using KVM as hypervisor. if i use vCenter (ESXi) and integrate OpenStack. would that make a high available solution?
Scott Carlson
@hui_kenneth If it doesn't, the people who want to run traditional enterprise apps CANNOT move.... you always have vmware with vmotion underneith though.
Shamail Tahir
@craig_tracey If you use vCenter then you could leverage vMotion, HA, etc. to make the instance resilient.
cloud foreign
again, that all boils down to your architecture. For me, relying on a hypervisor to provide "reliability" is not an option. The application itself needs to be fault tolerant and cloud aware.
Sean Winn
@hui_kenneth #OpenStack lets the architect/deployer choose what level of availability there is. I usually advocate application resiliency over redundant infra, but some policies require it.
Scott Carlson
@craig_tracey Many "COTS" software cannot be configured w/o much IT pain. Those are the apps that have the toughest time becoming "cloud ready" unless you stick them on VSphere with VMotion
cloud foreign
I can guarantee that the most heavily utilized web services on the planet are not migrating failed VM's.
Shamail Tahir
@craig_tracey @seanmwinn Agreed, but depending on the use-case the user may not always have access to modify app. This is the duality of OpenStack use cases.
Azhagarasu A
@relaxed137 Yes its a shared storage from dedicated storage nodes and also in my example its multiple compute nodes.
Kenneth Hui @rubrikInc HQ
@relaxed137 @craig_tracey What are ur thoughts on vSphere with OpenStack & with VIO, VMware's OpenStack distro?
cloud foreign
@ShamailXD 100% agree. But this is why folks need to stop considering lift and shift of legacy apps.
David Pollack
@relaxed137 Aren't there growing number of tools to help with the transition?
Scott Carlson
@hui_kenneth i'm not a fan of 'vendor' distro unless you are willing to go "all in" with that distro and make it meet your needs. sometimes you have to adjust your business process rather than adjusting your cloud. It depends on if you have devs and ops.
cloud foreign
@DavidMPollack absolutely! OpenStack even has an attempt or 10 at this sort of thing. My bets are are Cloud Foundry at the moment.
Shamail Tahir
@craig_tracey I see users making that decision but we need to, as a community, help them understand the decision.
Azhagarasu A
@ShamailXD But can i eliminate this by using ESXi hypervisor?
David Pollack
@ShamailXD Great as a community, but it sure does hurt when EMC or others step to plate from business driving perspective
Scott Carlson
@DavidMPollack sure there are and often the "Tech Fighting" is the hardest to overcome. There are still people who don't think workloads can be 'virtualized' and god forbid if we ever talk 'cloud'. remember its just servers, cpu, and ram
Shamail Tahir
You have to use vCenter along with ESXi to leverage the VMware drivers.
Azhagarasu A
@craig_tracey yes, but if someone wants to use the Instance just as a server and he/she plans to run a application of their choice would using ESXi or Citrix help to achieve 99 % availability of instances.
Azhagarasu A
@hui_kenneth Yes i think if we use ESXi or citrix the environment becomes redundant and highly available from the infrastructure perspective.
Azhagarasu A
@ShamailXD yes, vCenter provides good high availability features for a traditional DC. but if OpenStack is used on top of vCenter are we making OpenStack highly available because of the underlying infrastructure?
Shamail Tahir
There are different layers that you need HA. The API, the meta-data, the actual data plane, etc. In this case, you would be accounting for data plane HA and not necessarily the others.
Shamail Tahir
Good discussion going at the moment...
Let's throw out question #2:

What challenges do you still see today in trying to operate an OpenStack cloud?
cloud foreign
I have likened OpenStack to a modern day rube goldberg machine - lots of loosely coupled moving parts. Operating it open requires teams that are both broad and deep.
Sean Winn
Finding operators is still the hardest challenge in my opinion - so many skills to master!
David Pollack
Technical issues, mind set, budgets, ecosystem robustness?
Scott Carlson
People who can troubleshoot. When its broken, its pretty broken. that's not acceptable to a large company. There's lots of VMWARE people in the world, and not enough OPENSTACK people to go around yet
Kenneth Hui @rubrikInc HQ
How do we address the skills gap?
David Pollack
Lack of professional resources biggest issue at carriers?
Kenneth Hui @rubrikInc HQ
@relaxed137 How do we turn VMware ops ppl into OpenStack ops ppl?
Tim Bell
metering and networking at scale are not easy to solve. It's getting there but still needs care to configure correctly.
Shamail Tahir
@relaxed137 @seanmwinn Operators and Architects are clearly in demand. http://www.openstack... is proof. :-)
cloud foreign
@hui_kenneth think the skills gap extends even into the overcloud. Still many operators who are not even cloud (forget openstack) users, let alone operators.
Sean Winn
@hui_kenneth I think companies need to build a culture internally and foster innovation and enthusiasm, otherwise you're just trying to steal people away from the same companies you'll need to work with down the road
Tim Bell
OpenStack is only part of the service ... Subbu’s “OpenStack is not a cloud” gives good examples where a service needs much more (http://www.slideshar...)
Open stack@ebay
OpenStack  Private   Cloud  at  eBay   OpenStack  Design  Summit  –  Nov  2013   1  
Scott Carlson
@hui_kenneth Well, I think you have to pinpoint skill sets. compare and contrast features and then teach them how to do it in the 'other place' for example. vmotion is 'click' in VCenter and 'this and that' in OPENSTACK. then people will start to get it
David Pollack
Is training job for likes of Mirantis and Red Hat? Or by community?
Mike Onsing
Tim Bell from CERN on here. Great to see you sharing with the community.
cloud foreign
@seanmwinn totally agree! We have watched the same people move from OpenStack company to OpenStack company. I think making OpenStack easier to swallow is the first step at bringing more folks into the community.
Sean Winn
@noggin143 Excellent point, the swhat's 'not in the box' is even more important. Systems, storage, virtualization, networking. The day of the generalist is back.
Scott Carlson
@DavidMPollack At some point someone will make an 'openstack 101' that knocks it out of the park. who does that is up to the doers.
Sean Winn
@relaxed137 Challenge accepted :)
Shamail Tahir
@DavidMPollack I think the community will do it but as @relaxed137 said the source of that material is whoever steps up. Here is a good starting point: http://docs.openstac...
OpenStack Training Guides
OpenStack™ Training Guides offer the open source community software training for cloud administration and management for any organization.