DOES17

   4 years ago
#DOES17DOES17 London Join DevOps Enterprise Summit Community to preview DOES17 London and San Francisco
DevOps Enterprise Summit
Are there any anti-patterns of DevOps? If so, please explain.
Jonathan Fletcher
Creating a DevOps team that is itself a silo
DevOps Capybara
Creating a third team called “DevOps” that is the approval committee for anything “DevOps” to be approved and rolled out in the organization through the Project Management Office.
DevOps Capybara
@FletcherJofanon Exactly. I see this all.the.time.
Tapabrata Pal
My most favorite anti-patter is having a team that says "we are the devops team and we set up everything for you and we have a blacklog to manage your request"
Ben Grinnell
DevOps without regular releases and feedback from real users contributing to the product backlog - fake DevOps
Joe Brown
Just attaching DevOps as a label to everything because DevOps is a really cool word.
Rosalind Radcliffe
Silos - including creating separate DevOps team, breaking down the silos and providing visibly across the systems
Mark Schwartz
Commenting out failing tests!
Carmen DeArdo
centralized command control which create wait states in the value stream. This can be a change board or a centralized Performance testing team or Security testing team
Jonathan Fletcher
I read countless CV's from people that once wrote a batch file and that are now "DevOps engineers"
Joe Brown
Top down driven DevOps with practices defined at the enterprise level.
Paula Thrasher
Using the word DevOps to describe your old school, siloed way of working because you implemented one DevOps tool
Tapabrata Pal
Another anti-pattern - overhearing in post-mortem - "who did it?"
Rosalind Radcliffe
Pushing from the top without having bottoms up support and effort
Paula Thrasher
@FletcherJofanon Creating a silo DevOps Team is absolutely THE anti-pattern
Gene Kim
OH @carmendeardo : "like people going to quality conference & coming back and creating a Quality Team"
Scott Nasello
OH - The Load Balancing team can't load balance themselves @topopal @realgenekim
Paula Thrasher
DevOps anti-pattern is DevOps = tool. You need tools but you need collaboration, systems thinking feedback more
Tapabrata Pal
"Center of excellence" has a sense of eliteness to it.
Scott Nasello
@topopal - and lots of COEs often don't demonstrate excellence
Gene Kim
Sometimes not obvious at all: amazing PPT by @andyburgin on Sky Betting Data Warehouse https://www.youtube....
Paula Thrasher
Big Bang thinking is still a big anti-pattern. DevOps requires smaller batches. Hard to claim "DevOps" with a 6 mo deployment cycle
Gene Kim
OH @scottprugh "Once people/teams go above 50%, queue times go exponential"
Jason A Cox
@topopal Anti-pattern: Architecture and engineering excellence can only come from CoE experts & towers
Sam Guckenheimer
There is a ton of SrummerFall out there with the kabuki of Lean/Agile terminology over a big-batch waterfall process.
Gene Kim
OH: "[without slack time], we feel like we're just stuck in a feature factory [that doesn't actually create value]"
Sam Guckenheimer
Another very common antipattern is not investing in engineering system improvement.
Sam Guckenheimer
On antipatterns, I'd also tag V-model thinking, that forces slow testing over fast feedback.
Mark Schwartz
Assumptions about user behavior, rather than user-centric design and testing in production
Sam Guckenheimer
@schwartz_cio +! on testing in production, with quantitative and qualitative validation.
Gene Kim
How do you deal with balancing innovation vs innovation? @topopal @scottprugh
Scott Prugh
Innovation vs Standards... :)
Scott Prugh
Really hard. We have a bunch of standards that are non-negotiable: OS standards, build standards, config management standards.
Gene Kim
@topopal What are some of the carrots/sticks you're using to encourage some standardization?
Sam Guckenheimer
I think you need a rigid def of done for prod (e.g. deployed worldwide with telemetry) and to allow time for innovation.
Carmen DeArdo
We are working with teams to run experiments to accelerate their delivery. We take these results and apply "systems thinking" to figure out what should be a local optimization versus what we can implement across the enterprise.
Jonathan Fletcher
The increasing focus & awareness of security makes certain conversations easy
Rosalind Radcliffe
Understanding the goals behind the standardization is critical to help drive innovation
Sam Guckenheimer
Feature Flags are a great pattern for allowing innovation in previews and dark launches.
Scott Prugh
The stick of not having the CM/build standards is you have to spend weeks with auditors...
Scott Nasello
@scottprugh - how do you enforce your non-negotiable standards?
Carmen DeArdo
Too much non-value variation doesn't allows us to implement a pipeline product across 20+ lines of business.
Gene Kim
OH: @topopal : "downside of too much innovation: too many snowflakes, and difficult to comply with infosec needs"
Paula Thrasher
we meet as a team to talk IT standards. Sometimes its ok to innovate, but sometimes you need to pick just one tool
Chivas Nambiar
"You build it, you own it" is a great way to do that. Common standards are managed for you so you don't need to run large platforms, new innovation is managed by the app teams themselves.
Gene Kim
OH: @topopal : "Carrot: make our tools/platforms are much better, easier, more reliable than competing toolsets"
Rosalind Radcliffe
Standards that have reasons behind them help clarify why and providing capabilities that make it easier for people to do work
rzesz
configuration management can be a great tool to balance the give and take, if all innovation starts with configuration management, standardization is easier to swallow
Sam Guckenheimer
The best carrot/stick combo is automated tests that govern the pull request->test->commit->test->stage->deploy flow and obvious metrics in real time.
Mark Schwartz
Standardization is way overrated. There were good reasons we needed it in the past, but it's not a goal in itself.
Gene Kim
OH: @paula_thrasher "We have a mtg going on right now where we're asking 'do we really need 2/3/4 of these things?"
Tapabrata Pal
Encourage innovation but put constraints around it saying "it better be more than just a POC"
Sam Guckenheimer
Catch the errors in the pull request flow BEFORE commit. That's shifting left. Fast feedback works as both carrot and stick.
Carmen DeArdo
our metrics, such as lead time, are based on using the product pipeline. So if you want visibility into wait states and process times, you have to be using the product. We use Continuous Improvement to get associate input.
Mark Imbriaco
Innovate where you differentiate. If you need a new tool, use it. But make sure you need it.
Paula Thrasher
standards are good to keep management costs down, you just have to be flexible
Gene Kim
OH: @paulycomtois: "standards are really great for helping get conversation and teams going; otherwise like artist w/blank canvas
Rosalind Radcliffe
Providing the standard functions that help users and provide value help people move to it
Mark Schwartz
@paula_thrasher Keeping costs down is a goal, not standardization. But you have to weigh the costs of standards versus the benefits
Paula Thrasher
Standards should be a TEAM decision, not a rule from on high
Carmen DeArdo
We are also implementing tools with the idea we can utilize APIs and replace them quickly. So we didn't have a long debate about Slack we just started using Rocket.Chat and figure we can swap it out if we want to later for Slack or ...
Tapabrata Pal
Also harvest innovations to allow sharing and reuse across the enterprise
Jason A Cox
Standards and patterns should come from the wake of innovation, not restrict innovation.
Scott Prugh
Enforcement is hard. First we are working hard to make the data visible and map it to owners. Second, train and educate the importance of the what the standards bring: audit controls, information sharing, speed.
Sam Guckenheimer
@schwartz_cio Depends what you mean by standards. Standard pipeline automation and def of done with real deployment is essential.
Mark Imbriaco
Make the standard path so easy that you'd need a really good reason not to use it.
Carmen DeArdo
@SamGuckenheimer Dark Launches has been a big win for us
Scott Prugh
Third, make the standards visible and self service. Don't queue them behind a team that is the only one that can execute
Paula Thrasher
Teams will embrace standards if they have a role in choosing them, and the standards help them work
Gene Kim
OH @topopal: "many teams are starting to come back to our standard tools"
Scott Prugh
Fouth: Involve the teams in choosing the standards
Carmen DeArdo
We don't govern teams - we provide standard metrics based on the State of DevOps report and provide self-service capabilities
Mark Schwartz
@SamGuckenheimer Yes, but to me the pipeline is a tool. It takes the place of a standard, which says that you have to build everything a certain way. Using tool != standard.
Gene Kim
OH @topopal "Just b/c teams bring their own tools doesn't mean they can't comply with infosec/compliance standards, but..."
Rosalind Radcliffe
government regulations also force certain standardization that has to be followed
Paula Thrasher
Our cloud tools have a lot of standards too, the shared environment means it matters that we agree to the same set of tools to go faster
Sam Guckenheimer
@topopal Tools are more important than the product instances. The machine tools of the digital age are automated pipeline and telemetry. The tools will outlast the products made with them.
Mark Schwartz
perhaps we are mixing lots of different concepts: standards (do things this way), shared tool, regulatory constraints (to me, satisfying them adds business value and is a true desired outcome)
Sam Guckenheimer
@schwartz_cio Amen. It does not make sense to dictate process without automation to give fast feedback.
Carmen DeArdo
@SamGuckenheimer agree - the pull request is very powerful for us
Rosalind Radcliffe
Some standards are there just historically, those are the ones that need to be reviewed and removed if not really required
Jason A Cox
"Standard by Awesome" is also important - patterns that amplify capability, reduce friction in work & make things suck less will increase adoption.
Gene Kim
With freedom comes great responsibility. :) (compliance/security)
Scott Prugh
@jasonacox Yes!! Make them self serve, visible and easy to use!!
Rosalind Radcliffe
the other key point is these internal standards need to be reviewed regularly to make sure they are still valid, things do change over time
Mark Schwartz
@jasonacox I like that formulation a lot
Mark Schwartz
Face it, standards are a perfect example of what we call bureaucracy :-) her hee
DevOps Capybara
We use standardization as the starting point, then innovate on top. Big wins work back into the standardization where appropriate, but we never limit based on standardization unless it's a compliance issue.
Carmen DeArdo
the best teams have the street credibility that really drive others to want to use it and also improvements
Gene Kim
OH @allspaw: "you need all teams to be involved in achieving the goals; vs dictates from way up high"
Ben Grinnell
I think its more important to make sure where you are innovating you are doing it scientifically, working in a repeatable way so you get valueable organisational learning whatever happens, "Ad-hoc and agile are very different methods"
Stuart Miniman
the great thing about standards is that there is so many of them 😉
Gene Kim
OH @scottprugh: "State of DevOps Report showed teams who chose their own tools performer better"
GitLinks
Using continuous integration software to monitor vulnerabilities and legal compliance.
Gene Kim
OH: @allspaw: "at Etsy, we had architecture 'review' to encourage critical thinking; not to vet or enforce compliance
Carmen DeArdo
I read that - I think their also a level of maturity around dependencies. If you still have systems with many dependencies, you won't have visibility to the work and delivery without some standard integrated pipeline capabilities.
Scott Prugh
My worry is that this statement will be misunderstood to mean: "Every team should choose their own tool to solve a common problem... :)"
Paula Thrasher
If every silo picks a 'standard', thats not very DevOps. If all teams help choose the standard, we can debate trade offs and select the most benefit for the entire system/enterprise
Tapabrata Pal
Pipeline tools are first class citizens in an enterprise today
Scott Nasello
@paula_thrasher - agree! What would happen if you rotate engineers between silos?
Scott Nasello
Love this quote from @allspaw - Carpenters don't carry 100 hammers around
Paula Thrasher
@scottnasello I am rotating teams now! Four of my developers are helping our Ops and service desk team. We'll see what they learn
Chivas Nambiar
@scottprugh Choice is important, but any good engineering team evaluates their tools. If the common/standard tooling is better than what they can build, they will use it. Common tools teams need to think like product engineering
Mark Schwartz
@ScottPrugh What's interesting is whether we should stop this by mandate, or by encouraging teams to use good judgment and making them aware of why it might be a good idea to share tools and practices
Scott Nasello
@paula_thrasher - sounds like an awesome retrospective opportunity as well
Scott Prugh
@schwartz_cio @ChivasNambiar The education piece is important so that teams understand the required outcomes: compliance, maintainability, etc. Involve teams in discussion of to develop shared understanding. Get them to help with the final "mandate"
Mark Schwartz
@ScottPrugh yes yes yes. instead teams often view compliance, etc as an impediment
Scott Nasello
@paula_thrasher let's talk offline - We are 6 months ahead of you on rotations :)
Paula Thrasher
@scottnasello will have to hear more about your experiment. So far, the teams on rotation seem to be enjoying it and the teams that got our developers don't want them to leave (probably because they automated stuff when they got there!)