DOES17

Creating a DevOps team that is itself a silo

Creating a third team called “DevOps” that is the approval committee for anything “DevOps” to be approved and rolled out in the organization through the Project Management Office.

@FletcherJofanon Exactly. I see this all.the.time.

My most favorite anti-patter is having a team that says "we are the devops team and we set up everything for you and we have a blacklog to manage your request"

DevOps without regular releases and feedback from real users contributing to the product backlog - fake DevOps

Just attaching DevOps as a label to everything because DevOps is a really cool word.

Silos - including creating separate DevOps team, breaking down the silos and providing visibly across the systems

Commenting out failing tests!

centralized command control which create wait states in the value stream. This can be a change board or a centralized Performance testing team or Security testing team

I read countless CV's from people that once wrote a batch file and that are now "DevOps engineers"

Top down driven DevOps with practices defined at the enterprise level.

Using the word DevOps to describe your old school, siloed way of working because you implemented one DevOps tool

Another anti-pattern - overhearing in post-mortem - "who did it?"

Pushing from the top without having bottoms up support and effort

@FletcherJofanon Creating a silo DevOps Team is absolutely THE anti-pattern

OH @carmendeardo : "like people going to quality conference & coming back and creating a Quality Team"

@schwartz_cio lol! yes!

OH - The Load Balancing team can't load balance themselves @topopal @realgenekim

DevOps anti-pattern is DevOps = tool. You need tools but you need collaboration, systems thinking feedback more

"Center of excellence" has a sense of eliteness to it.

@topopal - and lots of COEs often don't demonstrate excellence

Sometimes not obvious at all: amazing PPT by @andyburgin on Sky Betting Data Warehouse https://www.youtube....

Big Bang thinking is still a big anti-pattern. DevOps requires smaller batches. Hard to claim "DevOps" with a 6 mo deployment cycle

OH @scottprugh "Once people/teams go above 50%, queue times go exponential"

@topopal Anti-pattern: Architecture and engineering excellence can only come from CoE experts & towers

There is a ton of SrummerFall out there with the kabuki of Lean/Agile terminology over a big-batch waterfall process.

OH: "[without slack time], we feel like we're just stuck in a feature factory [that doesn't actually create value]"

Another very common antipattern is not investing in engineering system improvement.

On antipatterns, I'd also tag V-model thinking, that forces slow testing over fast feedback.

Assumptions about user behavior, rather than user-centric design and testing in production

@schwartz_cio +! on testing in production, with quantitative and qualitative validation.

Innovation vs Standards... :)

Really hard. We have a bunch of standards that are non-negotiable: OS standards, build standards, config management standards.

@topopal What are some of the carrots/sticks you're using to encourage some standardization?

I think you need a rigid def of done for prod (e.g. deployed worldwide with telemetry) and to allow time for innovation.

We are working with teams to run experiments to accelerate their delivery. We take these results and apply "systems thinking" to figure out what should be a local optimization versus what we can implement across the enterprise.

The increasing focus & awareness of security makes certain conversations easy

Understanding the goals behind the standardization is critical to help drive innovation

Feature Flags are a great pattern for allowing innovation in previews and dark launches.

The stick of not having the CM/build standards is you have to spend weeks with auditors...

@scottprugh - how do you enforce your non-negotiable standards?

Too much non-value variation doesn't allows us to implement a pipeline product across 20+ lines of business.

OH: @topopal : "downside of too much innovation: too many snowflakes, and difficult to comply with infosec needs"

we meet as a team to talk IT standards. Sometimes its ok to innovate, but sometimes you need to pick just one tool

"You build it, you own it" is a great way to do that. Common standards are managed for you so you don't need to run large platforms, new innovation is managed by the app teams themselves.

OH: @topopal : "Carrot: make our tools/platforms are much better, easier, more reliable than competing toolsets"

Standards that have reasons behind them help clarify why and providing capabilities that make it easier for people to do work

configuration management can be a great tool to balance the give and take, if all innovation starts with configuration management, standardization is easier to swallow

The best carrot/stick combo is automated tests that govern the pull request->test->commit->test->stage->deploy flow and obvious metrics in real time.

Standardization is way overrated. There were good reasons we needed it in the past, but it's not a goal in itself.

OH: @paula_thrasher "We have a mtg going on right now where we're asking 'do we really need 2/3/4 of these things?"

Encourage innovation but put constraints around it saying "it better be more than just a POC"

Catch the errors in the pull request flow BEFORE commit. That's shifting left. Fast feedback works as both carrot and stick.

our metrics, such as lead time, are based on using the product pipeline. So if you want visibility into wait states and process times, you have to be using the product. We use Continuous Improvement to get associate input.

Innovate where you differentiate. If you need a new tool, use it. But make sure you need it.

standards are good to keep management costs down, you just have to be flexible

OH: @paulycomtois: "standards are really great for helping get conversation and teams going; otherwise like artist w/blank canvas

Providing the standard functions that help users and provide value help people move to it

@paula_thrasher Keeping costs down is a goal, not standardization. But you have to weigh the costs of standards versus the benefits

Standards should be a TEAM decision, not a rule from on high

We are also implementing tools with the idea we can utilize APIs and replace them quickly. So we didn't have a long debate about Slack we just started using Rocket.Chat and figure we can swap it out if we want to later for Slack or ...

Also harvest innovations to allow sharing and reuse across the enterprise

Standards and patterns should come from the wake of innovation, not restrict innovation.

Enforcement is hard. First we are working hard to make the data visible and map it to owners. Second, train and educate the importance of the what the standards bring: audit controls, information sharing, speed.

@schwartz_cio Depends what you mean by standards. Standard pipeline automation and def of done with real deployment is essential.

Make the standard path so easy that you'd need a really good reason not to use it.

@SamGuckenheimer Dark Launches has been a big win for us

Third, make the standards visible and self service. Don't queue them behind a team that is the only one that can execute

Teams will embrace standards if they have a role in choosing them, and the standards help them work

OH @topopal: "many teams are starting to come back to our standard tools"

Fouth: Involve the teams in choosing the standards

We don't govern teams - we provide standard metrics based on the State of DevOps report and provide self-service capabilities

@SamGuckenheimer Yes, but to me the pipeline is a tool. It takes the place of a standard, which says that you have to build everything a certain way. Using tool != standard.

OH @topopal "Just b/c teams bring their own tools doesn't mean they can't comply with infosec/compliance standards, but..."

government regulations also force certain standardization that has to be followed

Our cloud tools have a lot of standards too, the shared environment means it matters that we agree to the same set of tools to go faster

@topopal Tools are more important than the product instances. The machine tools of the digital age are automated pipeline and telemetry. The tools will outlast the products made with them.

perhaps we are mixing lots of different concepts: standards (do things this way), shared tool, regulatory constraints (to me, satisfying them adds business value and is a true desired outcome)

@schwartz_cio Amen. It does not make sense to dictate process without automation to give fast feedback.

@SamGuckenheimer agree - the pull request is very powerful for us

Some standards are there just historically, those are the ones that need to be reviewed and removed if not really required

"Standard by Awesome" is also important - patterns that amplify capability, reduce friction in work & make things suck less will increase adoption.

With freedom comes great responsibility. :) (compliance/security)

@jasonacox i love that

@jasonacox Yes!! Make them self serve, visible and easy to use!!

the other key point is these internal standards need to be reviewed regularly to make sure they are still valid, things do change over time

@jasonacox I like that formulation a lot

Face it, standards are a perfect example of what we call bureaucracy :-) her hee

@jasonacox @schwartz_cio @carmendeardo +1 on Standard by Awesome

We use standardization as the starting point, then innovate on top. Big wins work back into the standardization where appropriate, but we never limit based on standardization unless it's a compliance issue.

the best teams have the street credibility that really drive others to want to use it and also improvements

OH @allspaw: "you need all teams to be involved in achieving the goals; vs dictates from way up high"

I think its more important to make sure where you are innovating you are doing it scientifically, working in a repeatable way so you get valueable organisational learning whatever happens, "Ad-hoc and agile are very different methods"

the great thing about standards is that there is so many of them 😉

OH @scottprugh: "State of DevOps Report showed teams who chose their own tools performer better"

Using continuous integration software to monitor vulnerabilities and legal compliance.

OH: @allspaw: "at Etsy, we had architecture 'review' to encourage critical thinking; not to vet or enforce compliance

I read that - I think their also a level of maturity around dependencies. If you still have systems with many dependencies, you won't have visibility to the work and delivery without some standard integrated pipeline capabilities.

My worry is that this statement will be misunderstood to mean: "Every team should choose their own tool to solve a common problem... :)"

If every silo picks a 'standard', thats not very DevOps. If all teams help choose the standard, we can debate trade offs and select the most benefit for the entire system/enterprise

Pipeline tools are first class citizens in an enterprise today

@paula_thrasher - agree! What would happen if you rotate engineers between silos?

Love this quote from @allspaw - Carpenters don't carry 100 hammers around

@scottnasello I am rotating teams now! Four of my developers are helping our Ops and service desk team. We'll see what they learn

@scottprugh Choice is important, but any good engineering team evaluates their tools. If the common/standard tooling is better than what they can build, they will use it. Common tools teams need to think like product engineering

@ScottPrugh What's interesting is whether we should stop this by mandate, or by encouraging teams to use good judgment and making them aware of why it might be a good idea to share tools and practices

@ChivasNambiar exactly - see my reply to @scottprugh

@paula_thrasher - sounds like an awesome retrospective opportunity as well

@schwartz_cio @ChivasNambiar The education piece is important so that teams understand the required outcomes: compliance, maintainability, etc. Involve teams in discussion of to develop shared understanding. Get them to help with the final "mandate"

@ScottPrugh yes yes yes. instead teams often view compliance, etc as an impediment

@paula_thrasher let's talk offline - We are 6 months ahead of you on rotations :)

@paula_thrasher also this: https://youtu.be/pos...

@scottnasello will have to hear more about your experiment. So far, the teams on rotation seem to be enjoying it and the teams that got our developers don't want them to leave (probably because they automated stuff when they got there!)