#ChaosEngineering - CrowdChat

Chaos Engineering

This Chat creates future perspective on Chaos Engineering

#chaosengineering @devopssc @canada_devops @bajpaigarima1 @rjsriram15 @sureshgp @fictionalzeus @wszepesi @tinycyber

LeaderBoard

Stream Ended

Establishing a secure connection

Suresh GP, MBRM13

There are also a lot of misconceptions around Chaos Engineering that we need to address. 1.It is not about breaking things in Production 2. It is not about injecting random chaos experiments into the system and seeing what happens

[+] Show Hidden Comments

could you refer to some platforms for reference for our audience ?

Suresh GP, MBRM

Gremlin is a good platform to experiment in a controlled environment where you minimize the blast radius

verica also offers some good options for full spectrum Chaos testing

Suresh GP, MBRM

Another useful resource is https://chaostoolkit.org/

it is about breaking things at all levels. Just not breaking them in your customer facing application. We create env to allow test and this is the same application

Suresh GP, MBRM

Are organizations even ready from a risk appetite and maturity level to do Chaos Engineering? Considering DevOps 3rd way is followed by very few organizations across the globe?

always say, if you are still getting gains from the basics, you dont need to go advanced. Chaos is best in complex distributed systems where basic testing cant find the pain points

@TinyCyber could you help understand some features which stand out ?

Suresh GP, MBRM

@TinyCyber Spot on Mark. Lets not boil the ocean and do things for the sake of doing Chaos Engineering

latency, app calls, sysstem load, infastructure inefficiency, recovery practices, load balancing

What do you think are the top challenges & Opportunities - Chaos Engineering

[+] Show Hidden Comments

top challenge is translating results to new features that solve the problem. Complex issues have multiple strings

Systematic thinking around Chaos Engineering - more about building a practice less about tool

Suresh GP, MBRM

@bajpaigarima Challenges 1 Creating Social Awarness is key. Way of Doing Game Days will help. 2. Chaos Engineering is a Journey and not Destination 3. Start Small and see progress instead of Big Bang Approach 4.Get your entire value chain orchestrated well before you experiment

Suresh GP, MBRM

Opportunities : 1. Ability to move to Predictive Management. 2. Improving your longer version of the game. 3. Inculcate the Experimental and Growth Mindset

Creating agnostic view on Chaos practice

William Szepesi

Creating visibility can be a challenge. How are individual failures or performance issues made visible and secondly, how these issues have a knock on effect on the distributed system

cant practice without the right tools

@TinyCyber Can you talk about "right tools" ?

whatever gets you were you need to go smaller testing and localized chaos can use the hammer and nails, the bigger the network, the more you want automated aolutions and integrated results

if your chaos tool outputs logs and tracea, still a lot of work converting to usable framework. DAST tools can be low level chaos

If breaks are only visible with the chaos tool and you cant duplicate locally can be difficult to fix

What is key business value of Chaos Engineering, how organization benefit?

[+] Show Hidden Comments

Focus on the third way – DevOps, enable experiential learning

Suresh GP, MBRM

Prevents Long Outage from happening that invariably reduces business impact

User Experience at Scale. ..

Suresh GP, MBRM

Customers gain confidence that the team is fully equiped for any disasters and ensure business continuity.

Could it help in feature engineering ?

Suresh GP, MBRM

I look at Feature Engineering addressing aspects of Usability, Convenience and Reliability. So from that perspective it helps.

value is finding challenges and errors before the customer

William Szepesi

identifying failures before they become outages - as Gremlins whitepaper is titled - is a key business value, protecting clients from service failures and maintaining client trust in your service reliability - a key business value!

chaos can also verify the right fliw and feedback exist in your distributed practices

Suresh GP, MBRM

Can you share the whitepaper link @WSzepesi for our viewers?

Suresh GP, MBRM7

The purpose of Chaos Engineering is NOT to “Break Things on Purpose”.
If anything we are trying to “Fix them on Purpose”!

[+] Show Hidden Comments

Well said - how do you think we move from purpose of breaking things to creating resiliency

Suresh GP, MBRM

@bajpaigarima1 Understanding the Goal is important. What are we committing to deliver for a seamless customer experience.? How is our ecosystem tailored to meet the fit for purpose and fit for use objective.

the purpose is to break things but in a non-permenent and non-value losing way

with materials engineering we do nondestructive testing. Chaos offees ua that option for code

@TinyCyber Does it help prioritize the technical debt

could help identify but should realy find tech debt during dev and planning. Chaos tests unexpected gaps

@bajpaigarima1 Can help prioritize features. Initial chaos eng showed where things broke. Modern testing practices with chaos are more about when it will break

Suresh GP, MBRM

I agree with Mark. It is not a substitute for prioritizing technical debt. It has to be done during dev and planning

sometimes new features might fix chaos issues without being tech debt. If a feature plans to decrease latency both aspects might get to the same point

William Szepesi4

Can security be controlled when Chaos Engineering is being practiced?

[+] Show Hidden Comments

Suresh GP, MBRM

Tricky question. With DevSecOps being integrated as part of Pipeline, you can make it work. However needs some planning, coordination and VSM to make it work

may be part of the chaos experiments - how secure is your chaos experiments

William Szepesi

Yes and you want to automate the testing of it in a continuous deployment pipeline so you maintain that competence.

Suresh GP, MBRM

https://www.crowdchat.net/s/867zg - Strongly recommend this book from Aaron Rinehart!

I see chaos as a differwnt aspect of security. Not system security as much as resilence and recovery. Dont need it for vulnerabilities and network monitoring

Suresh GP, MBRM3

in my opinion Chaos Engineering can be summarized as follows - ●Its about Systems Safety
●Building safety margins into systems
●Replace blame culture with learning culture
●Telemetry, experimentation, and instrumentation

[+] Show Hidden Comments

none of those are limited to chaos. Those are good central practices that chaos can help with

bingo!! Convergence of SRE and Chaos outcomes

@bajpaigarima1 always looking to expand that toolbox

Suresh GP, MBRM

I always think that #SRE and #Chaos go very well together in achieving reliability and best user experience

Lets talk about tools , which tool you are looking out in the Chaos space ?

[+] Show Hidden Comments

Suresh GP, MBRM

In addition to Gremlin and Chaos Monkey (Netflix) I would recommend the following. Chaos Toolkit: This open-source initiative makes tests easier with an open API and a standard JSON format.
Pumba: Pumba is a chaos testing and network emulation tool for Docker.
L

Suresh GP, MBRM

Litmus: A chaos engineering tool for stateful workloads on Kubernetes.

https://www.gremlin.com/community/tutorials/chaos-...

William Szepesi

Limited set of tools available via open source and you may be re-inventing the wheel. Also, security vulnerabilities may not be well understood..

Suresh GP, MBRM

I would recommend Chaos Toolkit: This open-source initiative makes tests easier with an open API and a standard JSON format

@sureshgp havent uswd the chaos toolkit. I havw to check that,and litmus out. I like robot framework for basic testing