TestChat

Automation Testing
A testing chat on Automation, specifically the applications for using it in testing.
   3 years ago
#testchatMisunderstanding of TestingThe Misunderstanding of Testers and Testing. And What Can We Do About It?
   2 years ago
#TestChatExploratory TestingA testing chat on Exploratory Testing
Ministry of Testing
Well that was an excellent #TestChat, don't forget you can continue the discussion over on The Club, get involved https://club.ministr...
Richard Bradshaw
we have/want more discussions like this all the time over at The Club, so do check it out. Free to create an account.
Marllos Prado
Thanks for organizing it!
Toyer M
thank you, this has been quite insightful. Loved it! #testChat
Ministry of Testing
Q2. What are your biggest struggles in automation? #testchat
Heather Reid
A2. testability/automatability
Richard Bradshaw
testability, poor testability. If it's difficult to test manually, it's going to be harder to automate it. Testability is a big deal.
Tracey Baxter
Test Data - creation and teardown of meaningful data
Amit Wertheimer
Legacy code & timelines
Heather Reid
@heather_reiduff where automatability is affected by testability issues
David Lowe
Convincing everyone that you need to set up your test data and then tear it down. No you shouldn't point these at existing QA data.
Richard Bradshaw
time to production is reducing, which I think is awesome! However, it means a change to automation approach/sategy, some haven't caught up with this yet.
MelTheTester
Trying to stay ahead of changing tech/lang.
Peter M. Kruse
Maintenance. Once you have some tests automated and the product evolves. Lot of time needed for adopting existing test suite(s).
Ministry of Testing
@Bill_Matthews in what sense? The need to update the tests? Or new systems? #testchat
Heather Reid
convincing people not to automate things that shouldn't be automated
Amit Wertheimer
Funnily enough - other people. We seem to agree on some design, then the code each of us writes turns out to be completely different than what we thought we agreed upon.
gyro_gearloose
@FriendlyTester there's often a "translation" problem going from manual tests by a subject matter expert to an automated test likely built by someone else. Assumptions abound.
Richard Bradshaw
@vexrm I'm not sure I agree on tearing it down. Common mistake is to always have automation against 'clean' data. Production isn't clean
Stephanie Frias
Consensus on which Framework to use, but usually at the very initial phase of automation
Bill Matthews
Changes to the app under test generally - particularly in iterative projects where previous features are enhanced
Bill Matthews
Finding people who are good at automation
Heather Reid
Thread.sleep overusage in scripts
Peter M. Kruse
prioritization of tasks/tests (to implement/to execute/...) is also challenging here
Ministry of Testing
@FriendlyTester Do you put together suites of tests to combat this? Or use a separate data set up to run certain tests against? #testchat
Butch Mayhew
trying estimate the value that a particular automated test will bring in the future.
Ian Emery
Stability of tests and test rigs. Also, once you are so dependant on it, the loss of a test rig is crippling. Can't run 500+ tests locally...!
Richard Bradshaw
state management. Controlling the state and the environments to improve the 'determinsticness' of our checks
Bill Matthews
maturity of frameworks for automation generally lags way behind development frameworks - app dev is rapid, test dev less so
Jeri Levine
Right now, with end-to-end tests, managing test data across multiple microservices. It's a great pattern for isolating independent functions, but it is turning out to be something of a headache for integration and system testing.
Richard Bradshaw
curve ball. In some places, I've visited recently. I would say product knowledge is a struggle for them, as they are buried in automation world, they don't know the product anymore!
David Lowe
@FriendlyTester Ah. I don't think you need to clean everything up, but some clean up should happen. I don't need a thousand "dkldasl"s in the database. I also don't need an empty database.
Butch Mayhew
@tbaxter78 We've found a way to deal with this but it was expensive and not a great long term solution. We've built an API that allows us to mange our data in our databases.
Richard Bradshaw
A2) SDETs/Automators with great tool skills, but poor testing skills.
Ian Emery
Oh, and bloody Captcha! ;-)
Kevin Lamping
dealing with all the various dependencies of modern websites and trying to fit all the pieces together in to a maintainable script. Too often I get 80% of the way there then give up due to a tough roadblock.
Bill Matthews
Stakeholders thinking automation is the silver bullet
Richard Bradshaw
A2) In recent context for me, tool majority, specifically mobile. it's getting stronger now, but 1-4 years ago, it was weak.
MelTheTester
@Bill_Matthews - I don't think it has to be that way. Depends on the tech, no?
Toyer M
@FriendlyTester this type of context switching is difficult when the pressure is on to deliver
Richard Bradshaw
@FriendlyTester we should not be surprised by this though! It took WebDriver nearly 10 years, and required browsers to be stable. Mobile OS's are still new, it will take time.
Mark White
Choosing the correct test automation tools
Richard Bradshaw
@FriendlyTester but again, SDETs lacking testing mindset, expect the world immediately, and can't adapt, therefore code around and create very flaky solutions.
Richard Bradshaw
I would say my main concern or struggle though. Is people focusing on tools and not trying to solve the testing problem(s). Problem(s) first, tools second. Hammer/Nail analogy.
gyro_gearloose
proper test case design
Butch Mayhew
100% pass rate on the first run. We have implemented multiple retries to help drive confidence but I know there has got to be a better way.
Toyer M
@FriendlyTester totally agree with your struggle. Especially if there is manual vs Automation testing to be done-- switching between the 2 can prove challenging
Ministry of Testing
Q4. Clearly, there are lots of uses for automation. What value does your team get from automation? #testchat
gyro_gearloose
automation frees us from mundane tasks so that we can focus on more important "stuff"
Amit Wertheimer
we ship faster, and worry less. That's our main goal. all the rest is bonus.
Toyer M
faster feedback loop, regression is covered-more focus on newer functionality
Richard Bradshaw
some people say more time for 'testing'. I like to think of it as helping us solve the testing problem quicker and more efficiently. Allowing the team to move onto new items.
Ian Emery
Cutting down regression time from about a week to a few hours. Makes the job more interesting!
Tracey Baxter
A degree of confidence, quick feedback
gyro_gearloose
automation speeds up the feedback loop
Richard Bradshaw
@FriendlyTester which in turns means we can more improvements/features to our customers quicker, which should help the business
David Corrales
closing feedback loops, faster time to prod, becoming more responsive to bugs
Ian Emery
@FriendlyTester I tend to think of it as "more time for exploratory testing" which is where you tend to find the 'far out' bugs you may otherwise miss.
Ministry of Testing
@ir_emery This is a big one for a lot of people
Butch Mayhew
confidence in the code base prior to a release. Our tests cover our normal regression paths. #testchat
Steven Parker
adding extra steps to block potentially bad releases
David Lowe
What value does the team get? I can take a vacation day and the team has an Auto-Dave they use.
Mostly joking, but it gives them a confidence the basics are solid. Someone needs to look beyond that, but it's a safety net.
Bill Matthews
Generally faster and more comprehensive feedback
Toyer M
ability for the devs to push changes faster with the comfort of a safety net
Ian Emery
one step closer to Continuous Integration. That holy grail...!
Bill Matthews
Extends the scope of testing - one project I worked on had a rules engine with 1.3M unique paths. we built a tool that tested these in about 6 mins
Richard Bradshaw
fast feedback, constant information stream, that skilled humans can use to make judgements on the product and quality.
Mark White
Clear and quick visibility of issues
Jeri Levine
With automation, it's easier to communicate requirements and problems between product, test, and development teams. You can see exactly what was supposed to happen as well as exactly what DID happen.
MelTheTester
Absolutely! It's the testing equivalent of "Show Me The Money!"
Toyer M
more coverage in a shorter time frame, ability to test multiple inputs that would be tedious and mundane to execute manually
Butch Mayhew
Automation test failures give us a targeted area to explore/test. #testchat
gyro_gearloose
closer collaboration between developers and testers
Amit Wertheimer
@tbaxter78 In some aspects - yes. When I compare all of the data instead of a select samples, it's more comprehensive. In some aspects - less so.
Richard Bradshaw
@ButchMayhew indeed! "A failed check is an invitation to explore"
Ian Emery
Humans are more prone to get bored and make mistakes/miss stuff when performing long, tedious tasks such as regression repeatedly. An automated test doesn't get bored, start thinking of football etc. Less fallable.
Ministry of Testing
@ir_emery Definitely. I used to help run a regression test that took a week to perform with 4 testers, not ideal at all and things do get missed. #testchat
Toyer M
as a tester, automating something makes me ask more questions which gives me more business and technical knowledge
Jeri Levine
also human testers can make assumptions that may or may not be correct or written down/communicated anywhere. An automated test does not permit this.