TestChat

A2. testability/automatability

testability, poor testability. If it's difficult to test manually, it's going to be harder to automate it. Testability is a big deal.

Test Data - creation and teardown of meaningful data

Legacy code & timelines

@heather_reiduff where automatability is affected by testability issues

Change

Convincing everyone that you need to set up your test data and then tear it down. No you shouldn't point these at existing QA data.

time to production is reducing, which I think is awesome! However, it means a change to automation approach/sategy, some haven't caught up with this yet.

Trying to stay ahead of changing tech/lang.

Maintenance. Once you have some tests automated and the product evolves. Lot of time needed for adopting existing test suite(s).

@Bill_Matthews in what sense? The need to update the tests? Or new systems? #testchat

convincing people not to automate things that shouldn't be automated

Funnily enough - other people. We seem to agree on some design, then the code each of us writes turns out to be completely different than what we thought we agreed upon.

@FriendlyTester there's often a "translation" problem going from manual tests by a subject matter expert to an automated test likely built by someone else. Assumptions abound.

@vexrm I'm not sure I agree on tearing it down. Common mistake is to always have automation against 'clean' data. Production isn't clean

Consensus on which Framework to use, but usually at the very initial phase of automation

Changes to the app under test generally - particularly in iterative projects where previous features are enhanced

Finding people who are good at automation

Thread.sleep overusage in scripts

prioritization of tasks/tests (to implement/to execute/...) is also challenging here

@FriendlyTester Do you put together suites of tests to combat this? Or use a separate data set up to run certain tests against? #testchat

trying estimate the value that a particular automated test will bring in the future.

Stability of tests and test rigs. Also, once you are so dependant on it, the loss of a test rig is crippling. Can't run 500+ tests locally...!

state management. Controlling the state and the environments to improve the 'determinsticness' of our checks

maturity of frameworks for automation generally lags way behind development frameworks - app dev is rapid, test dev less so

Right now, with end-to-end tests, managing test data across multiple microservices. It's a great pattern for isolating independent functions, but it is turning out to be something of a headache for integration and system testing.

curve ball. In some places, I've visited recently. I would say product knowledge is a struggle for them, as they are buried in automation world, they don't know the product anymore!

@FriendlyTester Ah. I don't think you need to clean everything up, but some clean up should happen. I don't need a thousand "dkldasl"s in the database. I also don't need an empty database.

@tbaxter78 We've found a way to deal with this but it was expensive and not a great long term solution. We've built an API that allows us to mange our data in our databases.

A2) SDETs/Automators with great tool skills, but poor testing skills.

Oh, and bloody Captcha! ;-)

dealing with all the various dependencies of modern websites and trying to fit all the pieces together in to a maintainable script. Too often I get 80% of the way there then give up due to a tough roadblock.

Stakeholders thinking automation is the silver bullet

A2) In recent context for me, tool majority, specifically mobile. it's getting stronger now, but 1-4 years ago, it was weak.

@Bill_Matthews - I don't think it has to be that way. Depends on the tech, no?

@FriendlyTester this type of context switching is difficult when the pressure is on to deliver

@FriendlyTester we should not be surprised by this though! It took WebDriver nearly 10 years, and required browsers to be stable. Mobile OS's are still new, it will take time.

Choosing the correct test automation tools

@FriendlyTester but again, SDETs lacking testing mindset, expect the world immediately, and can't adapt, therefore code around and create very flaky solutions.

I would say my main concern or struggle though. Is people focusing on tools and not trying to solve the testing problem(s). Problem(s) first, tools second. Hammer/Nail analogy.

proper test case design

100% pass rate on the first run. We have implemented multiple retries to help drive confidence but I know there has got to be a better way.

@FriendlyTester totally agree with your struggle. Especially if there is manual vs Automation testing to be done-- switching between the 2 can prove challenging

automation frees us from mundane tasks so that we can focus on more important "stuff"

we ship faster, and worry less. That's our main goal. all the rest is bonus.

faster feedback loop, regression is covered-more focus on newer functionality

some people say more time for 'testing'. I like to think of it as helping us solve the testing problem quicker and more efficiently. Allowing the team to move onto new items.

Cutting down regression time from about a week to a few hours. Makes the job more interesting!

A degree of confidence, quick feedback

automation speeds up the feedback loop

VISIBILITY

@FriendlyTester which in turns means we can more improvements/features to our customers quicker, which should help the business

closing feedback loops, faster time to prod, becoming more responsive to bugs

@FriendlyTester I tend to think of it as "more time for exploratory testing" which is where you tend to find the 'far out' bugs you may otherwise miss.

@ir_emery This is a big one for a lot of people

confidence in the code base prior to a release. Our tests cover our normal regression paths. #testchat

adding extra steps to block potentially bad releases

What value does the team get? I can take a vacation day and the team has an Auto-Dave they use.
Mostly joking, but it gives them a confidence the basics are solid. Someone needs to look beyond that, but it's a safety net.

Generally faster and more comprehensive feedback

ability for the devs to push changes faster with the comfort of a safety net

@Bill_Matthews Comprehensive?

one step closer to Continuous Integration. That holy grail...!

Extends the scope of testing - one project I worked on had a rules engine with 1.3M unique paths. we built a tool that tested these in about 6 mins

fast feedback, constant information stream, that skilled humans can use to make judgements on the product and quality.

@melthetester Something to shout about! lol #testchat

Clear and quick visibility of issues

With automation, it's easier to communicate requirements and problems between product, test, and development teams. You can see exactly what was supposed to happen as well as exactly what DID happen.

Absolutely! It's the testing equivalent of "Show Me The Money!"

more coverage in a shorter time frame, ability to test multiple inputs that would be tedious and mundane to execute manually

fast feedback

Automation test failures give us a targeted area to explore/test. #testchat

closer collaboration between developers and testers

@tbaxter78 In some aspects - yes. When I compare all of the data instead of a select samples, it's more comprehensive. In some aspects - less so.

@ButchMayhew indeed! "A failed check is an invitation to explore"

Humans are more prone to get bored and make mistakes/miss stuff when performing long, tedious tasks such as regression repeatedly. An automated test doesn't get bored, start thinking of football etc. Less fallable.

@ir_emery Definitely. I used to help run a regression test that took a week to perform with 4 testers, not ideal at all and things do get missed. #testchat

as a tester, automating something makes me ask more questions which gives me more business and technical knowledge

also human testers can make assumptions that may or may not be correct or written down/communicated anywhere. An automated test does not permit this.