TestChat

Device Testing
A testing chat on Device Testing
   2 years ago
#TestChatAIA testing chat on AI
   2 years ago
#TestChatAccessibility TestingA testing chat on Accessibility Testing
Ministry of Testing
We're about to finish, thank you all so much for joining in tonight! Rockstars. You can keep the chat going, or head over to The Club for more - we've gathered some of the resources mentioned - drop in anything you think could be useful https://club.ministr... #testchat
Device Testing TestChat
Tonights TestChat about Device Testing, our second of 2018 got us all talking about what our test labs look like and how we decide what devices to test on. A brief summary of the answers is below. Q1. How do you decide what devices to test on? How...
Zena
it was nice to know other people working in same domain, and so much new knowledge I gained :) Thanks!!
Ministry of Testing
Right, let's get stuck in. Q1 How do you decide what devices to test on and how do you prioritise those devices? #testchat
Deborah Lee
Try and get stats to give an idea on what the website/app users are using
Deborah Lee
If no stats available, find general stats for popular models/OS/browser combos
Ministry of Testing
For example, back in the day, I had to focus on Blackberry testing as it was the only secure device at the time. Then of course the model was dependent on user base
Niranjani
We use info from analytics to see which of the devices are the most commonly used and ensure they are part of our device matrix.
Heather Reid
when I 1st did this I picked most up to date iOS device, older one (about 2 versions) & same for android until I had stats of usage & realised we had a lot of IE10 users!
Deborah Lee
You can't cover everything but aim to cover popular devices/OS/browser combos. If users report issues, ask them what environment they are on and try and reproduce with real or virtual device(s)
Simon Gilmurray
Yep, we also use Analytics to inform which devices are most popular out there amongst our app users
Niranjani
For android it is more challenging as there are n number of devices across the globe!
Deborah Lee
It also depends in which country/countries the app is used i.e. popular phone models vary according to the country
Uros Stanisic
It depends on target systems and how much certain devices are represented on those systems. We're trying to make an agreement together with the customer about that
Deborah Lee
If you are struggling to support an old version, state clearly that you only support X and above. e.g. only IE11!
Ministry of Testing
@RanjaniRambles is this historical analytics from your own app, or general data from your target audience? #testchat
Deborah Lee
some browsers are more hassle than they are worth e.g. IE10 or Opera!
Tester Doug
This is where metrics come in VERY handy. If you don't have them: get them! No need wasting time on devices your users don't use.

It's equally important to know if you have key users and find out what devices they use. That can out weigh your metrics!
Niranjani
from our own app
Ministry of Testing
@DeborahLee89 this is it. It's best to be explicit on what is or isn't supported #testchat
Ministry of Testing
@uros_stanisic By making an agreement with your users, do you mean that you're defining what you will or won't support? Or are you taking guidance from them? #testchat
Tester Doug
@DeborahLee89 Are we really still in that world though? There was definitely the case with IE6 & IE7 but isn't it less so now wiht "modern" browsers?
Heather Reid
we had a policy of the current version + previous 2 versions to support but if your users are e.g. government of huge pharma, they are easily running much older versions of IE
Deborah Lee
some businesses are stuck on an old version of IE or IE is the only way to access intranet so it MUST work
Heather Reid
@qatesterdoug oh you're one of the fortunate ones :-P
Deborah Lee
@heather_reiduff we have this policy in my current workplace. latest -2. it makes our lives much easier!
Tester Doug
Fragmentation is still an issue, so simplifying this to a "device" without considering OS version is shortsighted. As we've found out the hard way!
Deborah Lee
developers lives easier too!
Deborah Lee
@qatesterdoug and browser too! so many options for combinations
Heather Reid
@DeborahLee89 It's still a massive matrix to cover but it does make it somewhat easier
Uros Stanisic
it is a mix of both. We're trying to be on the same page together with customers. And we're also providing guidance to them as well ;)
Ian Emery
We use a combination of google analytics and our own internal MI to update a list of supported devices/browsers. Tend to concentrate on the software aspect rather than devices, but obviously keep mental notes on most popular devices of the time.
Ian Emery
In terms of supported devices, anything that carries less than 0.5% of our traffic is not supported (that includes Windows Phones these days). Annoyingly IE11 is still going strong.
Ministry of Testing
@ir_emery that's a good way of trimming it #testchat
Ian Emery
we've had plenty of, shall we say, energetic discussions about supporting older browsers. I am always on the ruthless side.
Ministry of Testing
Q3 What does your device testing lab look like? Do you use physical devices? Or Emulators, cloud, simulators? #testchat
Deborah Lee
I only really trust real devices as that's what the users have. But it is expensive.
Niranjani
For automation, I use Firebase labs - mostly emulators coz physical devices are expensive.
Deborah Lee
Chrome dev tools have a responsive mode for different devices which can be useful
Heather Reid
I had a handful of real devices that I borrowed from anyone and everyone getting rid of their old phones. We had a good variety of phones & laptops in the office too. The rest was browserstack & Chrome dev tools
Zena
Me and my team are provided with current new devices, so we work in Live scenario with all kinds of OS inlcuding android/ios/bb/feature phones/Windows.
Simon Gilmurray
Mix, mainly physical as it feels to replicate reality better, but Simulator in iOS and Genymotion. Used to use Xamarin Test Cloud which was a really nice way to submit automated tests up and run them simultaneously against many different devices and OS's.
Tester Doug
@DeborahLee89 This only changes the screen size though, so buyer beware. We recently had an issue where we verified an issue in chrome dev with "ipad" instead of the actual device (or emulator). It did not end well :)
Deborah Lee
you can link up physical devices to the computer to access dev tools etc if you're testing in a browser which is cool and useful
Zena
I have also worked in cloud based system like #Seetest
Wayne Ellis
Broswerstack, and a cupboard with some more popular real devices in.
Niranjani
I use a combination of phone emulators & tablets
Tester Doug
Is Xamarin Test Cloud actual real physical devices that you can access from their lab? I like that.
Uros Stanisic
mostly emulators and physical devices. We're also using some crowd testing approach between testers from different projects in order to cover more ground and more real devices
Zena
@uros_stanisic I have done that, they used to borrow my Nexus 5 when it was new and just launched.
Niranjani
@qatesterdoug The only caveat is Xamarin test labs were expensive than Firebase.
Ministry of Testing
@uros_stanisic That's a really good idea, share the devices around #testchat
Niranjani
I have also used AWS Device Farm in the past but ran into a lot of issues with the test failing on their device farm due to popups/device updates,etc
Tester Doug
@RanjaniRambles I have worries about all things google ;)
Niranjani
Firebase supports effective sharding - so when you kick off your tests, you can define in the config as to how many tests you want to run in parallel and the time taken is way less than AWS device farm/Xamarin Test Cloud
Uros Stanisic
@zenealous @ministryoftest yeah, and another benefit is that testers from different projects are eager to try something new and to help out ;)
Niranjani
@qatesterdoug Firebase does run tests in parallel effectively. For 400+ tests, AWS device farm would take 4 hours whereas Firebase labs takes 40 mins!
Deborah Lee
although be careful that a specific setup on a personal device may affect the functionality/compatibility performance!
Simon Gilmurray
What languages do you use to run your tests on Firebase? We have it and use it for A or B Testing and Analytics.
Deborah Lee
it can be really hard to pinpoint the cause or scope of an issue - model? model/OS combo? browser? Settings? combo of all??
Ministry of Testing
@DeborahLee89 I suppose that's where it's sometimes good to have more devices - is it happening on all or only some..good way to narrow the field.
Niranjani
I use Espresso framework with Java to run tests on Firebase
Ian Emery
We're about to pilot the use of Browserstack. What we've seen so far is very impressive. I have dedicated devices - iPhone, iPad, S8 and Galaxy tab. We also have a trolley full of older devices.
Ian Emery
We use Chrome Dev emulation a lot but would never sign off purely based on it. See too many bugs on real devices that don't replicate in emulation to trust it entirely.
Ian Emery
I've also got my first smartphone - a HTC Hero (remember them?!) from about 10 years ago that still works. Don't think it counts as a valid test though. More for nostalgia purposes really.
Ministry of Testing
@ir_emery some days we need that, so it still counts!