TestChat

Try and get stats to give an idea on what the website/app users are using

If no stats available, find general stats for popular models/OS/browser combos

For example, back in the day, I had to focus on Blackberry testing as it was the only secure device at the time. Then of course the model was dependent on user base

We use info from analytics to see which of the devices are the most commonly used and ensure they are part of our device matrix.

when I 1st did this I picked most up to date iOS device, older one (about 2 versions) & same for android until I had stats of usage & realised we had a lot of IE10 users!

You can't cover everything but aim to cover popular devices/OS/browser combos. If users report issues, ask them what environment they are on and try and reproduce with real or virtual device(s)

Yep, we also use Analytics to inform which devices are most popular out there amongst our app users

For android it is more challenging as there are n number of devices across the globe!

It also depends in which country/countries the app is used i.e. popular phone models vary according to the country

It depends on target systems and how much certain devices are represented on those systems. We're trying to make an agreement together with the customer about that

If you are struggling to support an old version, state clearly that you only support X and above. e.g. only IE11!

@RanjaniRambles is this historical analytics from your own app, or general data from your target audience? #testchat

some browsers are more hassle than they are worth e.g. IE10 or Opera!

@RanjaniRambles very true #testchat

This is where metrics come in VERY handy. If you don't have them: get them! No need wasting time on devices your users don't use.

It's equally important to know if you have key users and find out what devices they use. That can out weigh your metrics!

from our own app

@DeborahLee89 this is it. It's best to be explicit on what is or isn't supported #testchat

@uros_stanisic By making an agreement with your users, do you mean that you're defining what you will or won't support? Or are you taking guidance from them? #testchat

@DeborahLee89 Are we really still in that world though? There was definitely the case with IE6 & IE7 but isn't it less so now wiht "modern" browsers?

we had a policy of the current version + previous 2 versions to support but if your users are e.g. government of huge pharma, they are easily running much older versions of IE

some businesses are stuck on an old version of IE or IE is the only way to access intranet so it MUST work

@qatesterdoug oh you're one of the fortunate ones :-P

@heather_reiduff we have this policy in my current workplace. latest -2. it makes our lives much easier!

Fragmentation is still an issue, so simplifying this to a "device" without considering OS version is shortsighted. As we've found out the hard way!

developers lives easier too!

@qatesterdoug and browser too! so many options for combinations

@DeborahLee89 It's still a massive matrix to cover but it does make it somewhat easier

it is a mix of both. We're trying to be on the same page together with customers. And we're also providing guidance to them as well ;)

We use a combination of google analytics and our own internal MI to update a list of supported devices/browsers. Tend to concentrate on the software aspect rather than devices, but obviously keep mental notes on most popular devices of the time.

In terms of supported devices, anything that carries less than 0.5% of our traffic is not supported (that includes Windows Phones these days). Annoyingly IE11 is still going strong.

@ir_emery that's a good way of trimming it #testchat

we've had plenty of, shall we say, energetic discussions about supporting older browsers. I am always on the ruthless side.

I only really trust real devices as that's what the users have. But it is expensive.

For automation, I use Firebase labs - mostly emulators coz physical devices are expensive.

Chrome dev tools have a responsive mode for different devices which can be useful

I had a handful of real devices that I borrowed from anyone and everyone getting rid of their old phones. We had a good variety of phones & laptops in the office too. The rest was browserstack & Chrome dev tools

Me and my team are provided with current new devices, so we work in Live scenario with all kinds of OS inlcuding android/ios/bb/feature phones/Windows.

Mix, mainly physical as it feels to replicate reality better, but Simulator in iOS and Genymotion. Used to use Xamarin Test Cloud which was a really nice way to submit automated tests up and run them simultaneously against many different devices and OS's.

@DeborahLee89 This only changes the screen size though, so buyer beware. We recently had an issue where we verified an issue in chrome dev with "ipad" instead of the actual device (or emulator). It did not end well :)

you can link up physical devices to the computer to access dev tools etc if you're testing in a browser which is cool and useful

I have also worked in cloud based system like #Seetest

Broswerstack, and a cupboard with some more popular real devices in.

I use a combination of phone emulators & tablets

Is Xamarin Test Cloud actual real physical devices that you can access from their lab? I like that.

mostly emulators and physical devices. We're also using some crowd testing approach between testers from different projects in order to cover more ground and more real devices

@uros_stanisic I have done that, they used to borrow my Nexus 5 when it was new and just launched.

@qatesterdoug The only caveat is Xamarin test labs were expensive than Firebase.

@uros_stanisic That's a really good idea, share the devices around #testchat

I have also used AWS Device Farm in the past but ran into a lot of issues with the test failing on their device farm due to popups/device updates,etc

@RanjaniRambles I have worries about all things google ;)

Firebase supports effective sharding - so when you kick off your tests, you can define in the config as to how many tests you want to run in parallel and the time taken is way less than AWS device farm/Xamarin Test Cloud

@zenealous @ministryoftest yeah, and another benefit is that testers from different projects are eager to try something new and to help out ;)

@qatesterdoug Firebase does run tests in parallel effectively. For 400+ tests, AWS device farm would take 4 hours whereas Firebase labs takes 40 mins!

although be careful that a specific setup on a personal device may affect the functionality/compatibility performance!

What languages do you use to run your tests on Firebase? We have it and use it for A or B Testing and Analytics.

it can be really hard to pinpoint the cause or scope of an issue - model? model/OS combo? browser? Settings? combo of all??

@DeborahLee89 I suppose that's where it's sometimes good to have more devices - is it happening on all or only some..good way to narrow the field.

I use Espresso framework with Java to run tests on Firebase

We're about to pilot the use of Browserstack. What we've seen so far is very impressive. I have dedicated devices - iPhone, iPad, S8 and Galaxy tab. We also have a trolley full of older devices.

We use Chrome Dev emulation a lot but would never sign off purely based on it. See too many bugs on real devices that don't replicate in emulation to trust it entirely.

I've also got my first smartphone - a HTC Hero (remember them?!) from about 10 years ago that still works. Don't think it counts as a valid test though. More for nostalgia purposes really.

@ir_emery some days we need that, so it still counts!