At spriteCloud, we have our own test lab consisting of a broad range of mobile devices to test on. As you can understand, in the current market of mobile phones, it can be difficult to keep a test lab up to date; new phones are released all year round. Updates to operating systems are pretty common. So we always look for tools which can help us testing on as many devices and OS versions as possible. There are quite some solutions to subscribe to, which offer a test lab with physical devices in the cloud. But why do we still maintain our own test lab? What makes it so necessary to test on a physical device which you are holding in your hand, instead of a physical device in the cloud? The last couple of weeks, I have been testing three of those cloud based solutions; browserstack, testingbot and crossbrowsertesting. I’m not going into details on these services. I just want to share one observation and elaborate a bit on that.
Currently, I am testing an ecommerce website for a client. For ecommerce websites, mobile is very important. A lot of traffic comes from mobile devices and tablets. A good responsive website is a must-have for every company active in ecommerce. So for testing, I have a pile of tablets and phones on my desk. Next to that, I’m using browserstack’s physical devices every now and then for debugging, or just because I need one specific device. Today, I discovered a bug on a scrollable menu; in Chrome on Android the scrollable menu … well …, was not scrollable. That’s quite inconvenient for the customer, since all clues indicate the menu is scrollable. Our frontend developer asked me to use browserstack to debug the issue. So we started browserstack, opened the console to check for errors, loaded the page, fastened our seatbelts, and tried to scroll. Tremendous was our surprise when we were able to scroll!
That got me thinking and made me smack my head a bit. Our first guess was that we couldn’t scroll, due to the touch input not being recognized correctly. At first I wanted to know how browserstack connects to their physical devices, since scrolling the menu on a desktop browser worked. My magic google skills did not give me an answer to that question, but I realised that the question was not relevant. No matter how my input is translated to the physical device in their test lab, it is not done by touching the screen with a finger (at least, I hope there are no testing minions in their test lab). This made me come to the conclusion that we always will need to test interactions on our physical devices instead of devices in the cloud. Probably, this is kicking in an open door, but I just forgot about it, since testing with browserstack went pretty well.
Does this mean tools like browserstack have a very limited value? Of course not. It is especially useful for testing different responsive views on a lot of different devices; it even has a special feature for that. But when you test interactions, make sure you always also test on a physical device in your hand. And let’s be honest; it is also pretty cool to be one of the first people to try out new devices and new versions of a mobile OS.