Have you ever run automated UI testing for a feature-rich web application? If yes, how did you approach the matter of choosing a framework — or a platform — for UI testing automation? Actually, did you automate the testing at all?
If you’ve ever built consumer-facing software with a rich UI, you know this simple truth: clients don’t care about QA automation tools or tactics. That is, they don’t care as long as you don’t leak bugs into production, or as long as your QA operations stay within the budget.
For you, however, things are different. You’ll want to get the best ROI out of whatever resources you have. As a result, there’s always a whole lot of things to consider, and an overwhelming number of QA automation tools to choose from. To provide you with a good starting point, let’s consider the essential options that you get with UI testing automation.
Manual Testing
Okay, this one’s pretty obvious, but it deserves a brief mention. Having a real person eyeball the application seems like the most straightforward way to do visual testing. This makes a case for manual testing because popular frameworks like Selenium ignore visual bugs (like messed up layouts or overlapping texts). And this is why so many teams fall back on manual testing whenever issues of this sort occur.
Manual testing is also inexpensive and easy to set up as long as your project is small. On the flip side, manual testing is as reliable as your testers, and you can only scale the process by hiring new people. So once your project gets bigger, you’ll have no choice but to automate.
Code-based Test Automation Frameworks
When it comes to choosing a testing framework for browser or mobile testing, Selenium and Appium seem like unrivaled champions. These two have been ruling the world of QA automation for years, and they’ve become must-have tools for most QA engineers. Other mainstream options include Karma, Casper.js, and Cucumber with Capybara.
Сode-based frameworks have proven their worth, too. Hand-coding your tests allows for great flexibility, and it always makes sense to push more tests down the pyramid. On the flip side, code-based testing frameworks share quite a few weak points:
- Getting everything set up and ready takes forever. Integrating a testing framework into production and configuring the setup can last for days.
- WebDriver versions tend to get out of sync with auto-updating browser.
- Internet Explorer. Working with IE sucks so hard I’m not even going to dwell on this one. (Luckily, IE is less of an issue nowadays.)
- Headless browsers like PhantomJS and SlimerJS have their own pitfalls. A test can pass in a headless browser but fail in a real one, which results in leaked bugs.
- Things like the need to stop hanging browsers, monitor the host environment, and stage parallel executions… All in all, browser control can be 50 shades of headache.
- Selectors are only robust when based on IDs, but most elements won’t have them.
- The larger your test suite is, the longer it takes to run. Sure, setting up grids is an option, but the process is messy.
- As your test suite grows, it will also become less manageable. Maintenance is always a pain, even though keeping your test suite modular does help a lot.
- Using WebDriver for front-end frameworks like ReactJS has its own, erm, peculiarities (e.g. sendKeys(text) instead of setAttribute(value, text)).
- Screenshot comparison with things like WebDriverCSS or PhantomCSS is basic at best. Again, this is why teams often fall back on manual testing when it comes to UI tests.
Sure, each of the points above can be dealt with, provided that you have the right people onboard. After all, nothing is perfect, and automated testing is no exception to this rule. Still, automating with a code-based framework leaves a lot to be desired, which is why it’s always wise to consider other options.
Automating Tests on the Cloud
Cloud-based platforms like SauceLabs or Xamarin are one step closer to a more streamlined test automation experience. The great thing about tools of this sort is that they provide a ready-made, scalable infrastructure. If you haven’t used a cloud testing platform before, here are the three major options:
- SauceLabs. One of the pioneering companies that enabled tech teams to run Selenium tests on the cloud. The company provides some added value in the form of UI screenshots and videos of user actions. What’s more, the platform offers an infrastructure for parallel test execution.
- Browserstack provides its users with remote virtual machines for testing in every major browser-OS combo, on desktop and mobile. The company also supports Selenium, which is always nice.
- Xamarin Test Cloud is more of a walled garden with it’s proprietary scripting language Calabash, but it offers a great solution for mobile testing. It supports 2,000 devices, including smartphones and tablets.
By taking care of the infrastructure-related matters, cloud platforms save you time. You still have to deal with code, though, which means your productivity depends on your QAs’ programming skills. True, Xamarin Test Cloud has a test recorder, but it spews out auto-generated code which isn’t that easy to handle.
Not having to write your tests can save you more time (and money), so let’s take a glance at the testing tools that don’t require programming.
New-Generation Cloud-Based Platforms
In some sense, the idea behind code less test recording tools isn’t that new. If you’ve ever worked with record-playback tools in Ranorex or UFT (or the aforementioned Xamarin Test Cloud), you know what I’m talking about.
Today, there’s a whole lot of new-generation web platforms that build upon the same principle yet offer more advanced features for test maintenance. In this post, we’ll look at three platforms of this type.
Applitools
Applitools is a SaaS platform that promises to “turn manual testing into a repeatable, automated process”. The platform is code-based, it offers SDKs for major testing frameworks, and expands their functionality with UI comparison.
The great thing about this tool is that it’s screenshot comparison is one step ahead of the solutions offered by WebDriverCSS or PhantomCSS. As for the not-so-great thing, we’re still talking about basic image comparison: Applitools’ algorithms don’t recognize on-page objects.
Usetrace
When compared to Applitools, Usetrace seems to go further with the automation of manual functional testing. Based on the tools marketing materials, it seems like the platform is targeted at non-technical users.
With Usetrace, test creation is based on the recording of the actual user actions, and UI comparison includes object recognition. The platform enables you to manually verify n-page UI elements, and it’s able to scan the text content of a DOM element. It also has neat dashboards and collaboration features.
As for the downsides of Usetrace, you still need to understand how locators work (which is weird given that the platform caters to non-techies). Besides, the tool doesn’t seem to offer much in the way of CSS testing.
Screenster
Screenster is another neat cloud-based platform that promises to make UI testing simple enough for a non-programmer. It automates tests based on the same-old record-playback paradigm, yet it throws in quite a few advanced features into the mix.
The platform uses the concept of visual baselines that include screenshot and DOM comparison. It automatically handles selectors and timeouts, and it provides an easy way of working with dynamic UI religious. Just like Usetrace, Screenster has intuitive dashboards and supports collaboration.
The main downside of the platforms is that it’s new, so it still needs to amass a following of users. It doesn’t yet support all major browsers, with only Chrome and Firefox listed as supported ones.
Conclusion
Turns out the market of QA automation solutions is a lot more diverse that it may seem at first sight. The number of tools and platforms proves there’s a lot more to testing automation that the plain-old Selenium. Given this, developing an effective QA automation strategy today is more about combining the right tools that are best suited for particular tasks.
Specifically, code-based frameworks may be great for a wide spectrum of jobs, but they fail to deliver when it comes to visual testing of UI. Chances are using a cloud-based platform like Screenster or Usetrace for this sort of tasks might provide QA teams with the missing piece of the puzzle.
Author
Nikita Sagan Screenster Evangelist
