Search

ML | Improve tests stability with Machine learning





The problem

The current spring plans make developers work so fast, and usually, they don't have time to take care of details.

These details can include keeping sanitized web elements with tags or unique attributes that QA developers can use for automating.

If you have been working with selenium or any other web automation framework, for sure, you have found elements that changed. Some elements changed from spans to divs, or attributes altered, what is transparent for users but can cause tests to be flaky.


I'm sure you have read a lot of tutorials about automation and how important it is to keep elements for the QA untouched, like IDs, special QA custom tags, or CSS values, but the reality is so different. Usually, the developers don't mind going fast developing and rewriting the entire web component removing the old one and do not pay attention to these details, leaving the tests ready for fail.


I have spent a lot of time just rewriting my automation tests from spring to spring, just trying to keep the automation suite stable and fighting these changes. The result, as you can think, is that the automation suite is not stable, and I spend more time fixing It again than adding more tests that can increase the coverage of the app.

I was searching for a solution, a real one for a small company or a startup that can not afford a tool that costs 400$ month. The idea is to find the elements as a real user does, and do not care if they change, cause real users do not care if a web element is not a div anymore, the color of the button changed, etc.



Looking for a Machine Learning solution


After spending a lot of time checking how machine learning is improving the stability of the tests, I found different solutions that use a master key of the webpage to find where the element is now, based on where it was. Calculate multiple selectors for the same element and search for them all to check which one works best and others that use ML to find the best fit.


All these solutions needed their infrastructure and, of course, that you use their accounts and servers, which is not always a good solution for companies. I was looking for a solution that can be easily integrated with our tests and doesn't need to rewrite all of them again in a cloud platform, that usually is codeless.


The problems found were:

  • No open source

  • The codeless platform always has a pitfall or a test case that can not be implemented with the given methods.

  • Abusive price



Proposed solution


Stepping away from these platforms and taking the idea that they give, I found the solution for our case. The main idea itself was so simple, given a webElement, found the most similar element on the web page.


Base approach


  • Find a web element and save the html of it.

  • Recover the found element html

  • Find in the current page source the matching element.

  • Return the matching element or Null



Save and restore saved web element information.


This part is easy to implement in any programming language. Adding to each interaction, the logic for save the info and restore it. I wrap the methods with an alias attribute like


click(locator=locator, alias='loginUsernameInput')

Now I add a simple logic for check if the data exists recover it and if not save it to a JSON file. Find in the current page source the matching web element.

Here is where the tool shines, a way to validate if there is a similar element on the page


General notes about the main idea:


  • Should find the matching element

  • Should be unbreakable to element changes


The problems that I found in the first version was with the forms where all the inputs are similar.


The input in the creation form is different from the edited one because, in the edit form, the input has some value, and this was causing the machine learning model to find another element that was more similar to the original input.


The solution was to add the position to the find method, so now has to:

  • Find the top 5 similar elements.

  • Return the element with the position closer to the original element


This solution worked well, including language changes or edited form inputs on the webpage.



The algorithm


The solution can be found in this notebook.

Now we have the appropriate setup for make tests unbreakables




#topic #ml #selenium #machinelearning #webDriver #testing

15 views

© 2020 Qa lemming