Thu, Sep 10
Timezone: Asia/Kolkata (IST)
Srinivasan Sekar / Sai Krishna - Advanced Appium
At this workshop, you'll learn about advanced concepts in Appium. We will learn how to write a single script for Android, iOS and mobile web apps. Once our tests are established, we will work on framework design and report. We will also have a detailed look at how can we automate various mobile gestures using TouchActions and mobile endpoints in Appium and running distributed tests in parallel on various emulators and simulators.
Anand Bagmar / Alexei Barantsev - Selenium Deep Dive
Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.
This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.
Manoj Kumar / Diego Molina / Marcus Merrell - Selenium Grid
Selenium Grid can be a bit daunting to get up and running. Starting it is quite easy, but using it effectively can require pulling in third-party tools. In this workshop we’ll cover how to run your grid effectively, using best practices culled from several large grid installations.
Fri, Sep 11
Timezone: Asia/Kolkata (IST)
Anne-Marie Charrett - 2020 Vision:: Leadership
It’s 2020 and a new decade awaits us. 100 years ago, our world was famous for the Russian Revolution and World War 1.
In our century, we are also facing tremendous upheaval. The bushfires in Australia, floods in Indonesia. We are seeing the impact of climate change before our eyes.
Technology-wise, agile has become mainstream. Hardware is cheap, CI and CD are commonplace. Robotics and Machine Learning is a reality. This is the age we live in. This is the now.
How do we begin to plan for such a reality? How do we test software in such an ecosystem? Should we even try? How do we maintain skills, when frameworks change as rapidly as nail varnish on a teenager?
As leaders in software testing, these are the questions we need to be able to answer. We need leadership now more than ever. We can’t predict the future, but we can prepare ourselves to be able to deal with change, to develop strategies that facilitate rapid learning and rapid change.
We can upskill our people to be able to handle inconsistency and complexity. To know when technology is beneficial and when to rely on our ability to think critically.
We must create environments where talent can grow and thrive. We learning (and failure) are embraced. Today is the day, we become the test leaders of the future. Are you with me?
Welcome Address & Conference Overview - 20 mins
Coffee Break - 25 mins
Krishnan Mahadevan - My experiments with Grid
Everyone starts off with a simple grid setup which involves a hub and one or more nodes.
This traditional setup is a good start but the moment one starts to get serious with the selenium grid and decide to house their own selenium grid for their local executions, that is when issues start.
My experiences with the Selenium grid in the past couple of years has led me to get introduced some of the most prevalent problems with maintaining an in-house selenium grid.
- Nodes get unhooked randomly due to network glitches.
- Nodes introduce false failures due to memory leaks.
- Selenium Grid running out of capacity.
- Nodes require OS upgrades/patches etc.
- Needing to deal with auto upgrades by browsers (especially chrome and firefox)
Some of these issues I managed to fix by building a "Self Healing" Grid wherein the nodes automatically get restarted after they have serviced "n" tests. But that still didn’t solve many of these other problems.
That was when I felt, what if there was an on-demand selenium grid.
What if the Grid could do the following ?
- The Grid auto scales itself in terms of the nodes based on the current load.
- The Grid does not require a lot of infrastructure to support it.
- The Grid can plug itself into some of the cloud providers or leverage a solution such as Docker so that the nodes can be spun and shutdown at will.
That was how the idea of "Just Ask" an on-demand grid was born.
Just-Ask is an on-demand grid. It has no nodes attached to it.
It’s designed to spin off nodes on demand, run test against the newly spun off node and after test runs to completion, clean-up the node as well. The node can be backed by anything. It could be Docker (or) it could be a VM running on any of the popular clouds.
The session aspires to walk the audience through with my experiments with the selenium grid, my learnings on the selenium grid internals and how I used all of that knowledge to build my own On Demand Selenium Grid. What better avenue to share these learnings than a Selenium Conference.
The session will introduce the audience to the grid internals and their concepts such as
- What is a Selenium Remote Proxy ? What is it used for? What can you do with it?
- What is a Hub (or) Node level Servlet ? When would you need one ?
- All of this followed by a quick demo on "Just Ask", the on-demand grid that I have built and open sourced here: https://github.com/rationaleEmotions/just-ask
Moshe Milman - Fixing Your Automation Challenges in the Era of CI/CDContinuous Delivery is now the holy grail of IT organizations, but most companies are still struggling with the transition into shorter release cycles and faster, more frequent deployments. A key challenge that companies are facing in that transition lies around test automation, and in this talk we will cover these challenges in details and demonstrate how successful companies are addressing it.
The key takeaways from this session include:
1) Testability hacks + best practices2) Tips for reducing your testing matrix to support the automation speed required for frequent builds/releases
3) Examples of team structure/architecture from leading companies to support better alignment around automation4) Guidelines for picking the right open source tools / frameworks
5) Learnings/Issues to avoid when designing your automation architecture
Virender Singh - Rise of Shadow DOM - Lets solve it through WebDriver
We will also talk about the WebdriverIO tool v5.5.0, which implicitly support Shadow DOM elements.
Syam Sasi - When Ansible meets Selenium Grid - Story of building a stable local iOS simulator farm
There are dozens of matured docker based solutions available in the market for Android automation testing,
what about iOS testing?
It’s hard to create and maintain the docker based solution for iOS testing since it demands the Xcode and for the optimal performance the system needs the Apple certified hardware.
Combining Ansible with Selenium Grid yields a powerful combination because it allows us to set-up our grid and nodes in just a few seconds. In this talk I will demonstrate how to use simulators to build a reliable and scalable in-house iOS simulator lab using Ansible, Selenium Grid and Appium.
Harinee Muralinath - Building Security into your Continuous Delivery pipelines
Functional testing has gone through a paradigm shift from an afterthought in the waterfall model to being Agile and moved early and testing often as possible by the shift-left process. We strive to get feedback early, often and continuously. However, application security has been mostly an afterthought still, and awaits for 'vulnerability assessment' and 'penetration testing' towards the end of the app life cycle, just before releasing.
Including Security as a part of your DevOps process is a revolutionary shift, and it carries 3 main aspects - culture, process, and tools.
In this talk, we will focus on the ‘tools’ aspect, especially in a CI/CD pipeline. And a demonstration of setting up a build pipeline, with the ideal layers at which security testing should be added. There are vast open communities which strive towards bringing security to everyone's hands, and thus make the digital space safe. Thanks to them, and the ever-growing need of defensive security, there are a lot of tools available which a developer/QA can put in their pipelines to get appropriate feedback. I will share my experience in analyzing the diverse sets of such tools, what they do, why they are important, and by what parameters should you measure the right tool for your project.
The categories we will talk about will include SAST, DAST, Dependency checking, Container scanning, and Secret scanning.
Gaurav Singh - How to build an automation framework with selenium : patterns and practices
With an ever increasing no of businesses being conducted on web the testing need to write automated tests for the app's UI is something that can never be ignored. As you all know Selenium provides an API that enables us to do this quite effectively.
However, when tasked with setting up the automation framework, there are a lot of questions that arise in the minds of aspiring test developers regardless of what level they are in their career.
Some of such questions are:
- How does one actually go about the business of building a robust and effective automation framework on top of selenium?
- What are the elementary building blocks to include in the framework that an aspiring automation developer should know of?
- How should we model our tests? XUnit style vs BDD?
- Are there good practices, sensible design patterns and abstractions that we can follow in our code?
- What are some of the anti patterns/common mistakes we should avoid
A lot of literature, documentation and blogs exists on these topics on the web already.
However In this talk,
I would combine this existing knowledge and my years of experience in building automation frameworks and breakdown these elements and walk you through exactly the sort of decisions/considerations and practices that you can take while starting to implement or improve the UI automation for your team.
Hope to see you there!
Nemoto Tadashi - Creating stable and parallelized Windows environment using Azure DevTest Labs
Previously, I created Selenium Grid on Kubernetes environment for web test automation.
Because Docker image is distributed(docker-selenium), it became easy to create/scale Chrome / Firefox test automation environment.
However, there are still users who use Microsoft Edge or Internet Explorer, and we need to let them use our service/website comfortably.
It is clear then that we have to either run manual tests automated tests on these browsers before the release.
However, creating Windows 10 / Microsoft Edge browser test automation environment is still difficult because of its license.
Therefore, we decided UI Testing strategy for multi-browser(prioritize the execution frequency and the number of test cases)
In addition, in order to realize faster and more stable QA feedback of Windows10 / Microsoft Edge, I created parallelized Windows 10 / Microsoft Edge test environment using Azure DevTest Labs.
In this talk, I will talk how to create this environment.
Rajdeep varma - The Joy Of Green Builds - Running Tests Smartly
So you have got a few UI tests and they are running in parallel, great! However, life will not be so sweet once these 'a few' turns into 'a lot'. We grew from a few to 1500 UI tests (although not particularly proud of this number, there are situations and reasons)
We started with a simple parallel distribution of tests 3 years ago. As test count increased failure count run time increased along with increased flaky tests. Mobile tests had their own challenges (eg. device dropping-off, random wi-fi issues, etc) To keep up with this, we created a queue and workers based solution which could distribute the tests more efficiently (https://github.com/badoo/parallel_cucumber). Over time, we made more improvements, in particular:
- Segregation of failures based on infrastructure issues and re-queue the tests
- If a device/emulator malfunction, rescue the tests to another device
- Repeating a single test on 100s of the worker in parallel to detect flakiness
- Repeat a test if a known network issue
- Terminating the build early if more than a certain number of tests have failed
- Health check of each device, before each test to ensure reliability
- Muting a test if failure is known, and highlight outdated mutes if the related task is fixed
In this talk, I will talk about the initial challenges with running UI tests in parallel (Selenium and Appium), how we approached the queue based solution and continuous improvement of this solution; finally, how attendees can use it at their workplace or create their own solution based on our learnings.
Tarun Narula / sandeep yadav - What to do when tests fail
What to do when your tests fail? Read on...
Functional automation is known to be flaky. A test passes sometimes & fail the other times. The failure can be attributed to multiple factors. We need to find out the root cause & then work towards fixing it for increasing automation tests reliability.
In this talk, we will not only be discussing causes that lead to a test failure, but we will also talk about prevention, early detection & fixing these failures for good.
We will discuss some common test failure causes such as locator changes, browser compatibility issues, coding bloopers etc
You will get to know how you can get alerted early about any test failures. We will be discussing topics such as running tests on under development builds for getting early feedback, triggering slack/SMS/email notifications with failure details for immediate redressal and many others.
You will get to know how to prevent failures by building robust locators, exception handling, making use of APIs for test data setup, building atomic tests, making use of waits, retrying your failed tests, rebuilding your Jenkins jobs automatically based upon a failure percentage threshold & so on.
At the end of this talk, you will be confident on how to deal with your failing tests!!
LunchBreak - 60 mins
Tomasz Konieczny - Serverless - how to speed up tests over 300 times and achieve continuous feedback?
Automated tests can provide results faster and it’s possible to execute them more frequently than manual ones. That allows to test earlier in the development process, decrease overall time needed for tests and what is probably the most important it’s possible to release and deliver business value faster and more frequently.
But what if we have more and more tests and even automated execution of them takes too much time - 10 minutes... 30 minutes... maybe even hours? Should we consider the ability to execute full tests set just a few times a day as something normal? Is adding more compute resources the only option to reduce the execution time? Or maybe there are too many high-level tests and some of them should be replaced by low-level ones according to tests pyramid? Is the tests pyramid still valid in the cloud world?
During the presentation you will see how the serverless cloud services like AWS Lambda may be used to run tests in the highly parallelized environment that can speed up execution even hundreds of times.
Gopinath Jayakumar / Babu Narayanan Manickam - Expanding boundaries of WebDriver with DevTools Integration
Though Selenium taking most of the stake in the UI test automation tool market comfortably, there were always challenges that were for selenium test automation engineers are handicapped with especially when dealing with modern JS technologies. For example,
- dealing with DOM elements to solve the stale / loading / non-interactable elements,
- handling full screenshots to know how the elements at the left, bottom, etc,
- measuring the performance of request and response resources at different speeds,
- monitoring the memory of the pages, controls, etc,
- attaching to an existing browser for debugging the failed scripts and many more.
These problems were largely resolved with the integration of selenium with devtool protocols. And that makes the selenium engineer's life merrier than before.
Why this proposal can be different from others?
- Our solution can be executed as independent with chrome dev tools or with selenium. That gives the power back to the automation engineer to choose what and how to debug/run their tests.
- We used this solution for one of our largest enterprise customers and moved this solution to public repository this week (for this conference and beyond). With that said, we tested reasonably with more than 10,000+ test scripts and more than 1M tests.
- The present solution is completely (100%) packed with all Chrome Dev Tools API in Java and with that said, any Java Selenium automation engineer can bind in minutes for their existing code base with no additional dependencies.
- Finally, we love to present at the local home to start our selenium conference campaign. Where else?
The present proposal largely connected with Chrome and Selenium in Java language. However, there is no limitation to expand the boundaries for other language bindings and browsers.
Google Chrome, the most picked browser for browsing, which makes it the primary concentration for developers and testers. DevTools is one such boon for developers, testers especially the new aged test automation engineers. With that said, we built the following design pattern to allow chrome dev tools API to marry Selenium using debugger address / remote targets.
David Burns - Selenium: Giblets and all
Selenium has done a pretty good job in keeping the API surface friendly and usable, but what actually happens when you call some of the commands? In this talk, David will walk you through what happens when you make a call in your test, how it gets to the browser, what happens in the browser, and how it returns all the way back to your test.
Selenium is designed so that each of the commands works synchronously, so you know that a command has finished before it moves onto the next. This creates some interesting problems in browsers since they are mostly designed around asynchronicity.
We will start with how each of the bindings communicates with the browser and then move on to how navigation works. David will show all the different aspects that we need to figure out to tell if a page is “loaded”. He will also show where it goes horribly wrong and how you can write code in your tests to stabilise around these “anomalies”.
From there we will have a look at how clicks work from making sure they are trusted to what if they cause a navigation.
Deepak Koul - Is quality really everyone's responsibility - The Quality Accountability conundrum
"Quality is everyone's responsibility" has to be one of those phrases which looks great on t-shirts and posters but when put into action, often fails.
There is a funny leadership story which goes like this " There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it"
Therefore assuming that everyone in your team (developers, designers, QEs or analysts) would be instinctively inclined to incorporating quality in their day to day work is a terrible assumption to make.
Quality is everyone's responsibility might be true but it cannot work in isolation, it has to be supplemented with an accountability framework.
In this talk, I am going to present exactly that framework and help people especially leaders - technical and people, implement proactive quality processes in their teams.
The information presented in this talk does not require audience to have any technical knowledge and applies to all the roles.
Shi Ling Tai - Start with the scariest feature - how to prioritise what to test
It can be intimidating for inexperienced teams embarking on their test automation journey for an existing code base. There is so much to test, and so many ways to test. I often see teams stuck with debating on where to start and what tools to use and best practices:
"We should start from unit tests"
"No, integration tests are better!"
"Should we use tool A or tool B?"
I see this play out all the time, and I've been there before. And the worst that could happen is decision paralysis and inaction.
The bigger question really is "What to test?".
My rule of thumb is "Start with the scariest code". I'll share with you my framework for evaluating the ROI of writing a test for a feature and prioritising what to test.
Sujasree Kurapati - Accessibility Testing Isn't Hard.
Accessibility Testing is often ignored or placed as the last priority during the web or mobile development cycle. Sometimes a11y testing is not done until a lawsuit gets triggered or your potential customer declines to procure your product because it’s not accessible. Ignoring a11y can cost an organization dearly as it takes more time and effort to fix accessibility issues in existing websites post-release, than creating a fully accessible site right from the start (Revenue Loss). Your customers might be quickly moving towards solutions that are accessible while you go back trying to fix the accessibility issues (Customer Loss). Additionally an inaccessible platform fails to seek the attention of almost one billion people (Potential Prospects) – 15% of the world’s population.
In this 20 minutes session we will discuss about a11y APIs that would assist Quality Assurance engineers with various methods to automate a11y testing against accessibility standards, efficiency that automation tools can bring in making web and mobile properties accessible and how easy it is to automate accessibility testing using open sources API (axe-core engine, API and axe for Android). The attendees will also learn about the ROI that an organization could see by integrating accessibility testing in their QA Selenium framework.
Coffee Break - 25 mins
Ivan Krutov - Master-class: Bulletproof Selenium cluster
Browser tests are known to be the flakiest ones. This is partly because browser infrastructure is complicated to maintain. But the second reason is – mainstream browser automation tools such as Selenium server are far from being efficient.
During my previous talks I was speaking about Selenoid - a truly efficient replacement of the standard Selenium server. This year I would like to do a live demonstration how to organize a fault-tolerant and easily scalable Selenium cluster using virtual machines in the cloud. I will start by setting up Selenoid and show its powerful features like video recording, live tests debugging, manual testing and many more. Then I will configure Selenoid to send logs and recorded videos to S3-compatible storage. Finally we will run a Ggr load balancer instance allowing to use all running Selenoid nodes and organize a single entry point to the cluster.
Boni García - Developing Selenium tests with JUnit 5
Selenium has become the de-facto standard framework for end-to-end web testing nowadays. JUnit 5 is the latest version of this popular testing framework and implements a brand-new programming and extension model named Jupiter. This talk presents Selenium-Jupiter (https://bonigarcia.github.io/selenium-jupiter/), an open-source JUnit 5 extension aimed to provide seamless integration with Selenium. At first glance, it allows creating Selenium tests using local or remote browsers, reducing the required boilerplate thanks to the dependency injection mechanism supported in JUnit 5. Moreover, it will enable us to use different types of web browsers and Android devices in Docker containers out of the box. All in all, Selenium-Jupiter can be used to carry out different types of tests for web and mobile applications, including functional, compatibility (using test templates to define different devices or browsers -types and versions-) or performance tests.
Moreover, Selenium-Jupiter provides advanced capabilities for diagnostics, i.e., finding the root cause of a failed Selenium test. The first one is screenshotting. When a failure is detected, the state of the browser just before the failure happens can provide the clue to determine the nature of the problem. If the screenshot is not enough, the next step is to watch the browser during the test to observe the evolution of the web under test. This process is straightforward when running a test with the local browser, but it can be tricky in the usual case of running a test in a headless environment, such as Jenkins or Travis to name a few. In Selenium-Jupiter, this is done very smoothly thanks to the use of web browsers and Android mobile devices in Docker containers, in which the GUI session is recorded using VNC and FFmpeg. To ease the access to such information, Selenium-Jupiter provides seamless integration with Jenkins, publishing the resulting files (PNG screenshots and MP4 recordings) in the Jenkins GUI when a test fails.
Jesus Sanchez Martinez - Test and monitor one website is not that hard, but what if you need to do it to over 40 websites?
Onestic QA department made and maintained test suites, which was a huge bottleneck in our development process.
In order to solve it, management bought us an idea: our developers must be able to build their tests using a DSL framework without friction. QA maintain a big library that provides resources to developers. From there, they were free to extend this library in each project. With all of this we have a CI process, a 1 hour execution to monitor our results and of course our bot, SpongeBot.
SpongeBot can check 40+ e-commerce sites, with 4+ environments, for desktop and mobile platforms always available for developers.
With this solution we achieve to decentralize the work, add value testing production and the confidence in SpongeBot notifications if something goes wrong.
Chitvan Singh - From hours to minutes: How to write test cases for a faster regression suite?
Teams that release multiple times in a day cannot afford to have their regression suite run into hours. Most of these teams run their entire suites in less than 15 minutes. While infrastructure and configuration help bring this number down, the largest gains come from writing test cases that can cope with the requirements of a setup that runs so fast.
In this session, I’ll talk about:
How to identify test cases that need refactoring
How to write atomic test cases
How to use selenium locators smartly
Using product APIs to switch between user states
Using the right Selenium waits
Parallel execution and queuing
Zachary Attas - Selenium For All - Setting Your Team Up For Success So Anyone Can Understand and Write Tests
Are you the only one contributing to your test suite, because others feel too intimidated to contribute to it?
Have you ever looked at a test fail and not known where to investigate first?
This talk will cover techniques for uncomplicating your test suites, so they can clearly test exactly what's needed to be tested, allowing others to contribute tests. When the process of writing tests is shared, everyone has an impact on quality, and that can be quite contagious for a company's culture.
In addition, this talk will cover techniques for failing transparently, so you will no longer feel stressed triaging test fails. Quality is something shared by the entire team, so anyone should be able to triage test fails. When tests fail for a specific, clear reason, you'll be thanking your past self for writing logic to fail transparently, saving you half a day going down a rabbit hole.
Simon Stewart - Selenium: State of the Union
What's the latest in the world of Selenium since the last SeleniumConf? In this talk, we'll cover Selenium 4, how the project is run, how you can participate, reminiscing on the Old Days, and reflect on something that at first doesn't appear to be related to Selenium, yet is intimately related to it.
Lightning Talks - 30 mins
Sat, Sep 12
Timezone: Asia/Kolkata (IST)
Narayan Raman - A Tale of Two Automation Tools
Selenium and Sahi started as open source tools around the same time and from the same company. Both tried to solve the problem of UI automation and both have been successful in their own ways. But the resemblance ends there. Selenium became Webdriver and became one of the most popular free tools in testing. Sahi became commercial Sahi Pro and is successfully used in various organizations to automate complex UI's spanning web, desktop and mobile applications. Right from who the user of the tool is to whether the tool should be open source or commercial, there are various philosophical, technical and commercial junctures where Selenium and Sahi have diverged. Based on our journey of over 10 years, this talk reflects on those divergent decision points, the philosophy behind those and what their outcomes have been.
Special Announcements - 20 mins
Coffee Break - 25 mins
Abhijeet Vaikar - End-end test code as a first class citizen
"All tests in today's automated regression run have been marked as Untested. What happened?"
"No notifications are being sent for test runs on the channel"
"I pulled latest code, and the framework dependency shows compilation error"
"What does this new method in the framework do?"
How often do you hear such things within your team?
As Quality champions, we need to walk the talk. When we expect our developers to write quality code, write unit tests, build features without introducing bugs, the onus lies on us (as test engineers) to do the same. With almost every test engineering team writing automated tests to check functionality of their products and services, it becomes very important to ensure that the test automation framework and the test scripts are bug-free and follow good standards of software engineering.
It cannot be stressed enough that test automation code should be as good as production code. In order to build production-quality test automation framework and scripts, a number of steps can be taken at:
1. Code & System Level
2. Process & People Level
Our test engineering team went through a transition from having random & unexpected failing test runs to having greater confidence in the quality of the tests. Learn from this case study of our journey to ensure that end-end UI automated tests are built with quality in mind. We will also see demonstration of some of the use cases.
Shama Ugale - Webdriver connector for Botium - Tool for testing Conversational UI
Last year was dominated by the smart devices & voice based home assistants. These use the conversational interfaces unlike other applications to interact, built using advanced algorithms, ranging from natural language processing to AI/ML techniques. They are constantly learning by themselves improving the interactions with the user bringing up the challenge of non-deterministic output. To such interfaces, natural language is the input & we humans love having alternatives & love our synonyms, express using emojis gifs & pictures. Testing in this context moves to clouds of probabilities.
Unfortunately Selenium cannot be used to automate such systems and hence Botium was designed.
In this session I will cover the Selenium driver for Botium to automate E2E tests on Web UI and mobile along with testing strategy, testing NLP models & automating these tests to the CI/CD build pipelines with a DialogFlow based 'Coffee-Shop bot' as an example during my demo.
Rabimba Karanjai - Testing Web Mixed Reality Applications: What you need to know for VR and AR
There are already over 200 million users consuming VR applications by 2018. And with Google, Mozilla pushing WebXR capabilities in browser and vendors like BBC, Amnesty International, Universal, Disney, Lenskart and a lot of them adopting them to their websites, we will soon see a huge rise i demand for Web VR and Mixed Reality applications.
But how do you test them in scale? How do you define "smooth" as opposed to just responsive?
In this talk I will go over some key details about the WebXR specification. The work that Mozilla, Google and the W3C Immersive Web Group is doing. The differences between testing a regular web page and a Mixed Reality enabled one. What to watch for and how you can automate it.
Shweta Sharma - Advanced automated visual regression testing
In this era of Digital transformation, clients have been demanding shorter and quicker releases. Shorter and quicker releases mean your team should not only be able to develop them at the required pace but also test and release them at a sustainable pace. User interface plays an important role in the client's business and there are organizations that release new features, fancy CSS regularly that support multiple browsers, multiple Operating systems, mobile devices. Verifying the frontend on this browser/device/OS matrix by humans is not only extremely time consuming but also prone to human errors. In fact, testing by humans should primarily focus on discovery leaving the repetitive and error-prone tasks to tools. Hence, automating Visual tests is becoming less of an optional activity and more of a must-have activity within the team. Ensuring visually perfect user experiences is equally important as to have the functionality working.
Anuradha Konduri / Keertika Gangwar - How we are reducing Test Failure Analysis time using Machine Learning at Expedia
How many of us have spent those hours just to check Automation run reports to first determine whether it was an actual bug or environment specific issue or an automation issue. We agree that no matter how much robust we make our UI Automation frameworks, we always encounter Automation/Environment specific failures which increase the time spent on analysing those failures and spotting actual issues/defects. At some point, all of us have had a bad day where we see so many flaky tests, that we lose confidence in the reliability of UI Automation results and tend to ignore the results over a period of time. We propose a solution to help us overcome this problem using the most trending technology of last decade - Machine Learning. The fact that we run around 130k test runs a day with around 2.3 million test records saved in MongoDB every month motivated us to look into Machine Learning as an approach for the problem statement we have.
What if we can use ML algorithms to find patterns in the day-to-day UI Automation error messages that we see, to tell us if it is - an actual bug or not !!! All of us use various Selenium based Test Automation frameworks like Cucumber, TestNG, Scalatest, Nightwatch JS etc. which have their own libraries to report any Test Validation/Automation failures, hence vary quite a lot in the formats. Also, it's difficult to find a common pattern in user defined error messages. A typical error message from UI Automation would have the - Message Stack Traces , other error data dumped from Selenium etc. We can argue that we can take only the Message part of this as input to be able to predict the outcome for us. However, we have seen many instances where these Messages are not very self-explanatory and we will have to look into the trace/error details to actually determine the root cause. Considering the whole error message is not as easy as it sounds.
Hence, the problem we have at hand is unstructured text data. Our approach includes the steps -> Collect Training data (viz., pre-classified errors), Clean the data to be fed to the model, Identify a simple yet powerful algorithm such as SVM, Random Forest, Naive Bayes etc., Classifiers to work with, Tune the model to identify the right metrics to help us calculate the reliability of the resultant predictions.
rajni singh - End to end testing strategies for intelligently connected hybrid world of IoT
The intelligent mesh of devices is nothing but connecting millions of users, use cases, apps, and devices together to support application developed based on the internet of things. Currently, we see around the world that there are thousands of use cases, millions of apps, billions of users and trillions of things and if you think QA and testing in the interconnected intelligent devices then the scope would be very wide as verification and validation will be applicable in each interface and as and when grows.
I will talk about the challenges faced during IoT application testing and how it can go wrong. To thorough test all the area with the given challenges and IoT test lab is setup. What all are the solution to overcome the challenges. Some important aspects like continuous integration of hybrid environment and testing with multiple devices and with millions of use cases, to improve the existing conventional method with intelligent automation, and most important is scalability and security.
Although testing emerging technology and applications always is exciting, seeing their own strategies and tools fail in IoT testing can be frustrating even for well-seasoned testers. I aim to provide testers with a better understanding of connected systems like smart cities, connected enterprise and help them to apply their critical thinking to deal with uncertainty in their test objects.
Smita Mishra - Careers in Testing – Identify your SuperPower
Most testers follow the career path their organisation is offering them. Sometimes, this path seems dated. Specially when one compares them to other organizations which seem to be taking testing more seriously or are atleast more advanced in their approach to people and technology.
What could be the possible options beyond a certain point as a tester? Will present certain questions to the testers - What do you enjoy doing most? What comes naturally to you? How does the competition look like? What could you potentially learn? What are the alternates or extended skills you enjoy - data science more or engineering more or something else? What are your super powers? What could be the path of least resistance to make a shift. Will share leading examples of some "popularly successful" career.
In the 2nd half of the talk - will get the audience to engage to understand the designations and potential career paths that interests them. Will discuss out specific ambitions and goals and share what has worked for others and what could work for them & how to go about it.
Vinayaka Mayura G G - Search Relevancy Testing: QA in Machine Learning Models
The adoption of Artificial Intelligence is getting more traction, it is in need to enhance QA capabilities to cope up with these skills. Machine Learning is used extensively in retail applications for solving complex problems, one of them is solving the search relevancy. Showing the appropriate results for the user is important for the conversion rate to go high. As Machine Learning poses different challenges such as a Test Oracle, Fairness, Correctness and Robustness to do QA, We may need to follow different approaches and testing techniques to do the QA for Machine Learning models.
Different Machine learning types such as Supervised and Unsupervised Models have different characteristics and are used for different types of problems. Though these solves different complex problems, Machine learning Models also a unit of software code that needs to be verified as a normal software system. When a Machine learning model is seen as a whole system, it may look complex and unsolvable. We can group them into small modules and verify for quality. Black box and White box testing techniques can be applied to verify the functionality. Data, Feature Engineering and Algorithms are the major part of the Machine Learning model. We will see how we applied different techniques to validate these.
This talk is focused on viewing the Machine Learning software as a whole and performing the Quality Analysis for it. We look at how different is testing a machine learning model from typical software testing. We will discuss the challenges that came across, the Process involved in building an ML model. We take an example of Search Relevance for an explanation. We will dive into the areas where quality is assessed. The significant factors considered here are measuring Accuracy and Efficiency. We will look into the different black box testing techniques for different Algorithms. Let us also see how traditional testing is different from testing machine learning applications. I will go through different black-box testing techniques with examples following a live demo.
Zbyszek Mockun - Automation in the world of projects
The world of a software house is a constant search for compromise between quality and costs. In many cases, the cost-cutting starts from the test automation. Then you start to talk about ROI but recognize that numbers are not on your side. We were there and what we have found out is that only a complete change in our approach allows us to find common ground with our clients. I will reveal one detail from the presentation - we are not talking about test automation with clients anymore - as a result we do it more and more.
Are you surprised that success automatically generates new challenges which we further translate into opportunities? We had to reconsider our approach to the test automation environment, internal frameworks and the way we share them between projects, including code ownership, … And again, one simple but unobvious solution allows us to both deliver what we promise and to earn more on our projects.
As we have been reshaping our approach to the test automation, we had to change the way of delivery too. One of the main decisions was skip out the role of test automation engineer (or software developer in test). We decided to go with the whole team approach which is consistent with the way we sell it.
Find it interesting? Join me and listen to our story about how we have transformed test automation.
LunchBreak - 45 mins
Varuna Srivastava / Wim Selles - Build a responsive typescript wdio framework
You will leave with your very own example automation framework that demonstrates advanced principles of wdio using typescript automation design. We will integrate with allure reporting.
- A robust and scalable framework with the advanced principle for ui testing.
- A selection of design patterns for the designing framework.
- Concepts in designing your ui automation such as modeling data within your application and componentizing page objects.
- A framework which is responsive for web applications.
Tarun Maini - LifeCycle of Testing AI/ML Applications
With the seismic shift in industry and development of new technologies emerging, QA’s testing approaches are also changing, we must know the right strategies and algorithms to test. One of the latest technology emerging is Artificial Intelligence and Machine Learning. And its applications like Self driving cars, Virtual Assistants are everywhere. They have great impact in our life and most of our decisions, behaviour & destinations depend on them.
So in this presentation/Workshop i would like to present all the ways/strategies/ challenges faced while testing AI/ML applications. Join me in creating a Machine Learning application from scratch and then take it to testing stage, creating edge case scenarios and validations.
Time Management: To make sure that all people are upto date with with setup for hands-on, i will be sharing this document with the participants 12 days before in a temp slack channel, where they can share the progress and ask queries to resolve them quickly.
*No internet is required for participants if they follow the setup doc.
Simon Berner / Pooja Shah - Git Foundations
Copying of files in a time-stamped directory or by renaming the files itself is often used to keep older versions. This approach is very common because it looks so simple in the first place, but it is actually incredibly error prone.
In today’s world of CI/CD pipelines where new versions of our production and test code has to be checked out at speed, it is unthinkable to have such a version control system in place. So what are you waiting for? Let’s go and get a proper version control system in place for your CI/CD pipelines. But not only the selection and usage of a version control system makes you knowledgeable, no, learning by making frugal experiments, failing, pairing and explaining it to others is what makes you a brave explorer and brings you to success.
Start using the version control System Git can be daunting and frustrating. But knowing how Git works and how to use it to your advantage in your unique context is getting more and more essential if not even crucial. Especially if you want to work with CI/CD and take care collaboratively on the productive and test automation codebase in your team, you have to know Git.
It is a long way to the top if you want to Rock ‘n’ Roll with Git and this workshop is the perfect start for it. In 90 minutes you will learn how to use the basic commands on the command line.
Whether you have just started working with Git or want to refresh your existing basic knowledge or you are a novice and eager to learn it, this power workshop is for you. Come in and explore what Git feels like.
Coffee Break - 30 mins
Jim Evans - I'm Not Special.
Since nearly the very beginning, the software industry has prized the concept of the visionary, the extraordinary creator, the lone person who relentlessly pursues their goal against impossible odds, all to help make the world a better place through technology. The industry also reveres the concept of dedication to the software above all else in the lives of those visionaries. Even those who have chosen to shepherd large portions of popular open-source projects are often seen as people who have a special quality about them that places them above other developers. These concepts are myths and oftentimes dangerous ones.