Q & A with the Selenium Committee moderated by Anand Bagmar

Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.

This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

You always wanted to start Mobile app Automation using Appium but did not know where and what to start with? Or started but got stuck among understanding capabilities, locating elements, finding whats wrong and your daily work deadlines? Yeah, we all have been there. Hence here is this workshop, precisely designed for beginners to quickly get started with Appium in NodeJS.

At this workshop, you will learn everything from the scratch step by step. We will together understand the key concepts, apply them right there and write our “first test” in appium before we learn more nitty-gritties as takeaway for you to go and implement at your actual work directly.
Think of this as a one-day crash course in Appium. The course will be prepared in NodeJS but there will be help for those who want to get started in Java.

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)

When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….

What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.

Here, we will play a game – “Collaboration - A Taboo!” – where you will –

• Re-learn collaboration techniques via a game!
• Learning applicable for individuals & teams, in small or big organisations
• Re-live your childhood when playing this game

Be prepared for a twist which will leave you thinking!

How to build a product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about “loosely” is the feedback loop from your consumers / end-users and how it leads to making better design, feature and user experience decisions.

SO, what is this feedback loop? Is it a myth? How do you measure it? Is there a “magic” formula to understand this data received? How to you add value to your product using this data?

In this session, with the help of a case study, we will understand and also answer the following questions:

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about "loosely" - is the feedback loop from your end-users to making better decisions.

SO, What is this feedback loop? Is it a myth? How do you measure it? Is there a "magic" formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers
• How do you know your product is working well?
• How do you know your Consumers are engaged with your product?
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behaviour, before making any code change?

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)

I have been meditating since quite some time now. Meditation gives me peace of mind. It allows me to be more conscious and aware of things around me. Interestingly, I have experienced the benefits in Coaching & Mentoring, in Testing and also in Automation activities on Agile projects.

In this talk, I will be sharing how Meditation helped me in Testing on Agile projects - while working on various activities that help my team build a Quality product. I will also be sharing experiences of the way we approach certain testing activities - and how that correlates to the conscious mind, or not! These experiences can be applicable to any role on Agile projects.

PS: I will be referring to some Meditation techniques that work for me. This is not about any religion / faith / practice.

Analytics is changing the way products and services are being created and consumed.

In this session, we will learn

• What is Analytics?
• Why is it important to use Analytics in your product?
• The impact of Analytics not working as expected

We will also see some techniques to test Analytics manually and also automate that validation. But just knowing about Analytics is not sufficient for business now.

There are new kids in town - IoT and Big Data - two of the most used and heard-off buzz words in the Software Industry!

With IoT, with a creative mindset looking for opportunities and ways to add value, the possibilities are infinite. With each such opportunity, there is a huge volume of data being generated - which if analyzed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about.

1. How is the end-user interacting with the product? This will give some level of understanding into how to re-position and focus on the true value add features for the product.

2. With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns from what has happened, and find out new product / value opportunities based on usage patterns.

There is a saying ..."Sukh baatne se badhta hai, dukh baatne se kam hota hai", translated as - "happiness increases & sadness reduces on sharing with others".

We want to take this opportunity to share with our experiences - the good and the bad, in the journey of building a Test Automation framework for an AngularJS based application. 

We will learn, by a case study, what thought process we applied on the given context (product, team, skills, capabilities, long term vision) to come up with an appropriate Test Automation Strategy. This Test Automation strategy covered all aspects of Test Automation - Unit, Integration, UI - i.e. End-2-End tests (E2E).

Next, we will share how we went about narrowing-down, and eventually selecting a specific Tech Stack + Tools (Javascript / Jasmine / Protractor / Selenium-WebDriver) to accomplish the Test Automation for the product.

Lastly, we will share the challenges that came up in the implementation of the Test Automation, and how we overcame them. This will also include how we managed to get the tests running in Jenkins - a Continuous Integration tool. 

This discussion is applicable to all team members who are working on Test Automation!

P.S. We will be attempting to make a sample protractor-based automation framework available on github for anyone to use as a starting point for setting up protractor-based Test Automation framework.