Q & A with the Selenium Committee moderated by Anand Bagmar

Q & A with the Selenium Committee moderated by Anand Bagmar

Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.

This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.

Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.

This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

You always wanted to start Mobile app Automation using Appium but did not know where and what to start with? Or started but got stuck among understanding capabilities, locating elements, finding whats wrong and your daily work deadlines? Yeah, we all have been there. Hence here is this workshop, precisely designed for beginners to quickly get started with Appium in NodeJS.

At this workshop, you will learn everything from the scratch step by step. We will together understand the key concepts, apply them right there and write our “first test” in appium before we learn more nitty-gritties as takeaway for you to go and implement at your actual work directly.
Think of this as a one-day crash course in Appium. The course will be prepared in NodeJS but there will be help for those who want to get started in Java.

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)

When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….

What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.

Here, we will play a game – “Collaboration - A Taboo!” – where you will –

• Re-learn collaboration techniques via a game!
• Learning applicable for individuals & teams, in small or big organisations
• Re-live your childhood when playing this game

Be prepared for a twist which will leave you thinking!

How to build a product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about “loosely” is the feedback loop from your consumers / end-users and how it leads to making better design, feature and user experience decisions.

SO, what is this feedback loop? Is it a myth? How do you measure it? Is there a “magic” formula to understand this data received? How to you add value to your product using this data?

In this session, with the help of a case study, we will understand and also answer the following questions:

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)