Challenges in Testing & Automating Games

Testing real-time, multi-user games built for native apps and / or browser-based on phones / mobile devices / tablets / desktop browsers makes testing of regular products as apps, or websites appear like a piece of cake. Testing such real-time and multi-user games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and ease-of-use).

The key challenges I encountered here were:

• Millions of users, playing games on a huge variety of devices (Android & iOS native apps, Mobile-Web, and Desktop Web)
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

An approach to Functional Automation of Real-Time, Multi-User Game scenarios

I overcame the above mentioned challenges by doing the following:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Built a new functional automation framework using java / testing / appium-test-distribution / reportportal / jenkins with focus on – specify test intent once and run on all supported channels (ex: Android, iOS, Mobile-Web, and Web)
• Built a solution for Cocos2d-x layer automation
• Created a vision of CI-CD for the organization, and setup code-based CI pipelines to enable end-2-end visibility
• Made the framework extensible by providing ability to use same framework for multiple games

The focus of this talk will be to:

• Share an example of a particular use case

• Share the solutions, including code snippets, implemented for:
  • Functional Test Automation Framework Architecture & Design
  • Ease of Test Implementation, while maintaining code quality and promoting reuse
  • Test Execution on local Vs CI, on-demand as well as on every new build

Want to get into WebDriver, but don’t know where to start? Already using WebDriver and want to get more out of your framework? This practical, hands-on, course is designed to help automation engineers with all levels of experience. All attendees will leave with working WebDriver code on their machines and a long list of techniques on how they can successfully use the WebDriver API in their own companies.

This practical, hands-on, course will benefit all experience levels of programming, all attendees will leave with working WebDriver code on their machines and a long list on how they can successfully use the WebDriver API in their context.

The Test Automation Pyramid is not a new concept.

The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.

While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.

With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.

In this session, using numerous examples, we will explore:

• Why Automated Visual Validation is essential to be part of your Test Strategy
• Potential solutions / options for Automated Visual Testing, with pros & cons of each
• How an AI-powered tool, Applitools Eyes, can solve this problem.

Testing games as native apps on phones / mobile devices / tablets makes testing of regular apps, or websites appear like a piece of cake. Testing games becomes even more challenging when you think about automating the same.

I got an opportunity to build Functional Test Automation for a suite of games – and what an exciting time it turned out to be!

These games are built either using Cocos2d-x, or Unity (cross platform game engines for mobile games, apps and other cross platform interactive GUI and are known for their speed, stability, and easy of use).

The key challenges I encountered were:

• Millions on users, playing the games on a huge variety of devices
• Limited unit testing
• API testing & Functional Testing done in isolation (mini-waterfall approach)
• Usage of Cocos2d-x & Unity for game rendering – which cannot be automated via Appium
• Limited Functional Automation (for native apps)

Come to this session to understand how I overcame the challenges and helped build:

• A better way of working (you can call it ‘Agile’)
• Break down the walls by fostering a mindset of “build quality-in, as opposed to test for quality”
• Setup vision of CI-CD – and how to setup CI pipelines to enable end-2-end visibility
• Build a new functional automation framework using java/testing/appium-test-distribution/reportportal/jenkins with focus on – specify test intent once and run on all supported channels (ex: Andriod, iOS, …)
• Build a solution for Cocos2d-x layer automation
• Ability to use same framework for multiple games

You always wanted to start Mobile app Automation using Appium but did not know where and what to start with? Or started but got stuck among understanding capabilities, locating elements, finding whats wrong and your daily work deadlines? Yeah, we all have been there. Hence here is this workshop, precisely designed for beginners to quickly get started with Appium in NodeJS.

At this workshop, you will learn everything from the scratch step by step. We will together understand the key concepts, apply them right there and write our “first test” in appium before we learn more nitty-gritties as takeaway for you to go and implement at your actual work directly.
Think of this as a one-day crash course in Appium. The course will be prepared in NodeJS but there will be help for those who want to get started in Java.

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)

When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….

What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.

Here, we will play a game – “Collaboration - A Taboo!” – where you will –

• Re-learn collaboration techniques via a game!
• Learning applicable for individuals & teams, in small or big organisations
• Re-live your childhood when playing this game

Be prepared for a twist which will leave you thinking!

How to build a product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about “loosely” is the feedback loop from your consumers / end-users and how it leads to making better design, feature and user experience decisions.

SO, what is this feedback loop? Is it a myth? How do you measure it? Is there a “magic” formula to understand this data received? How to you add value to your product using this data?

In this session, with the help of a case study, we will understand and also answer the following questions:

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about "loosely" - is the feedback loop from your end-users to making better decisions.

SO, What is this feedback loop? Is it a myth? How do you measure it? Is there a "magic" formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers
• How do you know your product is working well?
• How do you know your Consumers are engaged with your product?
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behaviour, before making any code change?

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!

Attend this talk to understand the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion will include –

• Understanding the context,
• Why no cloud-based solution worked for me,
• The automation strategy for Android / iOS / Web platform,
• The tech-stack (cucumber-jvm / Appium), and,
• The core implementation to build your own MAD-LAB! (already open-sourced)
• How this results in Continuous Delivery (CD)

I have been meditating since quite some time now. Meditation gives me peace of mind. It allows me to be more conscious and aware of things around me. Interestingly, I have experienced the benefits in Coaching & Mentoring, in Testing and also in Automation activities on Agile projects.

In this talk, I will be sharing how Meditation helped me in Testing on Agile projects - while working on various activities that help my team build a Quality product. I will also be sharing experiences of the way we approach certain testing activities - and how that correlates to the conscious mind, or not! These experiences can be applicable to any role on Agile projects.

PS: I will be referring to some Meditation techniques that work for me. This is not about any religion / faith / practice.

Analytics is changing the way products and services are being created and consumed.

In this session, we will learn

• What is Analytics?
• Why is it important to use Analytics in your product?
• The impact of Analytics not working as expected

We will also see some techniques to test Analytics manually and also automate that validation. But just knowing about Analytics is not sufficient for business now.

There are new kids in town - IoT and Big Data - two of the most used and heard-off buzz words in the Software Industry!

With IoT, with a creative mindset looking for opportunities and ways to add value, the possibilities are infinite. With each such opportunity, there is a huge volume of data being generated - which if analyzed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about.

1. How is the end-user interacting with the product? This will give some level of understanding into how to re-position and focus on the true value add features for the product.

2. With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns from what has happened, and find out new product / value opportunities based on usage patterns.

There is a saying ..."Sukh baatne se badhta hai, dukh baatne se kam hota hai", translated as - "happiness increases & sadness reduces on sharing with others".

We want to take this opportunity to share with our experiences - the good and the bad, in the journey of building a Test Automation framework for an AngularJS based application. 

We will learn, by a case study, what thought process we applied on the given context (product, team, skills, capabilities, long term vision) to come up with an appropriate Test Automation Strategy. This Test Automation strategy covered all aspects of Test Automation - Unit, Integration, UI - i.e. End-2-End tests (E2E).

Next, we will share how we went about narrowing-down, and eventually selecting a specific Tech Stack + Tools (Javascript / Jasmine / Protractor / Selenium-WebDriver) to accomplish the Test Automation for the product.

Lastly, we will share the challenges that came up in the implementation of the Test Automation, and how we overcame them. This will also include how we managed to get the tests running in Jenkins - a Continuous Integration tool. 

This discussion is applicable to all team members who are working on Test Automation!

P.S. We will be attempting to make a sample protractor-based automation framework available on github for anyone to use as a starting point for setting up protractor-based Test Automation framework.

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

Teams already building 1000s of tests of various types - UI, web service-based, integration, unit, etc. is a proof of that.

However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

In this talk, we will focus on one of the critical aspects and patterns in building the Test Automation framework - Test Data!

We will look at different data patterns as options and techniques how to create, manage, use, reuse Test Data in a way to keep the tests running in an reliable and deterministic way. We will also discuss what questions to ask, what things to think about in selecting your approach for Test Data!

This discussion will be applicable for any type of Test Automation (web / mobile / desktop), but, we will focus primarily on UI automation frameworks, ex. using Selenium.

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

Design Patterns play a big role in helping achieve this goal of building a good and robust framework. 

In this talk, we will talk about, and see examples of various types of patterns you can use for:

1. Build your Test Automation Framework
2. Test Data Management
3. Locators / IDs (for finding / interacting with elements in the browser / app)

Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury! Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful. But, this is tricky and requires a lot of discipline, rigor and hard work by all the team members involved the product delivery. The Test Automation Pyramid is a great concept that helps teams focus on the "right" type of automation.

To add to the never-ending challenges, a new architecture style has evolved - Microservices Architecture. While there is no precise definition to explain Microservices style, it essentially talks about how to break a monolith application into smaller, independent pieces - usually based on business capability, automated deployment, and various other factors.

In teams building a complex product using the Microservice Architecture approach, the Testing approach also needs to evolve to be effective and validate the functionality quickly, and effectively.

But why does the Testing approach needs to be different for Microservice Architecture?
Here is an example: If your product architecture is something similar as seen here, or here, to make testing effective, and enable fast feedback, you need to break down the testing approach from an end-product (monolith black-box) into smaller components - in this case - at individual service, combination of services, etc. level.

In this workshop, learn, via a game-based activity, correlated with a case study of an Enterprise (, a large Bank), the Testing approach required to take them on the journey to achieving CD in the Microservice world. We will also learn how to break down the test automation pyramid at a Microservice level into smaller test automation pyramids - i.e. Micro-Pyramids to enable early feedback about quality of the product.

Distributed organisation / projects / teams is not a new phenomenon. It is a reality. Another reality is large organisations, primarily due to the size, have different operational structure - which results in separate role-based teams and often CoEs (Centre of Excellence) too!

Though this may work upto a certain level, the separate way of organising and structuring causes impediments, silos and blockers in the way a quality product is built and delivered.

This brings an important question in my mind - do we really work well on such distributed teams?

In this session, learn some practices and techniques to bridge the gap and make working in distributed teams “relatively easier”.

CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury! Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful. But, this is tricky and requires a lot of discipline, rigor and hard work by all the team members involved the product delivery. The Test Automation Pyramid is a great concept that helps teams focus on the "right" type of automation.

To add to the never-ending challenges, a new architecture style has evolved - Microservices Architecture. While there is no precise definition to explain Microservices style, it essentially talks about how to break a monolith application into smaller, independent pieces - usually based on business capability, automated deployment, and various other factors. 

In teams building a complex product using the Microservice Architecture approach, the Testing approach also needs to evolve to be effective and validate the functionality quickly, and effectively. 

In this talk, learn, via a case study of an Enterprise, a large Bank, the Testing approach required to take them on the journey to achieving CD in the Microservice world. We will also learn how to break down the test automation pyramid at a microservice level into smaller test automation pyramids - i.e. Micro-Pyramids to enable early feedback about quality of the product.