A Guide to Virtual Reality Application Testing

Reading Time: 6 minutes:

Virtual Reality: a modern technology

Virtual Reality is a powerful emerging medium that promises to change user interaction with digital technologies. It is now possible for individuals to interact in any virtual reality environment using VR goggles or mobile devices. Various forms of non-immersive, semi-immersive, and fully immersive simulation technology transform many industries at B2B and B2C levels.

At its inception, Virtual Reality was envisioned purely for consumer use cases, such as the gaming and entertainment industry. However, it has also proved surprisingly helpful in various ways for the medical sector, defence sector, and even industrial sector in recent years. The use cases are only set to increase as it gains mainstream adoption.

VR’s rapid growth in popularity, and the nascent stages of the technology, throw unique challenges for companies from the developers’ point of view. Lately, head-mounted displays have been introduced with eye detecting cameras and high-accuracy object detecting sensors. People want to try these features, especially the gaming community. Developers and startups need to constantly adapt and innovate if they’re going to stay relevant. As a result, VR applications have evolved a lot within few years and continue to rapidly grow for the foreseeable future. 

The most prevalent issue in VR apps is their Quality Assurance. The old traditional methods of 2D testing are no longer usable in 3-D VR testing. The industry has failed to establish a standard testing framework and testing technology due to the constant flux. We will go through the various tools available to test VR Apps and how you can effectively test your VR applications. 

Tools for functional testing of Virtual Reality applications

There are specific tools that are necessary for VR application testing. These are:

SteamVR Performance Tool

  • It is a fundamental tool to evaluate compatibility. It displays the compatibility of your PC with the VR app.

3600 EYETRACKERTM software solution 

  • This software tracks your eye movement and lets the user see anywhere in the environment. It also enables the audience to see the user’s point of view and the direction in which the user is looking. It extrapolates the user’s field of view. 

Unity testing tool 

  • Unity is an excellent tool to start with. It is for unit testing to focus on a particular object in your virtual reality environment. One can also use a google plugin called Google Instant Preview with Unity. It enables you to see the building unity in the editor. You can directly run your game, and it’s visible to everyone around you. 

Desired output formats 

  • A detailed Heat map with interactive controls
  • Video in the 2D environment from 360 content 
  • Relive personal viewer experience through VR headset. 

How to test your VR app

1. Define the Objectives 

There should be clarity in creating the aim for the functional testing of the VR apps. The testing must ensure the input and output parameters are fulfilled. Some of the most common objectives are: 

  • Ensuring expected behaviour: The test performed on the VR app must ensure that the app is running as per requirements. That is, it is functioning as expected. However, VR apps are often complex, with many hidden bugs. It also depends upon the human-machine interface to fulfil the objective. 
  • Manual Testing: To ensure human interaction and evaluate how well the user is adapting, utilizing, and immersing in the VR app. 
  • Environment effects: The objective must also ensure the effectiveness of the real-world environment in the virtual world. The user should feel a difference between Reality and virtuality. He also needs to judge how close to the actual environment that virtual environment is. This objective should be able to depict that difference. 
  • Automating specific test: To process the high degree of interaction, evaluation of inputs then sorting and evaluation of outputs, automating a particular part of the test becomes essential.
  • Create a list of desired user experiences that we want from the app and what the publisher will gain from the app 

2. VR Testing Methods 

Trial and Error approach

Virtuality reality apps are based on Reality. So, it relies on human perception. It means – detecting bugs, glitches, issues with the app; the “reality” aspect is subjective to that individual. It also means that testing results will vary from person to person. The best way is to test the behaviour of the application through testing tools. 

Dedicated testing tools like the SteamVR performance tool ease the process of testing. 

Recording sessions

The recording sessions offer a better understanding of the view and human –interaction with Reality. It gives deep insight into debugging. It can be shown to the people outside to get the necessary help. Sometimes, the bugs or glitches. Letting other testers see the recording sessions will allow you to spot errors in later stages.

Automation testing tools 

There are aspects of virtual reality apps that testing tools have drawn better results. Automated testing tools are helpful in UI tests and performance tests. The existing tools in the market boast the testing phase and ensure the quality of your VR app. 

Testing of motion sickness in VR apps 

Reality sickness is one of the significant challenges in the virtual reality environment. The human brain does not fully immerse into another reality. The reality movement directly leads to motion sickness; it is vital to make sure that even the very essential mind can withstand the effects of VR apps. 

User feedback 

Feedback reporting tools ease your work. Suppose you release the app in the market. These feedback tools will help you to discover more bugs and glitches in the app. It will also aid in enhancing the quality of the application. 

UI test automation framework 

For VR automation, we recommend AirTest. It is one of the best open-source frameworks and it is built for VR gaming apps, where you get an automated testing framework. It is cross-platform and it supports Windows, native apps, iOS and Android. AirTest includes app installation, assertion, simulated input and so forth. You can perform image recognition technology to locate and identify UI elements. Automate tests on games without code injections. You can see the results in HTML reports. It’s easier to find failed tests quickly.

Integration Test 

This test requires more skills than automating unit test. Two or more objects are combines and test in the simulated environment or a specific scenario. These kinds of tests are created and integrated into the test engine. This engine requires multiple systems, more skills and resources. 

End to End Testing / QA testing 

It means a full playthrough of the VR app that needs to be tested. It is done manually as well as automatically. But the automated version of QA still needs much input to become more usable.    

Things to consider before automating your VR Test cases 

  1. Some of the test scripts are already available. The Quality assurance process can use those existing test scripts. It is a heuristic method to find repeatable areas of the build. Those areas can be automated with optimum effort.
  2. A fully developed VR app can be tested by using source control and continuous integration. Consider the scale of your system, and size of the team and the type of app before leveraging the source control and continuous integration. You can take references from Git and Mercurial. 
  3. Always test the 2D version of your app. It saves time and accelerates the process of debugging. 
  4. Use telemetry data – it is helpful in long term work. Especially when you analyze a specific error, telemetry data should be relevant to the user walkthrough of the app and the user flows of a successful playthrough. 
  5. Whenever a new change happens, the compiling code and running the asset build on the CI systems detect those issues. The diligent method is to tell your team in advance about those errors and avoid meeting those conflicts in advance. This way, the team can move forward, avoiding any potential test blockers. 
  6. Monitor performance through ON-device automation. Monitoring will tell you the root cause of any issue. New changes can also harm the computer’s performance; On-device automation can helps us in monitoring the long term performance and detecting the bugs. 
  7. Simulate the trigger by giving commands. In simple words – tell the character to reach a particular spot “A” from another spot “Y”. Monitor the performance characteristics on ADB and also keep a checklist to run and check necessary events throughout the app. 

Tools to performance and analyzing your VR Apps

OVR Metrics tools 

  • It analyses the mobile application with real-time graphic overlay. It provides data on heat, frame rates, GPU/CPU throttling values.

Logcat 

  • It collects VrApi logs. One can collect system logs with its command line. The Quality assurance team can take a close look at the app performance and device setting. 

Ovrgupprofiler

  • It accesses real-time data of GPU pipeline metrics and provides stage tracing with the CLI tool. 

GPUsystrace 

  • It includes information about the timing data of the steps of rendering pipelines as well as the information on the rendered stages. 

RenderDoc 

  • It is a tool that is the best solution for capturing frames and analysis. It is a graphic debugger tool. It is compatible with multiple graphical APIs and development platforms.

Unity Profiler 

  • It records various aspects of the app’s performance. It can store 300 frames of any app and shows you detailed information about every frame. One can explore the script code also. 

Final words 

It is up to the Quality assurance team to pick the best tools for testing. However, some of the testing tools can accelerate the process of testing and debugging. One must learn to run few automation scripts on the available device to test. Once the test is done, one can start justifying more hardware. Consider bandwidth after testing; your team need to analyze and report based on these findings. Begin with the unit testing as it is the fundamental method and then consider the further stages of software testing. As VR apps and their testing are considered in their infant stage, we hope that more tools will be developed to automate and accelerate the entire process. 

Leave a Comment