I had the great opportunity to speak at AutomationStar 2024. If you don't know AutomationStar, this is one of Europe's biggest test automation conferences.
Most of the AutomationStar crowd is focused on automation for web and mobile testing. That's why my talk focuses on what sets video games appart from web and mobile apps, but also what we can learn from them.
Later I streamed the presentation on LinkedIn live, and you can find the recording below. It's a 30 minutes presentation and the rest is Q&A.
If the video elicits any question on your side, feel free to get in touch!
Here are the questions posted during the live stream, and their answers:
Isn't e.g. Android Vs iOS vs Web a fragmentation that happens to non-games?
For software as much as for hardware.
Yes there is fragmentation and that's why we've seen so many apps built on webviews. But there is enough standards for mobile apps that you can test your app with Appium. It covers native and webviews apps.
Video games suffer from that too, but on top of that there are more platforms (desktop and consoles), and the tools don't cover everything. It really restrict your choices.
How
do you determine if something in a game is worthwhile to automate
testing for it?
Some aspects of games are bespoke one-offs that may not
be repeated.
It is part of your testing strategy. If something will be repeatedly tested, either by QA or by the developers themselves, an automated test might be worth it.
But it depends on its complexity and if an issue for such feature is a big problem for the players.
Which automation tool is best for unity games?
I don't think there is a definite answer, it depends on your team, your skills, and what you're trying to do. I can recommand reading the comparison from an earlier blog post.
Can you talk a bit more about the disatvantage of using Python compared to C# in a Unity game?
It's not so much about Python but rather about the way the tools instruments the game. Since Python is non-native to any major game engines, running tests in Python requires to instrument the game with a network layer. The game receives commands while the tests send the commands (like Selenium or Appium does). So again, it's not specific to Python.
However it adds a layer of complexity. This can get in the way if you want to access the internals of the game. I can give an example regarding June's Journey. The objects to find in a level, won't be reflected in Unity's hierarchy. So you need to have access to the internals of the game to know which object should be clicked on. With a tool like AltTester, I would have to write my own command to achieve that, and serialize data back to the testing code.
So it depends on what you want to do.
I can recommand reading the section titled "Embedded vs Networked architectures and why does it matter?" in an earlier blog post.
What impact do you think AI will have on test automation in games in short to longer term?
It's hard to assess. I don't see AI being a huge game changer for the web for now, and the game industry is not even at this level in term of tooling and knowledge.
For me this is like trying to run before learning how to walk. Most studios don't know what are the pitfalls and the best practices for automation, so I don't expected AI to improve things here. We're still in a hype cycle and a lot of people talk about it, but I'm waiting to see some concrete applications.
You mentioned a lack of standardisation, how do you think this problem could be corrected?
I don't think the problem can be or should be corrected. Each game is different even if they're built with the same game engine. I believe a better approach is to have advanced tools fitting each game engine.
Following up on being close to the dev teams and using Unity Test Framework, this is similar to how I work. Would you have also invested in having tests on device, for instance via AltTester, or is that redundant?
In my case, the most important issues my team was tackling were not hardware specific. So we didn't find it relevant to invest into testing on real devices.
I think Unity (in my case) does a good job at being cross platform, so we could test the functionalities of the game without worrying too much about that.
Each situation will be different though. And if the need is here, I think it can be worth having a couple of tests focusing on hardware problems, then AltTester would be a good choice IMO.
What’s your experience with off the shelf tools like Game Driver?
I don't have experience with GameDriver, I only know their selling points. I think their querying language is interesting, but I didn't found querying objects to be huge a pain point in my experience.
How do you personally deal with the Pesticide Paradox in gaming automated testing?
So I never heard of the Pesticide Paradox before, but I found what I think is a good description of it: The Pesticide Paradox in Software Testing.
I don't have a better answer than the article, but I believe test automation in video games cannot replace manual testing entirely. A lot of things won't be automated because they're subjective.