Unity builds a game developer AI assistant with Azure OpenAI Service

More creation for all

Video games have captured the imagination and attention of people around the world, and Unity is at the forefront of making video game development accessible to more developers. The company makes a 3D game, application, and experience development platform known as the Unity Engine, which has been lauded for its approachability and power. Unity is widely used by creators around the globe, covering everything from small independent games to major studio titles. It’s flexible and accessible for building simpler 2D adventures, immersive VR experiences, and everything in between.

Developing software, especially games, is a difficult task. While Unity is approachable and reliable for game developers new to the field as well as seasoned veterans, it has extensive documentation and a great deal of functionality built in. While this broad capability means that the platform can handle a variety of projects, it also means that there are numerous potential interactions between moving parts that might not always work as intended.

Documentation alone can’t always diagnose the problem. “We have a vast amount of information and a vast number of subject matter experts who share their experience on our community forums,” explains Martina Johannesson, Senior Technical Product Manager (AI & Machine Learning) at Unity. “But that’s a lot of information to sift through if you just want to solve a problem.” And online advice is not necessarily tailored to the specific problems that a developer may encounter. Unity needed new tools to help developers understand and troubleshoot problems with reliable advice.

Building chat for Unity Muse

To help developers find answers faster, the company used Microsoft Azure Open AI Service to build Muse Chat, an AI assistant designed specifically to answer questions and provide coding solutions. Muse Chat is an essential part of the Unity Muse AI product suite that enables generative AI assistance and capabilities in the Unity Editor.

Even before Unity started building Unity Muse, the company launched a partnership with Microsoft Azure to advance its cloud infrastructure and to better meet the needs of developers. Additionally, the wider partnership with Microsoft enables ‘Made with Unity’ game creators to more easily reach players across Windows and Xbox devices.

“Azure OpenAI Service provides such a wide variety of features that you can pick and choose and deploy very freely,” says Srikanth Cherla, Senior Machine Learning Developer at Unity. “And when it comes time for users to interact with it, you can trust it will run smoothly without interruption.” The team ultimately used both GPT-4 and GPT-3.5 Turbo models as the foundation for Muse Chat, striking a balance between speed of responses and level of detail.

Unity had three main goals when building Muse Chat. The first was relevance, so that users could consistently use this AI assistant to get answers tailored to their specific issues. The second was speed of finding answers. Since game development is a fast-moving process that often requires rapid iteration, being able to find the right information to make quick and effective updates is important. Reliability was the third major goal, as Muse Chat needed to not just give the right answers quickly but do so consistently, even as product updates hit.

“The idea was that our users could be more productive if they could get information easily and quickly,” explains Alejandro Rodriguez Salamanca, Machine Learning Engineer at Unity. “It helps creators in their creative process.” Muse Chat uses Azure OpenAI Service content-filtering models provided by Azure AI Content Safety, which are built in to ensure responsible use of the software. Game developers need to be able to use the software to provide quick answers and support, but not at the cost of opening security loopholes.

Integrated guidance for ongoing development

Unity designed Muse Chat to be used throughout development and incorporated directly into the project-based structure of the Unity Engine. The AI at the heart of Muse Chat has access to not only the full library of technical documentation around Unity but also the specific details of how each project has been set up.

While Muse Chat does act as a chatbot, it doesn’t have users simply type in a prompt and then spit out a generic answer that may or may not resolve the issue; developers can use Muse Chat to directly analyze parts of their game that aren’t working correctly.

“If you try to apply a shader to your objects and it's not working as hoped, instead of having to dig through documentation and comb through information that’s not relevant to your project settings, you can simply ask Muse Chat ‘Why is this shader not working in my build?’” says Johannesson. “Muse Chat will understand which of your project settings are conflicting and can help solve for how the shader can be properly applied.”

Developers don’t have to search for a specific error code but can instead troubleshoot directly on their own with the specific framework of the project. This benefits development as every piece of information is crafted to answer specific problems based on the actual interaction between objects in the code. This frees Unity developers to spend less time hunting down obscure problems and more time thinking about larger design issues. It also means that developers can be confident in trying potentially odd or unorthodox methods to accomplish goals, knowing that Muse Chat can help them track issues down quickly.

Seeing creativity improve

Before Muse Chat was accessible to everyone, Unity gathered feedback from developers about features that would be necessary or helpful. The early version, which was available as a web tool, allowed for extensive early testing beyond what might otherwise be possible.

When Unity Muse was released, Unity was happy to hear feedback from users that Muse Chat was helpful for troubleshooting and finding Unity-related information. Not only was the tool useful for developers, but the scalability of Azure meant that it maintained consistent reliability. Creators were enthusiastic about getting reliable information and feedback to fix problems instead of diverting time to scour documentation and search for obscure parts of code.

The functionality of Muse Chat also helps set Unity apart, as it’s one of the only 3D content creation engines which has a built-in large language model that can access the details of users’ projects. While several chatbots are available that can digest coding documentation and offer generic code fragments, Muse Chat can examine specific code for errors and suggest improvements right within their Unity project. Developers are keenly aware of how much time they can save by doing things in a few clicks instead of days.

Creating a brighter future

While Muse Chat is already able to provide direct feedback on observed problems, in the future Unity is working hard to enable automated workflows as the next phase for Muse Chat in the Editor. When users request broad edits or updates, in the future Muse Chat will automatically be able to apply updates on the user’s behalf.

Although Muse Chat is still early in its rollout, Unity believes it is a major opportunity for game developers to solve problems in new ways. “Game development can be a complex art form, and many creators may not want to spend hours refining component setups in their games and rather spend their time elsewhere,” says Johannesson. “They’re more interested in the overall game flow and experience. The easier it is to focus on that, the better.” As the team gets more feedback on how Muse Chat can provide more functionality, it can be iterated upon and improved.

Unity believes that using Azure OpenAI Service pushes development further and makes game development more straightforward, ensuring that developers have access to guidance that can lead to better games across the board. As Johannesson sees it, this will help the entire industry become even more creative. “We think this is going to help unleash even more creative experiences into the world. And that makes us really excited about the future.”

 Find out more about Unity on Twitter, Facebook, and LinkedIn.