How Databricks is using synthetic data to make it easier to evaluate AI agents

Businesses are in full swing AI assistants. They want these systems to think and act differently in different areas, but they are often hampered by the complexity and time-consuming process of evaluating these agents’ performance. xToday, a leader in the data ecosystem Databricks he announced The ability of data to make this easier for developers.

The move, according to the company, will allow manufacturers to create high-quality datasets within their workflows to assess performance. This will save them unnecessary back-and-forth with story experts and bringing in sponsors to produce.

Although it remains to be seen how the generated data will work for businesses’ using the Databricks Intelligence platform, the company led by Ali Ghodsi says that its internal tests have shown that it can improve the performance of the agent on various metrics.

Databricks’ role in evaluating AI agents

Databricks discovered MoseicML last year and has successfully combined the company’s skills with examples on its Data Intelligence platform to provide businesses with everything they need to build, deploy and evaluate machine learning (ML) and AI solutions using their company’s ocean of data.

Part of this work revolves around helping teams to develop dual AI systems that can not only think and respond accurately but also take actions such as opening/closing support tickets, answering emails and booking reservations. To that end, the company has revealed everything new features of Mosaic AI technology this yearincluding support for optimized framework models, a catalog of AI tools and offerings for building and evaluating AI agents – Mosaic AI Agent Framework and Agent Evaluation.

Today, the company is expanding Agent Monitoring with a new data generation API.

Currently, Agent Evaluation has provided businesses with two main strengths. The first one enables users and subject matter experts (SMEs) to manually interpret the data containing relevant questions and answers and create a color scheme for reading the answers provided by AI agents. The second one enables SMEs to use this model to identify the supplier and provide feedback (labels). This is supported by AI judges who automatically record the responses and responses of people in the table and evaluate the quality of the agent on measures such as accuracy and harm.

This method works, but the process of building analytical datasets is time-consuming. The reasons are easy to imagine: Regional experts are not always available; the process is manual and users often struggle to identify the most important questions and answers to provide ‘golden’ examples of success.

This is where the data processing API comes into play, enabling developers to create high-quality analytical datasets for preliminary analysis in minutes. It reduces the work of SMEs for final verification and accelerates the process of iterative development where the builders can also explore how the permutations of the system – to improve the model, to change the recovery or to add tools – to change the quality.

The company conducted internal tests to see how the data generated from the API could help evaluate and manage agents and found that it could lead to significant improvements in various metrics.

“We asked a researcher to use the data that was created to test and improve the agent’s performance and analyze the results using the data stored by people,” Eric Peter, AI platform and product leader at Databricks, told VentureBeat. “Results have shown that for various metrics, the agent’s performance has increased significantly. For example, we have seen an increase of almost 2X in the agent’s ability to find relevant documents (as measured by recall@10). In addition, we have seen an improvement in the accuracy of the agent’s answers.”

How does it taste?

Where available many tools which can create structured datasets for visualization, Databricks’ offering stands out with its tight integration with Mosaic AI Agentic Evaluation – meaning developers building on the company’s platform don’t have to leave their jobs.

Peter said that creating data with the new API is a four-step process. Devs only need to analyze their documents (save them as a Delta Table in their water house), pass the Delta Table to the generated data API, run an analysis with the generated data and see the positive results.

In contrast, the use of an external tool can mean a number of additional processes, including speed (output, change and load (The cost of ETL) move the modified documents to an external location that can manage the data processing process; moving generated data back to the Databricks platform; and then convert it to Agent Evaluation. Only this measure can be done.

“We knew companies needed a turnkey API that was easy to use – one line of code to create data,” Peter explained. “We also noticed that most of the solutions in the market were providing simple opening instructions that were not designed to be good. With this in mind, we made a big investment in the quality of the data that was created while allowing the driver manufacturers to manage their business needs using a real-time interface. Finally, we knew that many existing offerings had to be ported to existing workflows, adding unnecessary complexity to the project.Instead, we created an SDK that was tightly integrated with the Databricks Data Intelligence Platform and Mosaic AI capabilities Agent Evaluation.

A number of enterprises using Databricks are already taking advantage of the production data API as part of a private preview, and are showing a significant reduction in the time taken to improve the skills of their agents and send them into production.

One of these customers, Chris Nishnick, director of Artificial Intelligence at Lippertsaid their teams were able to use API data to improve relative model quality by 60%, even before they had experts.

Increased capacity of pipeline agents

As a next step, the company plans to expand the Mosaic AI Agent Analysis with features that help domain experts to improve the accuracy of the product and its lifecycle management tools.

“In our previews, we’ve learned that customers want more,” Peter said. “First, they want a user interface so that their experts can review and update their analysis. Second, they want a way to manage and manage the lifecycle of their analysis so that they can track changes and make changes from the experts’ review of the type of data that is immediately available to developers. To solve these problems , we are already testing several products with customers that we want to launch early next year.”

In general, these developments are expected to advance the implementation of databrick’s Moseic AI offering, further strengthening the company’s position as a supplier of all things gen AI.

But Snowflake is also working in this category and has announced several products, including a an example of cooperation with Anthropicto him Cortex AI a product that allows businesses to develop gen AI software. Earlier this year, Snowflake also got its first exposure TruEra to provide the power to analyze AI functions within Cortex.