Early days of AI: Only 25% of businesses have posted, few reap rewards

2025 is expected to be the year AI becomes a reality, bringing real, tangible benefits to business.

However, according to the new State of AI Development Report from the AI ​​Development platform Vellumwe’re not there yet: Only 25% of businesses have deployed AI for production, and a third of those haven’t seen the potential.

This seems to indicate that many businesses have yet to find work use the AIskeeping them (at least for now) in a pre-built holding system.

“This proves that it’s still early days, although there are some jokes and discussions going on,” Akash Sharma, CEO of Vellum, told VentureBeat. “There is a lot of noise in the industry, new models and suppliers are coming, new ways of RAG; we just want to find a place where the industry is working AI design.”

Businesses need to identify the real-world situations that are used to make it happen

Vellum interviewed more than 1,250 AI designers and developers to find out what’s really happening in AI circles.

According to reports, most of the manufacturing companies are still in their various stages AI trips – Developing and evaluating strategies and proofs of concept (PoC) (53%) beta testing (14%) and, at the lowest level, talking to users and gathering requirements (7.9%).

So far, businesses have focused on developing analytics and analytics tools and chatbots for customer support, according to Vellum. But he is also interested in applications that combine analytics and natural languages, content extraction, production systems, code generation and automation and research itself.

Meanwhile, developers have mentioned the competitive advantage (31.6%), money and time saving (27.1%) and the number of users (12.6%) as the biggest challenges they have seen so far. Interestingly, however, 24.2% have never seen any profit from their products.

Sharma emphasized the importance of prioritizing use cases from the beginning. “We’ve already heard from people that they just want to use AI for the sake of using AI,” he said. “There is an experimental budget associated with this.”

While that makes Wall Street and investors happy, it doesn’t mean AI is delivering anything, he said. “The thing that everyone should think about, is, ‘How do we find the right ways to use it? Often, when companies can identify what they use, they make it and see a clear ROI, they go forward, they go beyond the hype. This leads to internal expertise, more money.”

OpenAI is still at the top, but model mixing will be the future

When it comes to the models used, OpenAI it keeps leading (no surprises there), especially its GPT 4o and GPT 4o-mini. But Sharma pointed out that 2024 offered more opportunities, either directly from model developers or through platform solutions like Azure or AWS Bedrock. And, providers with open source versions like Llama 3.2 70B are gaining traction, too – like Groq, Fireworks AI and Together AI.

“Open source models are thriving,” Sharma said. “OpenAI’s competitors are making progress.”

Ultimately, businesses won’t just have one brand and that’s it — they’ll lean more toward multi-brand systems, he said.

“People will choose the best model for whatever job they have,” Sharma said. “When you build an agent, you can have several guidelines, and in any case the developer will want to find the best, the cheapest and the lowest, and it may not come from OpenAI.”

Similarly, a the future of AI It is undoubtedly multimodal, and Vellum is seeing an increase in the adoption of tools that can perform different tasks. Text is the most used, followed by creating files (PDF or Word) images, audio and video.

Also, retrieval-augmented generation (RAG) is the way to go when it comes to data acquisition, and more than half of developers are using vector databases for easy search. Top open source and trusted brands include Pinecone, MongoDB, Quadrant, Elastic Search, PG vector, Weaviate and Chroma.

Everyone is involved (not just engineers)

Interestingly, AI is continuing to outpace IT and democratize all businesses (like the old ‘takes a village’). Vellum found that although engineering was involved in the majority of AI projects (82.3%), they are joined by leadership and management (60.8%), subject matter experts (57.5%), sales teams (55.4%) and production departments (38.2%).

This is largely due to the ease of use of AI (and the excitement surrounding it), Sharma noted.

“This is the first time I’ve seen programs being developed very skillfully, especially because the languages ​​can be written in natural language,” he said. “Traditional programs are often positive. This is not optional, which brings more people to development.”

However, businesses continue to face significant challenges – particularly around AI concepts and incentives; model speed and performance; data access and security; and reception by stakeholders.

At the same time, while more non-technical users are participating, there is still a lot of technical expertise in-house, Sharma said. “The process of connecting all the different moving parts is still a skill that most manufacturers don’t have today,” he said. “So it’s a common problem.”

However, many of the existing challenges can be overcome by using tools, or platforms and services that help developers test complex AI systems, Sharma noted. Developers can use tools internally or with third-party platforms; however, Vellum found that about 18% of designers are interpreting the information and ideas of the orchestra without any tools.

Sharma noted that “the lack of technology becomes easier when you have the right tools to guide you on the journey of development.” In addition to Vellum, frameworks and platforms that research partners include Langchain, Llama Index, Langfuse, CrewAI and Voiceflow.

Continuous monitoring and evaluation is essential

Another way to deal with common problems (including imagination) is to evaluate, or use specific measurements to test the correctness of the answer. “But despite that, (developers) are not doing the efforts they should,” Sharma said.

Especially when it comes to high-tech systems, businesses need analytical solutions, he said. AI agents have a strong element of non-determinism, Sharma said, in which they refer to external systems and act autonomously.

“People are trying to develop very sophisticated systems, therapeutic systems, and that requires a lot of test cases and some kind of testing process to make sure it works reliably,” Sharma said.

While some developers are using automated testing tools, A/B testing and open source testing methods, Vellum found that more than three-quarters are still doing manual testing and evaluation.

“Manual testing just takes time, right? And the sample size for manual testing is often much lower than what a single test would do,” Sharma said. “There can be difficulties in just knowing the way, how to know automatically, and on a large scale.”

Finally, he emphasized the importance of integrating integrated systems that work together – from the cloud to application programming interfaces (APIs). “Think of AI as just a tool in the arsenal and not a magic formula for everything,” he said.