With AI infrastructure in a new chapter, the focus is shifting. Constructing formidable models is only part of the equation; for millions of users, it is equally vital to run them in an expedient and cost-effective manner.
Why AI Inference Is Becoming the Next Big Infrastructure Race
IBM and Together AI have put their money where their mouth is on that front with a $240 million, multi-year pact. The deal will see the two companies put together a sizeable AI inference cluster on IBM Cloud, one that is driven by NVIDIA’s newest hardware.
A Push for Dedicated AI Inference
The arrangement makes for the first large-scale inference cluster of its kind on IBM Cloud to be built around HGX B300 systems. Together AI intends to put the cluster to work running open-source models for its enterprise clientele.
Inference is the crux of the matter. While training produces a model, inference is the act of generating an answer from it. With AI being adopted at an ever-faster pace, firms require the means to provide high-throughput responses with minimal latency and expense. The demand is evident in Together AI’s numbers: the company is now churning through some 400 trillion tokens a month.
NVIDIA’s Blackwell Ultra Powers the New Cluster
By Q1 2027, the new cluster should be up and running. It will be equipped with NVIDIA Spectrum-X Ethernet networking and the HGX B300, which is built on the Blackwell Ultra architecture. These are no ordinary systems; they are made for the rigours of AI workloads such as fine-tuning and training. A single unit can put out 144 petaflops of FP4 compute, supported by advanced interconnects and high-bandwidth memory. IBM has already put these at the disposal of a few cloud clients and has more in the pipeline.
Why IBM and Together AI Are Teaming Up
There is logic to the partnership. IBM is fortifying its standing in the neocloud and AI space, something that appeals to the regulated industries and enterprises that put a premium on security and hybrid-cloud options. As for Together AI, IBM Cloud is an enterprise-grade solution that allows the firm to grow its inference capacity without the outlay of constructing its own data centres.
Together AI’s Rapid Rise Signals Growing Inference Demand
Together AI has been on a tear. After closing an $800 million Series C round at an $8.3 billion valuation, the company has seen its token volume explode from 30 billion a month a year ago to over 400 trillion. It is a sign of how the industry is changing; companies are less concerned with just developing a model and more with the economics of running it at scale.
What the Deal Means for Enterprise AI
The agreement is good for all parties: a stronger position for IBM in specialised workloads, room to grow for Together AI and another installation for NVIDIA’s Blackwell Ultra. But for the enterprise, the real story is one of choice. Open-source alternatives like Llama or Mistral, available via Together AI’s platform, can afford a degree of customisation and cost control that closed models cannot.
As AI becomes part of the daily business routine, the contest is not merely to make the smartest model. It is to have the infrastructure in place to serve it reliably and at speed.
