Meeting the growing global demand for AI chatbots and image generators depends on both mundane components and flashy GPUs. Server cabinets, heavy-duty metal cases for storing GPU systems, have sometimes been a crucial bottleneck.
While rushing to build its facility, CoreWeave once ordered the wrong 1,400 cabinets. It was a costly mistake because supply chain backups have delayed new shipments by months. “You’re moving very fast, and at some point a process fails, and you don’t realize until you have 17 trailers with cabinets outside the door and you have to turn them all away,” Venturo says.
But in an example of the shrewdness that has been key to rapid expansion, Venturo’s team in that crisis quickly put frustration aside and decided to buy used cabinets in what he called the gray market. The move avoided a significant delay. “This was just one instance of challenges we faced and overcame to ensure we delivered for our partners,” he says.
To keep things moving, CoreWeave has turned to the “gray market” for more than just cabinets. He has purchased network switches and routers on eBay to avoid waits of up to two years for new equipment, a former employee says. The safety and reliability of used parts may be questionable, but amid the urgency of the AI boom, some conventional practices had to be put aside, the person says.
In Plano, CoreWeave last year equipped four 1-megawatt data center rooms in less than three days each, a feat that typically takes weeks. “We can start without gloves and build very quickly,” says Venturo.
CoreWeave was forced to adopt another creative solution when an Internet provider was slow to install a broadband connection at a new site, a problem familiar to many home Internet users. Three top executives met by phone one morning to decide how to avoid delaying the project. They all offered the same solution: buy satellite Internet through Elon Musk’s SpaceX’s convenient but not cheap Starlink service until the fiber provider came along. Eliminated weeks of potential delays. “We have to be incredibly flexible,” Venturo says.
Helter Skelter
Lessons from early projects inform CoreWeave standard procedures today. CoreWeave has chosen to pay a premium for custom manufacturing tens of thousands of fiber optic cables because their special design means installation takes just one hour, instead of 10. After U.S. customs authorities held up a vital shipment of important equipment, CoreWeave began immediately. ship orders through multiple alternative ports. To avoid shortages, the company now orders many more parts than it needs, betting that the leftovers can be sent to the next project that comes along.
Haste has sometimes had unintended consequences. CoreWeave’s data center in Las Vegas still smells like burning plastic, one source says, because it blew up some electrical components when it turned on too many GPUs at once when the site was set up several years ago.
At the center of CoreWeave’s operations are its data center technicians. The most skilled operate like a special operations unit, flying from site to site to activate new data centers rather than working on a campus full time. Venturo declines to say how many miles his team of top-notch technicians have covered, but says they installed about 6,000 miles of fiber optic cabling last year. “I probably have the most interaction with that team, more than any other team in the company, simply because they are incredibly important,” he says.