My Dream "home lab"

And so, each of these is 1,000 times more powerful than say a 5090 RTX, right? And there's eight in here. Yes. Okay, you got to show me this. Now, this is the type of home lab that I'd love to have. We've got GPUs, switches, fiber, storage, and a whole bunch more. The only problem is that it costs about 20 million US dollars, and I'd probably need a power plant just to run it. I went inside Cisco's AI lab to see what powers modern AI and how to optimize AI clusters, because you can't train large-scale AI models with just a single GPU. You need thousands or hundreds of thousands of GPUs, and they need to work together connected via some kind of high-speed, low-latency network. I asked Rakesh and his team to show me around this amazing lab and also explain some of the components that they are using to optimize for AI clusters. Rakesh, you've just shown me your amazing lab. How much does that cost? It's close to 20 million dollars we already spent on this lab. So, this is not really cheap equipment, to be honest with you. The most surprising number wasn't the cost of the lab, it's the cost of a small failure or even a delay. Even if just some packets go missing, it can cost a lot of money. The GPU cost is like 2 dollars per hour per GPU. And if you rent a 10,000 GPU cluster, that costs 175 million dollars to rent per year. And imagine that if you have 5% efficiency loss, that's like 8 million dollars per year. So, that's a very big thing that can make or break some of these businesses. So that's the reason the job we do is very important. A bad cable, a hot optic, a congested link, packet loss, even a tiny delay at this scale, even a small problem can hurt performance. Rakesh gave me an example of just that situation. I'll tell you, sometimes the kind of challenge we have seen is very interesting. So one time, when we build our clusters, sometimes some cable goes bad, optics goes bad, and sometimes people, they don't necessarily know, because over time the network becomes large. So let's say somebody finds whatever the cable size is there, and they plug that wrong cable size, and it took us almost 2 days to figure it out, because performance was dropping almost 75%. Can you imagine that? Now, that one example is why this lab exists, because modern AI is not just about buying the fastest GPUs. It's about making sure thousands of GPUs can communicate together. This relies on the servers, the NICs, the switches, the optics, the storage, and the software controlling the system to work together to optimize performance. So before we can even talk about 100 terabits per second switches and the future of AI clusters, we need to understand the basic problem. How do you connect GPUs together at this kind of scale? Inside a single server, GPUs can talk to each other using extremely fast internal connections. That's called scaling up. But even a single rack of GPUs is nowhere near enough in today's world. You need to scale out, which means you need multiple racks running multiple GPUs within each rack. They all need to work in unison to train AI models. But even that's not enough these days. Now, we have to connect multiple data centers together in what's called scale across. In scale out, we're connecting many GPUs together across a network fabric in a data center, and that network starts to behave like part of the computer itself. Ramesh showed me the H200 compute tray in the lab. He also showed me how that works inside an H200 server. So, this is only a single server, right? With eight GPUs in. But surely if you're training some big model, you're going to have a whole bunch of these. Yes. So, this one is actually not the whole server. This is just the compute tray. There are other components to this which you don't see here. There are the NICs which are actually helping the GPUs go out and talk to other servers that are part of the cluster, which is the scale-out fabric that you might have heard that term. So the one within the server is connected by something called NVLink, which is a scale-up domain. And these are extremely fast connections, and they help the GPUs talk and exchange gradients when they are training. Now, that's the key idea. Inside the server, NVLink helps the GPUs talk to each other, but outside the server, the network fabric connects many GPU servers together. And once you scale up to thousands of GPUs, the network becomes one of the most critical parts of an AI system. The speeds that they have these days is insane. We have 100 terabit per second switches, and each port is 1.6 terabits per second. These GPUs need huge bandwidth to communicate together. The big concept here is you could have hundreds or thousands or hundreds of thousands of GPUs in an AI training cluster all working as one system. And that's why we need 100 terabits per second switches and ports that are 1.6 terabits per second each. Once you understand that kind of requirement, the hardware in this lab doesn't look like overkill, because these switches are not just moving internet traffic. They're connecting GPUs to other GPUs in a way that keeps the AI jobs running. They need to be fast enough for these AI jobs to continue running without packet loss and without delay. That's why Cisco's building 100 terabits per second switches, 1.6 terabits per second interfaces, new Ethernet fabrics, and technology designed specifically for AI clusters. I asked Will what's changed since I last spoke to him at Cisco Live, and what's happening at NVIDIA GTC, and where he sees data centers going next. So, we saw each other in Amsterdam in February at Cisco Live. I wanted to ask you, because there's a whole bunch of stuff been happening. So perhaps you could just recap what we spoke about at Cisco Live, and then give us a rundown, because GTC's happened. There's a whole bunch of stuff that's happened since we spoke. Yeah, so the biggest announcement at Cisco Live Amsterdam was announcing the G300, which is the new 100 terabit switch. We were also just coming off having announced at the GTC DC things like the Spectrum 4 base switch from Cisco, where we use Nvidia silicon and put it in our switches, parallel to switches where their own silicon. We also announced the P200 family, which is relative for these more routing functions and scaling across. So there are really two problems here. Inside the data center, you need scale out. GPU servers talking to other GPU servers across the fabric, and that's where the G-series switches come in, including the G300, which are 100 terabits per second switches and 64 ports each running at 1.6 terabits per second. But that's once again not enough. Scale up is not enough. Scale out is not enough. So we have to do scale across, where we combine multiple data centers together. So we are connecting AI workloads across multiple data centers, and that's where the P200 switch is used. It has routing capabilities, deep buffers, larger tables, security, and the ability to connect these AI factories together across very high-speed, long-distance links. To some degree, routing has been there for years. We love our BGP, Border Gateway Protocol. But what is new in this concept of scale across is that you might want to run a job across data centers and have that really work, where the GPUs on the east-west back-end network have to talk across data centers. That's new. I still don't see too many customers doing it, but a lot are starting to investigate it. And so that's where we've positioned this P200, which not only is very good for routing (deep buffers, larger tables, they can hold large IP prefix tables and ACLs and all those kind of things), but also supports running a job across. So that's the P200 architecture. And then the G-series with G300 is really that leaf-spine core of the data center family for us. And that's the 64 times 1.6 terabits per second interfaces on the switch. And then we announced we've been in development on the Spectrum 6, which is the 100 terabit switch from Nvidia. And that is going to be orderable soon. That's something by, let's say, late summer that we expect to be shipping. So these cycles, it's just amazing how quickly they condense, how quickly we have to go from first touching a chip to being able to ship it. But the customers need these performances. And so we are very happy to announce the progress on that Nvidia partnership side with their silicon. So it's important to note that this is not just theory. These designs matter for companies building AI infrastructure right now, especially neo clouds and GPU as a service providers. They need huge GPU clusters, but often only have small teams. So the infrastructure has to be repeatable, manageable, and faster to deploy. Will gave me a real world example of a neo cloud in Australia. So Sharon AI is a newly funded neo cloud in Australia who has ramped up funding and strategy very quickly. And we've been very excited to partner with them from day one. So they joined me for a talk at GTC about architectures and strategies. And some of the things that we hit together were everything from operationally, they formed like many of these neo clouds very quickly. Small teams. So they are leveraging hyper fabric, which is a new technology we have to really simplify and enable small teams to manage large clusters. So they're using hyper fabric. So that's been interesting to see. They also are a customer where they're using Cisco Silicon One on the front end. But what they are looking for is that Spectrum switch on the back end, because they're looking to be a full NVIDIA Cloud Partner Program. And so that is again in this model of common operational, but being able to support them on either Cisco Silicon One or Nexus in order to support their end goals. We've also seen that putting together the full software stack, so they're using our full reference architecture that includes the VAST Data, all those pieces, and that has enabled them to move very quickly from concept, which feels like it was just the end of 2025. They were like, okay, the funding is almost here. Like, we're going to have to move fast. And then really plotting week by week when equipment arrives and how they go from arrival to deployment, and then getting this in the hands of their end customers. So I'm always amazed how quickly the Neo Cloud industry moves. It's very exciting to see. That example matters because it shows why this is not just about fast switches. Now I also spoke to Richard, who explained this in more detail. Speed alone is not enough. These switches also need the right silicon, the right operating system, the right optics, and the right management layer, so customers can operate these AI clusters in the real world. This is actually one of our switches that we position for AI networking. And what we're going to show you here today is talk about the internals of that switching complex and really what drives this communication within the AI cluster that we've already been talking about. Now that phrase is important, switching complex, because this is not just a fast box with a lot of parts. For AI networking, Cisco's thinking about the whole system: the silicon, the operating system, the optics, the automation, and the management layer. That matters because the switch has to move massive amounts of GPU to GPU traffic. But it also has to deal with congestion, latency, jitter, quality of service, visibility, and troubleshooting when something goes wrong. We start out here with the silicon. This is the foundational piece. That's the component here that really runs everything, and it's the heart. It's really the main vital organism that's inside the switch itself. And this is where you're going to see things like the packet forwarding, programmable pipelines. But before that works, I need to make sure I have the right type of software to implement the features for this AI networking kind of paradigm that we're seeing for solutions. So there's things like congestion mechanisms, whether it's reactive or proactive. What I need to make sure is, how can I have the operating system take advantage of what's in the custom silicon? So this isn't like the CPU in your laptop. This is a switching ASIC, custom silicon built to move massive amounts of traffic through the network. And in an AI cluster, that matters because the traffic is not like normal office traffic. These GPUs are constantly exchanging huge amounts of data across the fabric, and they cannot have packet loss, because if packets are lost, the entire training run has to stop and go back to a point where they've taken a snapshot, as an example, and continue. The operating system is very, very key, because when you start handling tons of these AI networking unique traffic loads, this is where you need to configure. And David, you probably remember quality of service, QoS. How many lines of code do you think that is? For a typical box, it could be anywhere from 50, 60. You mean in the CLI? On the CLI. So what's important about that is, imagine you're running your engineers, you have one box, it could be 60 lines of QoS config only. Now I multiply this. We saw earlier 16 scalable units. I could have more leaves. How can I get that to be fabric wide? I can do all that with unified management here with the Nexus dashboard. So what's single, it's consistent, it's an API first mentality. I can have different workflows to help guide me to those principles. That is an operational problem. One switch is manageable, but an AI fabric can have many switches, many links, many optics, and thousands of GPUs depending on the deployment. You don't want engineers manually typing configuration across the whole fabric and hoping every box is consistently configured. You need automation. And more importantly, when something breaks, you need to find the issue as quickly as possible. But what if something breaks? I need to see different jobs, scheduling, latency, and this is where I can find that needle in the haystack. That's why I need something that I don't want a fire hose of information. I want something that's pinpointed. So when there's an anomaly, an advisory, can I correlate that information to something that I can understand and relate to? This is where unified management comes into play. So I can take all this information, I can expose information that's in the silicon, whether it's micro burst, whether it's buffering, whether it's some kind of congestion, programmable pipeline capabilities. We have that ability with our unified management to really configure that. Because AI is giving you tons of unique workloads right now. And we're seeing that from the front end, converged front end to storage, the back end, customers want specific load balancing strategies and techniques. That connects us directly back to Rakesh's cable story. At this scale, the problem is not just that something can go wrong. The problem is finding exactly where it went wrong. Is it a congestion issue? Is it a buffering problem? A bad optic, a misconfiguration, a link problem, a workload pattern perhaps? There are so many things that can go wrong. And in an AI data center, every bit of downtime can cost huge amounts of money. That's why management and visibility matter as much as raw switch speed. Richard then brought up something that I think most people underestimate, the optics. Because when you're connecting thousands of GPUs together, the optics can become really expensive. They can actually be the most expensive part of the entire AI network. Now, you didn't mention the optics. Good question, because, believe it or not, one of the most expensive line items on the bill of materials in a heavy quantity are the optics. Because there's so many of them, right? There's so many of them, and that's something customers do take for granted, because you want quality in these optics. You need integrity. You want investment protection. Sometimes you need something that's forward and backward compatible. So we give you two different flavors, whether it's OSFP or QSFP. So you want to make sure this entire system is going to be very, very low power, but you want to make sure you're giving customers enough of a radix, a high radix to flatten that level of network, the number of tiers. Because remember with AI networking, job completion time, latency, jitter, those are all very important factors that can really destabilize your jobs. Imagine you have a job that's scheduled for 14 days. It fails in day 13. You're looking at some of the coolest 800 gig optics. This one particular, this one is OSFP form factor, and it's the integrated heat sink. But we also have the same LPO optic. So, LPO optics, we technically remove the DSP or digital signal processing out of the optic and will make it more efficient for unique use cases if you have connections between your switch and your GPUs. Those are very popular because you save a lot on the power. Now, that's brutal. A training job failing on day 13 can mean days of wasted time, huge GPU costs, and engineers trying to figure out whether the problem was the workload, the network, the optics, the storage, or the configuration. And that brings us back to why Cisco built this lab, because at this scale, you can't just trust a spec sheet. You need to test the whole system, the GPUs, the NICs, the optics, the switches, the storage, the software, the automation, and the actual AI workloads that customers are going to run. And that's what Rakesh's team is doing. So, what we do in our lab is previewing all the equipment. I'll walk you through. I'll take you to my lab and I'll walk you through everything. We bring all this equipment, we bring all the components, and we also work very closely with the storage partner, because this AI infrastructure is a very, very big, complex piece of beast. It has all the components. So part of my job and part of what Cisco does is, how do I make sure my customer, when they deploy a Cisco solution, it will work and perform very well. So we put all this together, and we make sure we test. We do a lot of different kinds of performance benchmarking. We test the networking layer, the RDMA layer. We look at everything from the RDMA perspective. When we look at the network perspective, as you scale these clusters to tens of thousands of GPUs, the network plays a very, very critical role. If you're not using the latest and greatest technology like load balancing techniques, then even though you have network capacity, your inefficiencies can lead to suboptimal performance. Then we also look at the storage. We talk to the storage partner. We bring all these products. We put everything together. We want to make sure when customers deploy our validated design, it works, it works very well, and we give them all that visibility, performance benchmark numbers. It's not just talk. We give them all the numbers, and we work with our partners AMD and NVIDIA to make sure that we meet the performance requirements. And sometimes the bottleneck is not where you'd expect it to be. It's not always the GPU. It's not always the switch. In one test, Rakesh's team found the problem was actually storage. So, one time we were training the DLRM model, and our performance was very, very poor. So, when we analyzed that, there are a lot of tools available so that we can analyze and profile, and we found out the storage was the bottleneck. So we sat with our partner. We spent almost a couple weeks fine-tuning that. And once we did that, you won't believe that we were able to do even better than what the competition has shown on the DLRM. So, that's the point we are trying to make. This is very important, the kind of work we do, because our customers can benefit from the work we do in the lab. But that raises another question. If Cisco only has a 128 GPU cluster in the lab, how do they prove that these designs can scale to tens of thousands of GPUs? Rakesh explained that they use 128 GPUs as a scalable unit, and then use RDMA testing and a 512 node CPU cluster to simulate tens of thousands of flows. We have the AI cluster. Like I was telling earlier, we have 16 H100 nodes. Each one is eight GPU. That's 128 GPU. And 128 GPU is used as a scalable unit. So what we do is we work very closely with our partner Nvidia. They have a very nice performance benchmarking document. They specify all the test tools you have to run and all the KPI matrix you have to collect, and they also give the numbers. So if you say that, hey, my scalable unit, which is 128 GPU, it performs very well as per Nvidia's guideline. We have to not only follow the test methodology, but the numbers. We have to beat those numbers. And once we test that scalable unit, you can create very, very large clusters, hundreds of thousands of GPU. Because of our partnership with Nvidia, we're doing a Spectrum-X solution that allows us to horizontally scale their scalable units. There's one more thing I want to tell you. Even though we don't have such a large cluster, we have a very large number of CPU nodes, our CPU cluster, that's like 512. So what we do is we use RDMA and we have written our own home-grown tool where we simulate tens of thousands of flows and we understand how these flows work, how they get load balanced, what congestion. So these are the methodologies, and this is a scientific approach, that even though I have a small 128 GPU cluster working, but we can mathematically and scientifically prove to you that the solution we Cisco proposing, it works when deploying tens of thousands of GPU scale. So what Spectrum-X does is, when the traffic is run from a GPU, how does the leaf make sure that I'm equally distributing traffic to all the spine? If you do not equally distribute that, some link would be over utilized, some link would be under utilized, and that can cause artificial congestion. And the reason I'm saying artificial congestion is, you have enough bandwidth, but you're not efficiently using the resource of the bandwidth. That is very important, and that's what we are working very closely with Nvidia on, packet spray and Spectrum-X, we call the solution, where we make sure that all the traffic is equally distributed. We calculate a Jain's fairness index, which is very important. So when we train this model, when we run any workload, we compute all the metrics. We calculate how much traffic is distributed across the links. So if your Jain's fairness index is one, that means you have 100% perfect distribution. So, Cisco is not just saying here's a very quick switch, a fast switch. They're testing a scalable unit, simulating massive traffic patterns, measuring congestion and load balancing, and using that data to prove how the design behaves at a much larger scale. That also leads to one of the biggest questions in AI networking. If you're using RDMA and building massive GPU clusters, should you use InfiniBand or Ethernet? So, are you running InfiniBand or Ethernet or both in your lab? No, very good question. So my lab is only Ethernet. Rakesh gave me the lab answer. Cisco's AI lab uses Ethernet, but I wanted the bigger industry answer, so I asked Will directly. InfiniBand or Ethernet? What's winning in the data center? So, we still see a lot of InfiniBand out there, but I think in the last 12 months, clearly the position even from Nvidia, you see a lot more push and support towards Ethernet. I think that's for a few different reasons. First of all, customers want choice, and so with multi-vendor as a key asset. I think also we're getting to a point where InfiniBand does hit, what is it, 30,000 GPU, there's a scaling limit where you do top out. And so as customers start to think about hundreds of thousands of GPUs, even if that's not today, that is an Ethernet-based network rather than InfiniBand. And so for multiple reasons, I think it's very clear that it's all Ethernet. We don't see that being a major topic. Most customers come, even if they've had a lot of InfiniBand, with an okay, I'm heading towards Ethernet. Now, one of the things that we have had in those discussions is there's some features in InfiniBand, some multi-tenancy features that they've had that aren't native in Ethernet. So beyond operational simplicity, which we've put a lot of effort into, we are looking at pulling some of those feature functions out of InfiniBand and adding that to Ethernet. And so that's something you'll hear more from us in the next few months. So, Ethernet gives customers scale and choice. It's also very familiar to all of us. We have Ethernet almost everywhere these days. But there's another problem that you need to be aware of. If these AI clusters are worth hundreds of millions of dollars, and most of the traffic is moving east-west between servers, security cannot just sit at the edge of the data center anymore. It has to move deeper into the cluster. Another major announcement we talked about this week is around services. We're in many regards a hardware vendor at heart. We love shipping news. But many times customers are already, if you think about a cluster, they already come with very sophisticated chips in the servers. And so the DPUs, these super NICs, there's the ones that go in the east-west, so connecting GPUs. But the ones that go in the north-south, so connecting servers to the internet, connecting servers to other servers, which we call north-south NICs, those, for instance, BlueField 3, BlueField 4, those are actually very sophisticated. They have a very large number of beefy Arm cores. You can do a lot of programming in those. And so one of the announcements of GTC has been we're now providing security services, our first entry point onto those NICs, on the north-south, so that we can provide, for instance, firewall type function without having to buy more hardware. So you just enable it, and that's something we plan to keep extending. And that interconnects with Isovalent, which is the company we acquired, which allows us to do distributed policy enforcement point. And so our goal is, you should be able to have policy defined without the Isovalent controller and then deploy the enforcement across all of these different points, including in your AI cluster with hardware you already have. So you can enforce policy on the DPU in the server. Yes. You can enforce it on the switch as well, because it's doing something similar, right? So your traditional firewall no longer is all you have. Now you're putting the firewall everywhere basically. Absolutely. There's still a place for a firewall in your DMZ, for internet, but within these architectures, trying to put large beefy firewalls all over within whether it's your AI clusters or in your server to server communication within a data center, I think that phase is going to be coming to an end, and we at Cisco believe that's something that's a very critical architecture transition. That's a big shift. The firewall doesn't disappear, but in these AI data centers security cannot be one big box at the edge. The traffic is moving inside the cluster between the servers, the GPUs, NICs, DPUs, and switches. So the security has to move closer to the workload. And that brings the whole story together. The future AI data center is not just faster GPUs, it is faster networking, better optics, better visibility, better automation, validated designs, and security built into the fabric itself. And if a 100 terabits per second switch and 1.6 terabits per second interfaces is not insane enough, Will told me that it's not slowing down. Expect the speeds just to continue ramping up. Speeds in the data center. Yes. So we've got a 100 terabits per second switch, 1.6 terabits per second interfaces. Is it just going to keep on going? Are we going to get like 3.2 terabits per second at some point, stuff like that? Yeah, it's interesting, I was visiting with a hyperscaler last week and we were very closely co-designing where it's multi-years out. We were talking about 400 gig SerDes. We were talking about 3.2 terabit interfaces. We were talking about where is the limit where we will no longer be able to support pluggables. Liquid cooling being required instead of just desirable. So it is, I don't see an end yet. I see this is coming fast. And some of these like the SerDes, the analog technology where you actually serialize data into a single pair of wires, I'm still amazed how far that has come. So my biggest takeaway from Cisco's AI lab is this. AI is not just about GPUs. The GPUs are the part that a lot of people talk about, but the network is what makes it work together. The switches matter, the optics matter, the storage matters, the software matters, the automation matters, the security matters, and the validation matters, because when one small problem can cost millions of dollars, you don't want to guess. You want the whole system tested before you deploy it. That's what Cisco is doing in this lab. Testing AI infrastructure as a complete system, so customers can build these clusters faster, operate them more reliably, and avoid wasting expensive GPU capacity. And yes, I'd still love to have a home lab like this. Can't afford it. I mean, I need a power plant. I need lots and lots of money. I want to thank the entire team for showing me around, showing me the cool technologies that they get to work on. What do you think about this? What do you think about the technologies? Did you learn something new in this video? Hopefully you did. I'm David Bombal and I want to wish you all the very best.