Podcast: The Nuts and Bolts of Broadband Internet Operations, With Robert Rockell

There was a time, a decade or so ago, when many people thought it would be a long while before telecommunications networks could handle the migration from cable TV to over-the-top video streaming. Clearly a lot of Americans still do both, but it is striking how easy it has become to stream HD content on multiple screens at home at once. Rob talks about what happens behind the scenes to make this possible with Robert Rockell, vice president of network infrastructure at Comcast.

Mentioned

• Doug Brake, “Lessons From the Pandemic: Broadband Policy After COVID-19” (ITIF, July 2020).

Auto-Transcript

Rob Atkinson: Welcome to Innovation Files. I’m Rob Atkinson, founder and president of the Information Technology and Innovation Foundation. We’re a D.C. based think tank that works on technology policy. And while we work on technology policy, it doesn’t mean we always understand it. So my co-host Jackie is... The technology is not working today. So unfortunately you’re going to just be stuck with me. I think as you know, this podcast is about the kinds of issues we cover at ITIF. From the broad economics of innovation, to specific policy and regulatory questions about new technologies. And this episode is about the internet and broadband infrastructure. The engineering, or the guts behind what we all take for granted. And what so many of us are relying on so heavily these days. We’re going to talk about the nuts and bolts of how these systems actually work. And I think I’m going to learn a lot, and I hope you will too.

So we’ve got a great guest for that. Rob Rockell works as the Vice President of Network Infrastructure at Comcast. One of the leading internet providers in the U.S., where he and his team design and build the routing, switching, and optical infrastructure for Comcast’s cable business. He’s an early adopter and technologist. He’s authored standards and technologies, such as IP multicast, and IPv6. He’s designed VPN products and technology. And he has experience across the spectrum, in the service provider industry, from telco, to cable, to vendor environments. Welcome Rob.

Robert Rockell: Thanks for having me.

Rob Atkinson: Our pleasure. So, a lot of folks, including myself really, you turn on your computer, and you hope your wireless modem’s working. And off you go, and somehow something happens. The bits come on, and you can surf the web, or stream a video, or do whatever you do, and do what we’re doing now. And oftentimes we take it for granted that that just magically happens. But in fact, it’s incredibly complex. It’s almost like when Arthur C Clarke’s at any good technology is indistinguishable from magic. This is like magic. It’s like, “Wow.” And so I guess, if you could start off by just saying, can you just give us a lay person’s description, of what happens when I put a URL on my browser? And how does Comcast, or other network providers play a role in that? What do you do?

Robert Rockell: Sure. Internet plumbing 101. You plug a webpage into a browser. And first thing that your computer needs to do, is figure out who to ask for all the content, that’s going to make that webpage come up. And so, the very first thing that happens before you even send a request out to that server, is you’ve got to figure out where that server is. Much like your mailing address gives you a location, so the post office can find you geographically. In the internet you need an IP address, which tells you where you’re located topologically on the internet globally. And so the first thing you do is what’s called a DNS query. DNS stands for Domain Name System. And what that does, is that translates the name of the webpage you put in like itif.org, into the IP address where the data that webpage is built from is hosted. And so, you send a packet out to a DNS server. And either that is something that your ISP provides for you, or there’s many of them you can use publicly that you can configure yourself.

And if that DNS server knows the answer because it answered that for another customer recently, it’ll just tell you. And if it doesn’t, it gives you a referral. And you can go through a hierarchical set of requests, until you get to the DNS server that’s authoritative for that name. Like itif.org. It sends a response back, and now you know where to send your request. And that all happens in the order of milliseconds. And so once you know the IP address that you’re trying to get to, you send data out the wire. And inside your home you essentially have information about what are the other hosts within your home. But your home router says, “If this is not [inaudible 00:04:24] someone else in the home, shoot it out on the wire.”

And so it shoots out on the wire, and enters the internet core, where I spend most of my time. Once it gets to the internet core, it attaches to a router. And that router has information about, really all the reachability on the internet. And it doesn’t have every IP address. IP addresses are aggregated into groups and blocks, much like a zip code or a State will tell the post office in aggregate, where to send a piece of mail. But there’re hundreds of thousands of entries in a routing table. Really every router in the internet. And when you enter the Comcast network where I work. We’ll do a look up to find that destination address. And we’ll find out two things. One, we’ll figure out if that server is outside of the Comcast network, what is the best exit point to get to it?

And then second, how do you traverse the Comcast network in the most efficient way, to get to that exit point? And so maybe if we’re in Washington D.C., maybe where the servers hosted you want to exit the network in Chicago. So they’ll say, “Okay, go to Baltimore.” When you get to the Baltimore router, it does the same lookup. And it’ll say, “Go to Pittsburgh.” And then from Pittsburgh, maybe to Cleveland, and then Chicago. At which point it finds the exit point of the network, and hands off to another provider. Or to a data center inside Comcast where that webpage can be served.

Rob Atkinson: And all of that, go into Pittsburgh, Baltimore. That could be on a Comcast network, but it might not be?

Robert Rockell: Exactly. The goal is you really have two sets of communication. We have a set of information about our internal network, and all the reachability inside our network. And then you have a set of information about all the ways to get to everything outside the network. And what is the best exit path to take to get there.

Rob Atkinson: I hate to tell you I’m not a Comcast subscriber, I shouldn’t admit that. But I’m in another company. But if I were a Comcast subscriber, and I were communicating with my brother who was in Illinois, who’s a Comcast subscriber. And my data then goes to my Comcast router. And then there’s probably co-ax in the Comcast network, that ultimately then goes into some box where it goes to a fiber cable. Could that possibly go all the way on Comcast wires to my brother? Or does it not necessarily?

Robert Rockell: If you’re using a Comcast application, or you have one Comcast subscriber talking to another, it never leaves you at the Comcast infrastructure. It stays on that end to end.

Rob Atkinson: Okay. That was really a really good basic one-on-one take. The other thing I think people get confused about. I remember early, early on my second job out of graduate school, there was a thing called the Congressional Office of Technology Assessment, OTA. The advisory body for Congress. And we were working on some project there, I was helping out. And it had to do with the information super highway. What Al Gore used to call it, had termed it. And I was like, “Well, if bits travel at the speed of light, why are we talking about the need for faster networks? It all travels at the speed of light.” And that’s because, it does travel the speed of light. But if the hole’s really small, and going back to Ted Stevens, the pipes get crowded. It doesn’t go to that fast. So can you explain a little bit about the broad in broadband? And how do we get faster and faster speeds, as surely Comcast has been doing?

Robert Rockell: Yeah, certainly. Depending on which part of the network you’re in, there’s a little bit different techniques. In the core of the network, you’re entirely fiber and you’re using optical. And it’s really about using lasers, to put more bits over a shorter period of time, onto that piece of fiber than you could in the past. And so there’re different technologies that allow you to use the way the light is polarized, the frequency and the amplitude of that light. But essentially, you’re trying to build the ability to put multiple signals onto that fiber at the same time. And then you have a receiver that can decode those. And so you’re multiplexing them on. Today a piece of fiber, in 2019 you could get up to about eight terabits on a piece of fiber. As we move to the next step, that’s going to double, and quadruple, and continue to increase as those technologies that allow really sensitive lasers, that can receive the data and interpret it. And then, powerful transmitting lasers to send more bits on the wire. But those, they continue to double and quadruple overtime.

Rob Atkinson: That’s crazy, because my understanding is that it’s similar to Moore’s Law, doubling on a pretty regular basis. Moore’s law being computer chips. It’s related but not exactly the same. Because it’s more about optical advancement. Is that right?

Robert Rockell: That’s right. When you have physics involved, and Moore’s Law, it doesn’t account for heat, or power, and some of the other factors that really go into building optical systems. And so it doesn’t follow exactly. But the evolution is certainly happening at a scale like that, where it is exponentially increasing.

Rob Atkinson: I remember a decade ago when we had been working on broadband policy for 15 years. But I remember when we were working on it. And there was a lot of talk about this issue of net neutrality came up. And there was this concern that carriers, ISPs would want to limit the size of their pipe, because they didn’t want competition with their cable offering. You wouldn’t want TV to go over their OT, to what’s called over the top. And at the time we thought, “Yeah, that’s really dumb. It’s a dumb idea. It’s not going to happen. I think those carriers have every incentive to expand the speed of their networks.”

The thing we wondered about though, was with the rise of video, and not just two-way video, but streaming video. There are times in our household, where we have three people in our household streaming different movies. And when you think about it 10 years ago, that was unprecedented. You didn’t really see the broadband pipe as a video pipe. It was more of an internet e-commerce pipe. And then the other pipe was the co-ax cable streaming one show at a time, or 500 shows at a time. How did we get there? Why did that, now all of a sudden we can do this and have no problems?

Robert Rockell: It’s amazing. I think it’s a number of factors. Obviously we talked about innovation driving network scale. And we talked about getting more symbols on the wire. When I started in the industry in the mid ‘90s, we were moving from T1, which was 1.5 Megs in the internet core, to DS3 which is 45 Megs. I think about the paths we have in our core between the cities now. And it’s in the multiple terabytes per second, in terms of capacity. So it’s millions of times larger than it was. And so scale certainly plays a factor. But not only that, these days we have tremendous amounts of redundancy. And so when we’re building networks, we’re building networks so you can take out nearly any fiber in the network, or any router in the network.

And not only will you re-converge around that, and services will continue to work often in under a second. But we can handle that capacity even on Friday night when everyone is consuming their over the top video. And so we’ve got enough redundancy, to be able to handle any failure even during peak. And then layered on top of that, there’s lots of software running in the network. So the software in the network, is in addition to the routing software that makes the network forward traffic and move traffic around. We’re looking at telemetry. We’re looking at characterization of what’s going on in the fiber, in the routers, in the optics. Looking at the memory consumption of our infrastructure. And so we’re constantly making assessments through software, and feeding data so humans can make assessments. So we’re auto correcting issues.

And when there is a failure, we’re much more quickly directing humans to engage in the right fix activity. And so that software is driving a lot of intelligence, reducing outage time. And then on top of that, you look at over the top video today. And those application environments as well have a lot of optimization in place. They have a lot of geographic redundancy. There’s a lot of feedback being sent back to the overtop video owners. So they’re getting real-time view into what your experience looks like. And they have the ability to optimize even outside of the network, how that application is being delivered. What kind of encoding they’re using to get to you. So that all these things really work together to create a very smooth experience.

Rob Atkinson: That’s really fascinating. And I assume that’s why it’ll lead into my next question, the answer to that. But earlier this summer, ITIF have released a report. Where we looked at the performance of U.S. broadband networks after the pandemic, in part compared to other countries. And what we found was that actually U.S. networks performed among the best in the world. In terms of this master shift, it’d be like having a freeway system, where you’ve designed and synchronized all the lights for going to work at eight, and coming home at six. And now all of a sudden everybody’s going to work at noon. The whole patterns were changed. Where people were online, when they were online. The kinds of things they were doing. And yet, your network performed well. Pretty much everybody’s network in the U.S. performed quite well. And I assume it’s one thing that you just said, but maybe you can just go into a little bit more detail on that.

Robert Rockell: Sure. Well before COVID, we were making massive investments in the network infrastructure. Because we want to stay ahead of demand. We want people to be able to drive innovation and new experiences into the network. And it’s bread and butter for us. And so that investment has us in a position where, I mentioned redundancy, where we can take a lot of outages, and customers won’t feel it. Well that extra capacity is there. And so the fact that we do have fiber cuts, and we do get to exercise a redundancy in the network. And sometimes we’ll intentionally take down our infrastructure as well, to make sure that redundancy works. So there’s a really active practice around making sure, there’s available secondary and tertiary infrastructure for people to use. When COVID first hit, that meant we had a lot of buffer to really absorb that increased demand, inside the core of the network. On the access side of the network, a different ball game a little bit. You’ve got an HFC. You’ve got both downstream and upstream considerations to take care of.

Rob Atkinson: Can you say what HFC is?

Robert Rockell: In cable, we have this hybrid fiber co-ax network. Where, your TV often connects by HDMI now, but throughout the house you have coaxial cable wired. And our goal is to use science as much as we can, to put as much signal on that co-ax as we can. And then it converts into fiber during the access network before you get to the core. But in that part of the network, we had launched some software just beyond trial, right before the shelter in place had happened. Where we would look at the radio characteristics on that co-ax HFC network, for all of our subscribers every 20 minutes. And so we were able to roll that out to the entire nation very, very quickly. And produced, I think one of our press announcements said, something like more than 35% more aggregate capacity. Just by optimizing the way we send and receive signals across that co-ax, to give people cleaner signals using available spectrum on that signal. So in terms of incremental video conferencing, or schooling from home that occurred, it really helped us absorb as well.

Rob Atkinson: So that’s a little bit like the Tesla car, where they’ll download software upgrade. Which, for cars isn’t... So really what you’re almost doing, is you’re just having a software upgrade to your network. Is that right? You weren’t putting in new wires in the ground, I guess.

Robert Rockell: We’re constantly building construction, and so we have that capability as well. Because you want to respond to that demand, and add more redundancy in those things. But in this case, it wasn’t really so much as a new software upgrade, as much as we’re looking at the ways that our equipment was using the network. And we would tweak it. So we’d say, “Rather than using this frequency, use this frequency in the HFC.” In the core of the network we’d say, “Rather than put it all on this path, let’s load share it over these two paths. And that way we’ve got more capacity we can use.” And so we are constantly milking the network to get more out of the existing infrastructure.

Rob Atkinson: A lot of people when they talk about broadband, the focus is principally on speed. “Hey. I did a speed test recently, and I think I’m at 65 symmetrical. Or 65 megs symmetrical. Oh, this is great.” But what people oftentimes don’t talk about, that’s what’s called latency. In other words, how quickly something refreshes. And that’s pretty important in gaming if you’re on a real time shooting game. On a video game, you want to be able to make sure that it goes quickly. But low latency is going to become a lot more important. Especially as we move to what people call the fourth industrial revolution, in these kinds of applications. How do you see that playing out? And what’s Comcast doing in that space around latency, as well as speed? You’ve already talked about speed, but latency.

Robert Rockell: When I think about latency, and we’re designing for latency, we really have three different concerns we try to deal with. You have to have the things in the access network. And the mechanics for how your access network gets traffic onto the network. You have the core of the network. And then you have congestion concerns. Because one of the things that can cause a latency, is if you put too much traffic onto a link. And so going through each of the three of those, in the access network we use a protocol called DOCSIS. And it basically specifies how you signal data onto that HFC, that hybrid fiber co-ax network. Every generation of DOCSIS that we deploy, we work to figure out, how can we tune that protocol to reduce latency there?

Because just the sharing of that medium across the neighborhood can produce some latency while it just makes sure everyone’s getting their traffic onto the network. When you get to the core of the network, latency is really dominated by distance. Photons travel at two thirds of the speed of light in fiber. And so to some extent, if you’re traveling across the country, or traveling across an ocean, you’re really dominated by that actual distance. And so for us, we’re very careful what fiber paths we pick. And so if you looked at a U.S. map, the lower 48, you looked at the highways and the railways, that looks very similar to the dominant fiber paths in the U.S. as well. Not only because those are a lot of the rights of ways, but because those are the efficient routes. We consider geographical barriers and those sorts of things.

And so we’re always looking to make sure we’ve got great paths picked. And then great secondary paths. So if that one ever gets cut by a backhoe, or some storm or something, you’re not going all the way around the country to route around it. You’re staying as efficient as you can. From a congestion standpoint, like I said, we’re building our network to be able to withstand one or even multiple failures, at the same time in our peak usage periods. So because of that, we have a lot of extra capacity in nominal state to be able to use. And so by doing that we can assure, even if there’s a large event like a gaming download, or a shelter at home that happens, we’re able to absorb that without creating congestion in the core.

Rob Atkinson: You mentioned earlier, and I think it relates to this question about congestion and management. And that’s, in almost every sector with technology now, people are focusing on AI, artificial intelligence, machine learning. You don’t hear about it as much in the ISP world. But are you doing much in AI? And how is that helping the network management process?

Robert Rockell: Oh, absolutely. I’ll give you a couple examples. We have AI and supervised machine learning, working in the background in all parts of our network infrastructure. In the access layer we have something we call Octave. And that is what I mentioned earlier, that software that’s looking at really over 20 million cable modems, over the course of 10 or 20 minutes. And making decisions, about how best to program them to send them receive data, so that they’re maximizing throughput. So if there’s any impairments in the way signals are being processed, they can adjust that very quickly. And that’s driving all the capacity we can get out of the co-ax, with the HFC in there. In the core of the network, we have something we call the smart network platform. And that’s processing millions of bits of data. Whether they be log messages coming out of our routing equipment. Or characteristics of our fiber. But millions of bits of data are coming in, and being processed to try to look at, number one, if there is an event, how do we correlate it?

So we take many different signals we got, that said that something went on. And help us pinpoint exactly where that is. So we can either use software to correct the problem. Or more quickly direct humans at taking corrective action. It’s also looking at patterns to see... We found that we found that certain lasers, certain optics, we can figure out they’re going to fail within the next 48 hours, by watching the way the laser will skew, and its power will ramp up and down. And using machine learning techniques, we found that there is a signature there. So now we can find out that a piece of optical equipment, is going to fail a day or two days ahead of time. So we can make that replacement on our timetable, rather than the equipment’s timetable when it fails. And so, lots of software learning, and making recommendations about what to do in the network.

Rob Atkinson: That’s very cool. We have been, at ITIF, and also our branded center, the Center for Data Innovation, we’ve been focusing on AI and AI policy for a whole number of years. And one of the things we argue, is that yes, privacy is important. But so is the ability to use data for important functions. Like keeping networks working. Do you have any concerns about... I know you’re not in the government relations shop, you’re actually running the network. But Congress is seriously thinking about some national privacy bill, which we support the intention. We certainly don’t support a European style model. Do you have any thoughts about that, or any concerns about that? Where Congress might be careful of not going.

Robert Rockell: I’m sure they are good conversations to have. It’s not really my area. I can tell you for what we’re doing, I can get a lot of great data to help optimize experiences for everyone, regardless of application, by looking at tonnage and the network, holistically as itself. And that gets me out of ever having to look at even a specific application, much less a specific user’s applications. And so we’re making a lot of hay without getting into that space.

Rob Atkinson: So I guess one last question, before I ask you this final question. There’s been a push for a number of folks, particularly folks who think that networks are like roads, and should be operated and owned by the public or by government, and the like. And the model tends to be municipal broadband. Where a city would do this. And listening to what you’ve described Rob, it’s strikes me that this is an industry that number one, has a lot of scale. And number two, there’s an enormous amount of technological innovation. And hard work. And R&D. And management of technology that goes in. It’s not like just, “Hey. I got a computer in the back office here. And I’m going to put some wires out, and hopefully the thing works.” Can you say a little bit more about, if that’s right, I think it is, why do you think that’s the case? And why is it that we need sophisticated technology companies to be running our networks?

Robert Rockell: If I think about the few hundred people on my team that develop, and deploy, and make the design decisions for the network infrastructure at Comcast. We all derive a lot of meaning in our careers, by the ability to unleash innovation by building networks that support demand as fast as it can grow. And new experiences and things. And so for us it’s been, that there’s a lot of passion there. And so the innovation is not for its sake. If the innovation is there, its we love solving problems. And I think that the passion that you see in the organization, is really what’s driving the software decisions we make around using AI and ML. We’re out there to really help society move forward. So I think that meaning has been enough of an impetus on its own, that it drives a tremendous amount of creative thinking. Draws fantastic talent to these companies. And creates a great work experience.

Rob Atkinson: I’ve had the pleasure of visiting the Comcast headquarters. Visited a number of technology company headquarters around the country in Philadelphia. And I was surprised, maybe I shouldn’t be surprised, but it’s an R&D lab partly up there. They’re real engineers that are just exploring the future. It’s not just, “Hey. I’m running this piece of wire and managing it.” It’s about the future.

Robert Rockell: Yeah. We really love it. The ability Comcast gives us to innovate. The autonomy they give us to experiment in ways that are safe for our customers. And really produce, whether they are improvements in our infrastructure, whether that be, if it’s one side of our products, or in the network infrastructure. It’s really great. It makes for a place where people really love to work in and learn.

Rob Atkinson: Rob, last question. Sometimes people forget that the internet is actual real wire. So hopefully after this, they won’t forget that. And all sorts of things can go wrong. What’s the weirdest or oddest source of a service outage you’ve ever heard about.

Robert Rockell: When you’re building infrastructure that spans from the neighborhood to the internet core, there’s a lot of opportunities to find some interesting things. My team at our Christmas party every year, we actually pull up notes from some of the tickets of some of the craziest stuff we saw. And so I’ll give you a couple. We had a guy running a weed wacker on Interstate 95. Hit a fiber tube one time. That created a fairly large and complex fiber cut. It was just some guy, cutting the weeds in the meeting of the road. Luckily we had fiber that could route around it. But it took down a lot of fibers. There was a birthday party in a hotel one time. And those foil helium balloons, someone let go of it, and ended up shorting out some electrical circuitry, at the roof of the hotel in the lobby.

And that happened to be a circuit that was powering some equipment, that was used to run cable through the neighborhood that was adjacent to that. You see all sorts of stuff. And so for us, trying to build a network that is defended against every kind of outage is impossible. Because something’s going to come up. Whether it’s a squirrel chewing through something, a pole fire. You’re going to find these kinds of things. And so for us, it’s about building the resilience. That no matter what kind of wacky thing happens, the impact is minimal, or non impactful to our customers.

Rob Atkinson: Yeah. I think as we have learned in the last six months, resilience is probably an under appreciated aspect of our society. And something we’re all going to have to be focused more on as we go forward. So Rob, thank you so much. And that’s it for this week. If you liked it, be sure to rate us and subscribe. Feel free to email show ideas or questions, to [email protected]. And you can find the show notes, and sign up for our weekly email newsletter on our website, itif.org. And follow us at Twitter, Facebook, @ITIFdc. We have more episodes and great guests lined up. New episodes drop every other Monday. So we hope you’ll continue to tune in.