(#4) Sat. Jan 22nd, 2022 - BGP Hell

Border Gateway Protocol, oh how we rarely get along. The thing about starting an ISP is that you usually don't have a lot of money flying around like the incumbents that took billions of taxpayer dollars to build sub-par networks in the 80s and 90s. With said lack of money comes the frustration of not being able to buy into the gold standard networking ecosystems like Cisco or Juniper. In our case, we decided to go with MikroTik. This is not a roast of MikroTik, as they've proven time and time again that they can compete with the big players. This is, more or less, just me shouting into the void about BGP compatibility and adhesion to RFCs between vendors.

When you graduate from single broadcast domains and into the world of routing, there's a lot to learn. What are all of these acronyms flying around? BGP? ASNs? OSPF? VRFs? Yeesh, the subject matter explodes. Luckily, everyone's got it right for the most part and any manufacturer in the network space worth their salt has implemented all of these standards.

LACP backhaul to our colocation rack

Moving towards a fully routed network brings with it some challenges, namely if you don't own a lot of IP space. Fitchburg Fiber is the proud owner of a /24 block, or 256 public IP addresses. We paid almost $7k for the block at auction back in January of 2021. Nowadays similarly sized blocks go for nearly $14k at auction. Needless to say there's very little chance we're getting more IPv4 addresses anytime soon. IPv6 is still looming around the corner waiting for more widespread adoption. When our tooling and hardware supports it to the depth it supports IPv4 we're taking the plunge, but for now the name of the game is conservation.

This brings me to my point. MikroTik is very great nearly-carrier-grade gear for budding players in the ISP game, but their BGP implementation in RouterOS v7 has consistently left a sour taste in my mouth. I just spent all day battling a spotty session with Cogent only to realize that v7 BGP doesn't like session encryption keys that are 80 characters long. Thanks Aaron from Cogent for bearing with my suffering for half an hour. I had to roll back to v6 for the time being just to re-evaluate my decisions.

I guess it's back to the lab again to build a replica of our core network and figure out where I went wrong.

Live well.


You'll only receive email when they publish something new.

More from Andrew DeChristopher
All posts