I’ve long had a /48 from Hurricane Electric (HE)/Tunnelbroker for my IPv6 connectivity for servers I run at home and for general end-user purposes. I’ve also had Xfinity for years which has native IPv6, but never have used it for a couple of reasons. Firstly, I assume it’s a dynamically allocated prefix subject to change anytime which could unexpectedly break inbound connectivity to my Internet-facing servers when the address changes. Since iPhone 7 I’ve had IPv6 on my AT&T data service and it’s rather nice to hit something at home without port forwarding. I also run an authoritative DNS server on IPv6 at home for my domains, something not easily scriptable for changes.
Secondly, both Xfinity and HE do source address filtering, so traffic sourced from HE gets dropped when it exits via Xfinity and vice versa, so I can’t just run them in parallel without work. I didn’t want to deal with trying to script or dynamically update DNS hostnames and firewall rules to access my servers over Xfinity v6, and the HE tunnel has been very reliable. So, I want to keep my Internet-accessible servers on my HE /48 and let client-y things use Xfinity v6 otherwise.
The main problem is leaving v6 performance on the table once I got to using a faster Xfinity service. 6in4 traffic (IP protocol 41) is not handled by hardware offloading on Ubiquiti EdgeRouters. Even on an EdgeRouter 12, I can only get about 400 Mbit/s of tunneled IPv6 on my gigabit Xfinity service before the CPU runs out of steam (due to soft interrupts). Testing IPv4 only, I’m able to get ~950 Mbit/s through NAT. I’ve artificially throttled my IPv6 performance by only using the HE tunnel.
Another problem is that I’ve learned that some CDNs like Cloudflare and ticketing websites do not like requests from HE IPv6, presumably because they don’t like VPN and tunnel users.
Edit: I forgot, I run an authoritative name server for several of my domains from home over IPv6 because I haven’t always had another VPS somewhere to serve as a required secondary name server, that’s another reason for wanting a static /48.
Enter policy routing
So to use both Xfinity and HE connections, I need some policy routing. Unfortunately EdgeOS and the Vyatta stack on the EdgeRouter doesn’t natively support IPv6 policy routing through the GUI or config tree, just IPv4. Fortunately it runs a Linux kernel and it has the iproute2 packages installed so it’s possible to manipulate the kernel routing tables directly to set up IPv6 policy routing via ip rule and ip route commands.
I had looked at this years ago and it sounded annoying to set up, but in the end taking it command by command it wasn’t so bad. I referenced these web pages to get ideas what to configure:
https://serverfault.com/questions/854094/linux-ipv6-policy-based-routing-fails
https://www.sixxs.net/forum/?msg=setup-10320966
https://jsteward.moe/he-ipv6-routing-on-machines-with-ipv6.html
https://web.archive.org/web/20130812091825/http://itkia.com/ipv6-policy-routing-linux-gotchas/
Topology
My home network looks something like this:
There is one router connected to my Xfinity cable modem, two segregated LANs. One is my plain simple home network where all of my wired and wireless phones, laptops, desktops, IoT devices live. It has a 192.168.x.x IPv4 /24 and the first routed /64 that HE assigns to your Tunnelbroker account. The second LAN is behind another router serving up ikeacluster and the rest of my development network. It’s where most of my Internet-accessible servers live and it’s part of the routed /48 network from HE.
In this scenario, I have two IPv6 networks I need to policy route. One is my daily driver /64, and one is my homelab /48 network. I am only interested in Xfinity IPv6 addresses on my daily network, and therefore do not do any sort of prefix delegation to the ikeacluster/homelab side. For the most part devices either get v6 addresses as static assignments or SLAAC.
What I will wind up with is that homelab addresses will stay exactly the same, within the same HE /48. What will change is that now my home network will get both RAs. Things like my phone, laptop, Apple TV will get 2 IPv6 addresses now, one from the same HE /64 as before, and now a Xfinity IPv6 2601:: address. Only HE traffic is special cased, I have not mangled Xfinity traffic at all. I could have gone as far as removing RAs for my HE prefix on my home LAN so things like my laptop/phone/TV only get Xfinity v6 addresses, leaving static HE address assignments to a few select boxes, and it probably would work just fine, but I decided to let them have both.
Router config
This assumes you already have a working tunnel to HE. You have just a /64 or you have a /48 routed from HE toward your tunnel.
To set up v6 policy routing here, expect some temporary IPv6 breakage as you move default routes around. You’ll need to remove the existing default IPv6 route you have pointing at HE’s tun0 interface, then configure DHCPv6 PD for Xfinity. What will happen is as you configure the EdgeRouter for Xfinity IPv6, you configure the DHCPv6 prefix delegation and router advertisements on your LAN and commit, all of a sudden all of the devices on the LAN will learn two v6 addresses (HE and Xfinity) and two gateways. Until you get policy routing going, v6 traffic could start egressing via the wrong ISP/connection. If you get in a panic and try to un-configure the Xfinity PD and RAs on the EdgeRouter, all of your LAN devices have more than likely already learned an Xfinity address and default route and will stay that way until the lifetimes expire (several minutes) or you manually go around fixing them. It’s not the end of the world, just be aware of it.
I recommend setting up DHCPv6 prefix delegation first on your WAN interface facing Xfinity. Commit this first and you should have a /128 address show up on your WAN interface. This at least lets you know Xfinity and PD is somewhat working before you get further.
e.g.
set interface ethernet eth0 dhcpv6-pd pd 0 interface eth8
set interface ethernet eth0 dhcpv6-pd pd 0 prefix-length 60
set interface ethernet eth0 dhcpv6-pd rapid-commit enable
commit
save
[bwann@home-gw1 ~]$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 98.47.xxx.xxx/22 u/u outside
2001:558:6045:c0:4040:aaaa:bbbb:cccc/128
Later, or now, #yolo, enable prefix delegation and begin sending out RAs on your LAN:
set interface ethernet eth8 ipv6 router-advert prefix ::/64 autonomous-flag true
set interface ethernet eth8 ipv6 router-advert prefix ::/64 on-link-flag true
set interface ethernet eth8 ipv6 router-advert prefix ::/64 valid-lifetime 2592000
commit
save
Firewall considerations
You’ll need to apply an IPv6 firewall policy to your WAN interface along with your HE tunnel interface because now you have v6 coming in on two interfaces. My firewall rules are default deny-all on inbound, with rules to allow from specific addresses and/or to specific addresses, with no consideration of the actual interface names. This means I can re-use my existing HE tunnel rules on my WAN interface to keep everything in one place (here I have a rule for v6 traffic passing /through/ my router and another ruleset going /to/ my router)
# WAN interface rules matches tun0 interface
set interface ethernet eth0 firewall in ipv6-name outside6 set interface ethernet eth0 firewall local ipv6-name system6
Policy script
Next, I have written a script to bring up my policy routing in an idempotent manner. I actually wrote and tested this one line at a time so it should be safe running it over and over again. I saved this to /config/scripts/policy.sh so it persists between EdgeOS upgrades:
#!/bin/bash
#
# IPv6 prefixes to policy route
prefixes="2001:470:1f05:2c9::/64 2001:470:8122::/48"
# Add he-ipv6 table
if ! grep "^200 he-ipv6" /etc/iproute2/rt_tables ; then echo "200 he-ipv6" >> /etc/iproute2/rt_tables ; fi
# Flush everything and re-add standard EdgeOS rules
ip -6 rule flush
ip -6 rule add priority 32766 from all lookup main
ip -6 rule add priority 220 not from all fwmark 0xffffffff lookup 220
ip -6 route flush table he-ipv6 || true
for prefix in ${prefixes} ; do
ip -6 rule add from ${prefix} table he-ipv6 || true
done
for prefix in ${prefixes} ; do
ip -6 rule add to ${prefix} table main || true
done
# Comcast dhcpv6-pd prefix, punt to main table
ip -6 rule add from all to 2601:646::/32 lookup main
# for prefix in "${prefixes}" ; do
# ip -6 route add unreachable ${prefix}
# ip -6 route add unreachable ${prefix} table he-ipv6
# done
# Set default route in he-ipv6 table to HE's end of the tunnel
ip -6 route add default via 2001:470:1f04:2c9::1 dev tun0 table he-ipv6
Walking through what this does:
# IPv6 prefixes to policy route
prefixes="2001:470:1f05:2c9::/64 2001:470:8122::/48"
A simple string list of prefixes to policy route, separated by spaces.
# Add he-ipv6 table
if ! grep "^200 he-ipv6" /etc/iproute2/rt_tables ; then echo "200 he-ipv6" >> /etc/iproute2/rt_tables ; fi
Create a new, separate route table to hold our rules and routes for HE sourced traffic. If it’s already in the rt_tables file, do nothing.
ip -6 rule flush
ip -6 rule add priority 32766 from all lookup main
ip -6 rule add priority 220 not from all fwmark 0xffffffff lookup 220
Blow away any existing IPv6 rules to ensure we’re working with a clean slate. Re-add what should be standard out-of-the-box rules, using the main routing table. I’m not exactly sure what the fwmark lookup with table 220 is for, as there’s no mention of it in /etc/iproute2 anywhere, but we’ll faithfully re-add it because it’s what was there to begin with. (You can run ip -6 rule showto see what was there to begin with before flushing them.)
ip -6 route flush table he-ipv6 || true
for prefix in ${prefixes} ; do
ip -6 rule add from ${prefix} table he-ipv6 || true
done
Do a similar thing, blow away any existing routes in our he-ipv6 routing table so we’re sure we’re only policy routing what the script expects. Always return true even if we don’t flush anything. Then iterate through our prefix list, adding from rules to our he-ipv6 table, this is what will actually match on the source address of the HE traffic.
for prefix in ${prefixes}; do
ip -6 rule add to ${prefix} table main || true
done
Add rules so if there’s traffic coming into the EdgeRouter then send it to the normal routing table. Basically if it’s local traffic, keep it local. This solves an embarrassing side effect that when I would try to connect from my laptop in the livingroom on a Xfinity v6 address to a machines in the homelab with an HE address in the bedroom, the traffic would bounce down to San Jose and back.
# Comcast dhcpv6-pd prefix, punt to main table
ip -6 rule add from all to 2601:646::/32 lookup main
2601:646:: is currently the /32 that my /60 prefix is ultimately delegated out of. I don’t know if/how often the /60 changes, so I’m trying to take into account any future changes.
Add a rule I believe similarly, any traffic from a Xfinity address, send it to the main routing table. This is hacky because who knows when I won’t get a prefix delegation from 2601::646::/32 anymore and could be scripted better.
ip -6 route add default via 2001:470:1f00:___::1 dev tun0 table he-ipv6
Lastly, add the default route for the HE routing table to point at the remote end of my HE tunnel.
In some of the examples I linked above, people were going further an adding statements to make certain addresses/endpoints unreachable. I don’t think that’s necessary in my case, or at least so far I haven’t experienced any weird side effects after running this for several months.
TODO: This script needs to run every time at boot to set up the policy routing. I haven’t gotten around to that, probably needs to be a task-scheduler job or similar.
Multiple router advertisements on same LAN are fine
On my home LAN it doesn’t really matter that my EdgeRouter is sending out RAs for both the HE /64 and the delegated Xfinity /64. The next-hop for the default router goes to the same place. I can stick a random Raspberry Pi on the home network, it’ll get an address from both /64s and work just fine reaching the Internet and the rest of my networks.
Results
After running the script, thing should look something like this:
root@home-gw1:/home/bwann# ip -6 route show | grep default
default via fe80::21c:73ff:fe00:99 dev eth0 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium
root@home-gw1:/home/bwann# ip -6 route show table he-ipv6 | grep default
default via 2001:470:1f04:___::1 dev tun0 metric 1024 pref medium
The main v6 routing table should only have a single default route, in this case the fe80:...:fe00:99 is the Xfinity v6 gateway I learned from DHCPv6. Likewise the he-ipv6 routing table should have a single default route, pointing at the remote end (HE’s side) of my tunnel.
root@home-gw1:/home/bwann# ip -6 rule show
0: from all lookup local
214: from all to 2601:646::/32 lookup main
216: from all to 2001:470:8122::/48 lookup main
217: from all to 2001:470:1f05:___::/64 lookup main
218: from 2001:470:8122::/48 lookup he-ipv6
219: from 2001:470:1f05:___::/64 lookup he-ipv6
220: not from all fwmark 0xffffffff lookup 220
32766: from all lookup main
root@home-gw1:/home/bwann# ip -6 rule show table he-ipv6
218: from 2001:470:8122::/48 lookup he-ipv6
219: from 2001:470:1f05:___::/64 lookup he-ipv6
Rules configured, the key part here is that traffic from my HE addresses are going to be routed according to the he-ipv6 . I.e. traffic sourced from my HE /64 and /48 will egress via my HE tunnel, anything not matching it egresses via Xfinity.
Traceroute results
Here’s proof in the pudding or something:
From one of my Linux boxes on my home network that only has an IPv6 address from the HE /64, traceroute to facebook goes via the HE tunnel:
[bwann@raptor ~]$ traceroute6 www.facebook.com
traceroute to www.facebook.com (2a03:2880:f131:83:face:b00c:0:25de), 30 hops max, 80 byte packets
1 2001:470:1f05:___::1 (2001:470:1f05:___::1) 0.270 ms 0.196 ms 0.161 ms
2 tunnel263332.tunnel.tserv3.fmt2.ipv6.he.net (2001:470:1f04:2c9::1) 15.380 ms 22.593 ms 21.033 ms
3 10ge11-19.core4.fmt2.he.net (2001:470:0:45::1) 19.478 ms 22.419 ms 19.383 ms
4 * * *
5 xe-0.equinix.snjsca04.us.bb.gin.ntt.net (2001:504:0:1::2914:1) 23.900 ms 23.379 ms
xe-0.paix.plalca01.us.bb.gin.ntt.net (2001:504:d::6) 23.790 ms
6 * * 2001:418:0:5000::751 (2001:418:0:5000::751) 30.911 ms
...
10 po243.psw02.sjc3.tfbnw.net (2620:0:1cff:dead:beef::5ddf) 20.820 ms
edge-star-mini6-shv-01-sjc3.facebook.com (2a03:2880:f131:83:face:b00c:0:25de) 20.502 ms 20.415 ms
From my laptop on the same LAN with both HE and Xfinity addresses, to facebook, this time it goes out via Xfinity:
lapdance:~ bwann$ traceroute6 www.facebook.com
traceroute6 to star-mini.c10r.facebook.com (2a03:2880:f131:83:face:b00c:0:25de) from 2601:646:9e01:1600:ad2f:8fee:48af:d42e, 64 hops max, 12 byte packets
1 2601:646:9e01:1600:7683:c2ff:fe14:5129 4.050 ms 3.651 ms 4.056 ms
2 2001:558:1014:30::2 10.578 ms
2001:558:1014:30::3 15.969 ms
2001:558:1014:30::2 15.631 ms
3 po-324-346-rur202.sanjose.ca.sfba.comcast.net 22.185 ms
2001:558:82:840a::1 10.807 ms
po-324-346-rur202.sanjose.ca.sfba.comcast.net 16.639 ms
4 2001:558:80:168::1 11.184 ms
po-2-rur201.sanjose.ca.sfba.comcast.net 12.387 ms
2001:558:80:168::1 12.626 ms
...
13 po7.msw1at.01.sjc3.tfbnw.net 16.717 ms
edge-star-mini6-shv-01-sjc3.facebook.com 15.436 ms
po7.msw1ag.01.sjc3.tfbnw.net 18.761 ms
From my laptop to my Chef server running in ikeacluster:
lapdance:~ bwann$ traceroute6 chef.wann.net
traceroute6 to chef.wann.net (2001:470:8122:1:227:eff:fe26:3238) from 2001:470:1f05:2c9:ad2f:8fee:48af:d42e, 64 hops max, 12 byte packets
1 2001:470:1f05:___::1 3.843 ms 2.304 ms 3.729 ms
2 2001:470:8122:___::2 2.702 ms 3.229 ms 3.116 ms
3 2001:470:8122:1:227:eff:fe26:3238 3.886 ms 2.440 ms 2.128 ms
Caveats and side effects
OS source address selection, in particular “use longest matching prefix” on dual-addressed machines can have undesirable effects with this policy routing.
In the case where a machine has multiple IPv6 addresses, RFC 6724 lays out a whole slew of rules to pick what local address is used when connecting to something on the Internet. Depending on the address of what you’re connecting to, traffic may go out via different route than you expect.
For example early on I was wondering why my speed tests from my laptop using Xfinity’s website weren’t showing any improvement. My laptop has both a HE address starting with 2001:: and an Xfinity address starting with 2601::. Lo and behold, www.xfinity.com resolves to `2001:559:19:6089::2af2` so MacOS is sourcing my request from my HE address, and thus my Speedtest was going through the HE tunnel and not over Xfinity native IPv6! Thus it was still hitting the original performance bottleneck and showing I could only do 400 mbit/s of IPv6. Temporarily removing the HE address, I was then able to hit 990 mbit/s of IPv6 using Xfinity natively. Quite a difference!
tl;dr anything that resolves to 2001::something may always go out HE.
