Service discovery across firewall zones

It is good practice to put untrusted devices into separate firewall zones on the home network. The most common instance of this is a separate guest firewall zone. The goal of such separation is usually that the untrusted devices can't connect to services hosted by the trusted devices on the lan zone. A separate zone for Wi-Fi IoT-devices is also a good idea for making sure such devices can't connect to the internet.

Such isolation leads to problems in everyday use though. A few examples:

• IoT devices (e.g. ESPHome) can't be connected to from the lan zone
• IoT devices can't be discovered from the lan zone
• A Chromecast device on the guest zone can only be connected to from inside the guest zone
• Peer-to-peer connections to devices in the guest zone can't be established

Some of these problems can be solved by allowing devices from the lan zone to establish a connection to the untrusted zones. The problems with discovery are more complicated to resolve though.

This guide will explain how to configure an OpenWRT or other Linux router so that devices in the lan zone can discover and connect to devices in the untrusted zones while keeping the untrusted devices isolated.

Disclaimer: Most of this is at the limit of my knowledge, I don't understand these topics much deeper than explained here and can't guarantee that the explanations are completely correct.

Allowing direct connections

Without any setup the firewall is configured to block all packets to other zones.

Direct connections to other zones can be allowed using the LuCI Firewall settings. The example configuration shown in the image allows devices from the lan zone to reach devices in the guest and the iot zone.

This will allow devices in the lan zone to establish a connection to the untrusted devices and to the router itself. The firewall connection tracking will then allow the untrusted device to send reply packets, but it will not allow untrusted devices to initiate a new connection.

Connection tracking works roughly by keeping track of the sources and destinations of the packets that have previously been allowed through the router. In this way the router tries to keep a list of all logical connections that have been established and will accept any packets belonging to such a logical connection, even if the firewall would otherwise block them. Identifying logical connections typically works by recording the protocol (TCP/UDP/…), source IP, destination IP, source port and destination port of a packet.

If you are able and willing to enter the IP-address of the untrusted device you want to connect to, and all the necessary connections are initiated by devices in the lan zone you can stop here. However most of the problems listed above require automatic discovery of devices. This is usually done using multicast or broadcast packets, but such packets are not forwarded to different firewall zones. The following sections will explain how to make common discovery protocols work.

Many of these configurations can't be done through LuCI, so you will need to do them over SSH.

mDNS

Multicast DNS is used for looking up hostnames on the local network and can also be used to discover specific services that are provided by a device. mDNS uses the IP address 224.0.0.251 and port 5353.

This IP can not be forwarded between networks, so the packets need to be processed and resent by some software on the router. The avahi-daemon is a software for processing the mDNS packets, it can be installed by running opkg install avahi-daemon. The avahi-daemon has a built-in functionality to answer discovery requests even when the discovered device resides in another zone.

To activate it edit /etc/avahi/avahi-daemon.conf and add the following configuration:

[server]
allow-interfaces=<lan>,<guest>

[reflector]
enable-reflector=yes

allow-interfaces selects on which interfaces the avahi-daemon should listen. By default it listens on all interfaces. This includes the WAN interface. By explicitly defining this option we exclude the WAN interface. enable-reflector=yes enables the functionality of the avahi-daemon to answer the discovery requests even when they are for a different interface.

The configuration file expects interface device names here, so <lan> and <guest> should be replaced by the respective interface devices of that firewall zone. For lan this is commonly br-lan while for guest it could be br-guest or perhaps phy0-ap1 if it only consists of a Wi-Fi network and has no bridge interface. You can find the correct interface device name by looking through the Network > Interfaces view in LuCI or by running the command ip address on your router and finding the interface device which has the IPs assigned to that zone.

In order for the device advertisements in the guest zone to reach the router, add a firewall rule allowing packets to the router to 224.0.0.251 on UDP port 5353 from the guest zone. The rule should read like this:

Incoming IPv4 and IPv6, protocol UDP, from guest, port 5353, to this device, IP 224.0.0.251 or ff02::fb, port 5353 > Accept input

Beware that this will also allow devices in your untrusted zones to discover devices in your lan zone, but the firewall will still stop untrusted devices to connect to them.

The devices in the lan zone are already allowed to connect to the router, due to the Input rule from the section on Allowing direct connections.

Credit to Chris Smart for providing this solution. Read their blog post for a more detailed guide on configuring the avahi-daemon in OpenWRT: https://blog.christophersmart.com/2020/03/30/resolving-mdns-across-vlans-with-avahi-on-openwrt/

SSDP / UPnP discovery

SSDP is a common protocol for discovering services on the local network. It is also the protocol used by UPnP for discovering other services. If a device wants to discover nearby services it sends out a UDP-multicast packet to the IP-address 239.255.255.250 on port 1900.

Multicast and broadcast traffic will stay in the network it originated from. This is not because the firewall blocks the traffic, but because they are not directly addressed to the other network. Normally this is a good thing, otherwise every multi-/broadcast packet would reach the whole world, but we want to route some specific packets to some predefined zones. To do this, the Linux kernel can be configured to forward certain multicast packets through to another interface.

SMCRoute

SMCRoute is a daemon for registering static multicast routes with the Linux kernel. It can be installed on OpenWRT by running opkg install smcroute. Which multicast packets should be routed where can be configured in /etc/smcroute.conf.

mgroup from <lan> group 239.255.255.250
mroute from <lan> group 239.255.255.250 to <guest>

Just as in the avahi-daemon configuration, this file expects interface devices names and not the names of the firewall zones. So use the interface device you figured out in that step here as well. As a reminder these are usually named br-lan or phy0-ap1.

After updating the configuration file, reload it by running /etc/init.d/smcroute reload. You should also consider adding /etc/smcroute.conf to the backup file list under System > Backup > Configuration in LuCI so that it persists across firmware upgrades.

This configuration first creates a new multicast group for the IP address 239.255.255.250 and then configures such packets to be routed from the <lan> interface to the <guest> interface. By default SSDP packets have a TTL of one, meaning they are discarded as soon as they are routed. Because we now want to route these packets we need to increase their TTL to at least two. This can be done by adding a custom firewall rule. Create a new file /etc/config/firewall.user with the following content:

# increase multicast ttl

nft add rule inet fw4 prerouting \
  iifname "<lan>" ip daddr 239.255.255.250 udp dport 1900 \
  ip ttl set 2

# iptables -t mangle -A PREROUTING \
#   -i "<lan>" -d 239.255.255.250 -p udp --dport 1900
#   -j TTL --ttl-inc 1

I will list the nft commands as well as the older iptables commands. Replace the <interface> placeholders by your interface name here as well.

Then configure OpenWRT to load this file by modifying the /etc/config/firewall configuration and adding the following section:

config include 'user'
        option type 'script'
        option path '/etc/config/firewall.user'
        option fw4_compatible '1'

Then reload the firewall by running /etc/init.d/firewall reload.

The SSDP packets to the multicast destination IP should now reach the devices in the guest zone, but reply packets are still blocked by the firewall because it can't establish a connection tracking relationship to the request packet: The lan client sent the packet to the SSDP ip, but the reply packets are sent directly from guest.2. Because the reply is sent from a different IP address than the one the request was sent to the packets look unrelated to the router.

Allowing replies

To allow replies to multicast packets, the reply packets must be accepted separately from the connection tracking. To do this we will create a firewall set. Firewall sets record tuples of some data from a packet and can be configured to automatically remove these entries after a few seconds. We will use a firewall set to record which clients have recently sent an SSDP request packet, and then allow replies to the recoded clients.

Add these rules to the /etc/config/firewall.user file that you created before.

nft add set inet fw4 ssdp \{ \
  type ipv4_addr . inet_proto . inet_service\; \
  timeout 3s\; \
\}

# ipset create ssdp hash:ip,port timeout 3 -exist

This will create a new set called ssdp which stores tuples of IPv4 address, protocol and port. The entries in the set will timeout after three seconds, which should be more than enough time for the devices to reply.

nft insert rule inet fw4 forward \
  oifname "<guest>" ip daddr 239.255.255.250 udp dport 1900 \
  add @ssdp \{ ip saddr . ip protocol . udp sport \}

# iptables -I FORWARD \
#   -o "<guest>" -d 239.255.255.250/32 \
#   -p udp -m udp --dport 1900 \
#   -j SET --add-set ssdp src,src --exist

This rule will record the source IP, protocol and port of the packages that have been forwarded to the guest zone on port 1900. The firewall set will then contain the IP of the lan device that has sent the ssdp discovery packet.

# accept forwarded packets if their destination ip and
# destination port are in the ssdp set

nft insert rule inet fw4 forward \
  iifname "<guest>" ip daddr . ip protocol . udp dport @ssdp \
  accept

# iptables -I FORWARD \
#   -i "<guest>" -p udp -m set --match-set ssdp dst,dst \
#   -j ACCEPT

This is the rule that accepts the reply packets from the guest zone. It tests the packets received from there and tests their destination IP, protocol and port against the entries in the firewall set. If a match is found the packet is accepted.

Credit to Pali on ServerFault for providing this solution: https://serverfault.com/a/911286

UDP Broadcasts

Some older devices discover services on the network by sending UDP broadcasts to the IP 255.255.255.255 on a protocol-specific port. SMCRoute can't forward these because it is only intended for managing multicast routes in the kernel and not for forwarding broadcasts.

These packets need to be forwarded by repeating them on the other interfaces, similar to the mDNS packets. udp-broadcast-relay-redux is a software which can be configured to forward arbitrary UDP broadcast protocols. Install it by running opkg install udp-broadcast-relay-redux.

Edit the configuration file at /etc/config/udp_broadcast_relay_redux and add a configuration like this. This is a configuration example for the LIFX protocol on port 56700, you can find configuration examples for other services in the README of udp-broadcast-relay-redux.

config udp_broadcast_relay_redux
       option id 1
       option port 56700
       list network lan
       list network guest

This file expects the OpenWRT interface names, not the interface devices as before.

You will need another set of firewall rules that are very similar to the ones you created to allow SSDP replies above, which will work in the same way. Add these rules to the /etc/config/firewall.user file that you created before.

nft add set inet fw4 udp_broadcast \{ \
  type ipv4_addr . inet_proto . inet_service\; \
  timeout 3s\; \
\}

nft insert rule inet fw4 \
  output oifname "<guest>" \
  ip daddr "<broadcast ip>" ip protocol udp \
  add @udp_broadcast \{ ip saddr . ip protocol . udp sport \}

nft insert rule inet fw4 forward \
  iifname "<guest>" \
  ip daddr . ip protocol . udp dport @udp_broadcast \
  accept

Here <guest> should be the interface device as before and <broadcast ip> should be the broadcast IP of the guest zone, for example 192.168.102.255.

Then reload the firewall /etc/init.d/firewall reload and start the udp-broadcast-relay-redux /etc/init.d/udp-broadcast-relay-redux start. I had to fix a bug in its initscript by commenting out the procd_add_jail ubr-${PIDCOUNT} line for it to start successfully. If you want you can start udp-broadcast-relay-redux manually using this is the command:

udp-broadcast-relay-redux --id 1 \
  --port 56700 --dev br-lan --dev phy0-ap1 -d

Debugging with tcpdump

If you are stuck in some step and can't figure out why the communication with a device does not work it is useful to see where the packets get dropped. A good way to do that is by using tcpdump on the router.

By default tcpdump shows all packets which is a bit overwhelming, you should at least make sure not to include the traffic to your SSH session :). The most common arguments to filter the packets that are shown are

• -i <interface> to only show packets received on or sent to a particular interface
• tcp or udp to filter by protocol
• port <port> to only show packets for a particular port
• host <host> to only show packets to or from a particular device

With this you can see where packets get lost, or if the IP where packets get sent to is the correct one.