What kind of ISP are you dealing with?

And maybe PPPoE.

traceroute --mtu 1.1.1.1

Pick the lowest value displayed for F=xxxx like e.g F=1492 and subtract 80.

For my DSL connection the optimal value is 1412.

nonfree drivers accessible right away

Non-free firmware is included in the Debian installer since Bookworm.

Do you really know how Wireguard works?

Updating without a reboot only works for wireguard-go. The default implementation runs in the kernel. An update to it would require kernel live patching.

Wireguard doesn’t answer to unsigned packets. Using obscure ports or even port knocking is rather pointless. It’s indistinguishable from a closed port.

I’d rather take Casaos out of the equation and target Ubuntus’ Wireguard stack instead.

Jellyfin is completely free. I only used it shortly in my LAN environment so I can’t give you any numbers. It should roughly be in the same ballpark as plex though.

You can skip fail2ban for SSH. I missed the important bit. Duh…

Never used Plex but had a good experience with Jellyfin.

Just a few thoughts:

• don’t cheap out. Building your whole stack on top of free or ultra budget providers is going to backfire eventually
• check the traffic limits if you want to stream 4k content from your NAS
• if latency and bandwidth is a concern, you need to select a VPS provider with good peering. This fully depends on your ISP.
• i’d recommend setting things up with split DNS. Your DNS server would answer with local IPs for queries from within your LAN and with the IP of the VPS for external queries.
• take a look at AdGuard Home
• you can skip fail2ban if you go straight for ssh keys
• 100% wireguard

Did you enable forwarding via sysctl?

sysctl net.ipv4.ip_forward

This should report 1

You only need the masquerade rule.

iptables -t nat -A POSTROUTING -s 10.11.13.0/24  -o enp3s0 -j MASQUERADE