Here is my quick list for outward facing servers:

1. Isolate them from your main network. If possible have then on a different public IP either using a VLAN or better yet with an entire physical network just for that - avoids VLAN hopping attacks and DDoS attacks to the game server that will also take your internet down;
2. If you’re using VLANs then configure your switch properly. Decent switches allows you to restrict the WebUI to a certain VLAN / physical port - this will make sure if your server / IoT devices are hacked they won’t be able to access the Switch’s UI and reconfigure their own port to access the entire network. Note that cheap TP-Link switches usually don’t have a way to specify this;
3. Only expose the game server service to the Internet. Everything else such as SSH, configuration interfaces and whatnot can be moved to another private network and/or a WireGuard VPN you can connect to when you want to manage the server;
4. Use custom ports with 5 digits for everything - something like 23901 (up to 65535) to make your service(s) harder to find;
5. Disable IPv6?
6. Use nftables / iptables / another firewall and set it to drop everything but those ports you need for the game sever and the management VPN access to work;
7. Use your firewall to restrict what countries are allowed to access your server. If you’re just doing it for a few friends only allow incoming connection from your country (https://wiki.nftables.org/wiki-nftables/index.php/GeoIP_matching)

Realistically speaking if you’re doing this just for a few friends why not require them to access the server through WireGuard VPN? This will reduce the risk a LOT and won’t probably impact the performance. This is a decent setup guide https://www.digitalocean.com/community/tutorials/how-to-set-up-wireguard-on-debian-11 and you might use this GUI to add/remove clients easily https://github.com/ngoduykhanh/wireguard-ui.