I’m pretty new to selfhosting, but one thing that I know to take seriously is log collection. Since there are a lot of different type of logs (kernel log, application logs, etc) and logs come in many different formats (binary, json, strings) - it’s no easy task to collect them centrally and look through them whenever neccessarly.
I’ve looked at grafana and tried the agent briefly, but it wasn’t as easy as I thought (and it might be a too big tool for my needs). So I thought to ask the linuxlemmy community to get some inspiration.
For a bit enhanced log file viewing, you could use something like lnav, I think it’s packaged for most distributions.
Cockpit can be useful for journald, but personally I think GUI stuff is a bit clunky for logs.
Grep, awk and sed are powerful tools, even with only basic knowledge of them. Vim in readonly mode is actually quite effective for single files too.
For aggregating multiple servers’ logs good ol’ rsyslog is good, but not simple to set up. There are tutorials online.