How do you manage the distribution of internal TLS network certificates? I’m using cert-manager to generate them, but the root self-signed certificate expires monthly which makes distribution to devices outside of K8s a challenge. It’s a PITA to keep doing this for the tablet, laptop and phones. I can bump the root cert to a year, but I’m concerned that the date will sneak up on me. Are there any automated solutions?
Some of my internal stuff goes out to Let’s Encrypt, so I don’t worry about it at all. My internal AD stuff is set for like three years. If anyone has compromised the CA, they’re already past where issuing malicious certs would be useful.
I would up your root cert expiration. You can keep the root CA offline if you’re concerned about compromise.
There are also ways to run LE-style automatic renewals internally, but I’ve never bothered because what I’ve described above means I don’t need it.
Bump root cert to 10 years and use intermediate with shorter lifetime. root cert should be stored and processed off net.
I think this is what I’m going to do.
For most of my internal services that are sitting behind Traefik I use step-ca which basically gives you a Let’s Encrypt style certificate while working over the local network. The root CA has a long expiry (so might not be what you want if your goal Is a short lived root CA) but the actual certificates for each service are short lived (a touch over 24 hours from memory?)
I tried step-ca to start with, but my primary use case was for certs in the cluster, which cert-manager is more suited for natively. Maybe step-ca has improved, I was using it in the early days. My goal isn’t a short lived cert as much as it is to have an easy configuration and to learn.
I looked into this recently.
There’s a DNS challenge designed for this exact scenario called (from memory) DNS01, but it’s more of a faff than I’m willing to get involved with.
Basically you push proof that you own the domain into a DNS record instead of to a file on a web server. It requires a DNS provider that has an API and a client that speaks that API.
It also leaks private DNS stuff into the public domain.
I’d love it if someone devised an easier way. Maybe there is an easier way?
I do the dns challenge with letsencrypt too but to not leak local dns names into the public I just run a pihole locally that can resolve those domains
The DNS server is only one thing you tell the domain, the other is the certificate authority. And those publish all issued certificates as part of certificate transparency. https://en.m.wikipedia.org/wiki/Certificate_Transparency
To mitigate the amount of published information, you can request wildcard certs to keep the subdomains private.
You can also use a wildcard cname entry to capture all subdomains and leave out the pihole faff, given that you use a reverse proxy that forwards to respective services by subdomain.
Yep I also run wildcard domains for simplicity
This is the way.
This is me.
In public dns, configure *.home.example.com as an A record pointing to the local IP for my traefik container.
Traefik then manages all certificates. It sets a TXT record with my dns providers API like privatesercice.home.lebowheatcroft.com, requests the cert from letsencrypt, then deletes that TXT record.
Yes the local IP of my server is leaked, but names of services are not.
Use a secret manager?
Cert is a secret, add a small agent to your containers that pings your secret manager and gets back the current cert. Then saves / imports it (or whatever is appropriate).
MONTHLY?? That’s a bit much, don’t you think?
If you’re regenerating certa that fast, I can’t think of anything that’s going to secure AND easy enough to satisfy automating this.
Whatever tool you want to use to secure the contents of the cert from its initial creation, to distribution, is fine enough. If you want super easy, use an SSH/SCP script. If you want something more elegant, think Hashicorp Vault or etcd.
Ansible is probably more effort than it’s worth (plus securing the secrets of the cert), and any other config mgmt tool won’t deal with the distribution portion simply, so I’d skip all of that.
Yes, monthly is too fast. I’m using a K8s operator for cert-manager which defaults to a month. I think I can patch the CSV with an annotation that will bump that out, but when the operator updates the CSV then I need to repatch it.
I was polling the community to see if there’s something that is easy to use but I was not able to find in my searches. It seems like a common problem.
Part of my problem is that I chose to use a K8s operator for cert-manager which isn’t easy to configure. Had I used a helm chart, i’d have bumped the root cert to 10 years and forgotten about it.
If the operator doesn’t allow it for some reason, uninstall it and try with the helm chart instead?
Or is there a reason to use the operator?
I think it may support it, but it’s not well documented. I’ll need to read up a bit. I started with helm charts but like how operators, um operate. They upgrade on their own and are very stable. Honestly, though, it was mostly because I wanted to learn how they work.
Yeah, either secure copy (SCP) or rsync are the ways to do this securely.
DNS challenge with a reverse proxy is that answer. I’ve been doing this for a while now and it works great. Most other answers here are work arounds or not very robust.
This is the way: https://youtu.be/liV3c9m_OX8
I do this with authentik for sso
I have local only things like vaultwarden and external things like seafile.
I started watching the video. I was not aware that LetsEncrypt supported wildcard certificates. Does this mean that your internal network uses the same domain name as your externally-hosted services?
Yes.
And
This is the best and most robust way to do this
Not sure if it’d fit your use case 100%, but this has been a nice middle ground solution for LE certs in my lab: https://www.certwarden.com/
Hadn’t heard of this before. I’ll give it a whirl.
Is *warden the new hotness? Bitwarden, vaultwarden, linkwarden and now certwarden?
Synching maybe?
