Are there any Discord servers or somewhere in the Matrix to chat about hosting a Lemmy instance? I’ve got Lemmy running, but I think there are several of us in the same boat struggling with federation performance issues and it might be good to have some place to chat real time.
I was able to adapt the docker compose manifest into a nomad job(yay high availability), but I am really struggling with federation. I have a domain/proper ssl certificate, accessible remotely everything seems OK, but when I try to subscribe to other instances, I get an initial load of posts, then it’s just stuck in subscribe pending. Any time I try to subscribe I see this log message which isn’t exactly helpful about what to do about it…
‘ 2023-06-19T20:11:18.426743Z INFO Worker{worker.id=06aa9ebe-1cab-42fb-ac4b-54bbe7954ba2 worker.queue=default worker.operation.id=fe75d47d-f50d-43d6-921f-795aa50a1b68 worker.operation.name=process}:Job{execution_id=83235752-79dd-4e42-a6f5-d6e32c2e95a9 job.id=ed8bcdbd-4e78-464e-9ae0-871f3d79fd92 job.name=SendActivityTask}: activitypub_federation::core::activity_queue: Target server https://lemmy.ca/inbox rejected https://lemmy.my-domain-redacted.ca/activities/follow/c4b74591-767e-42a0-a160-5023e67c77aa, aborting’
FWIW i see that too on several instances. I dont think it affects anything but syncing with busier instances is a struggle and the destination instance is not acknowledging you following it.
For example if you look at my subscribed you will see the following
But when i go to that page i get the following
And when i click to go the the instance page I get a 500 error, then it works on refresh. And my comments are clearly struggling to stay in sync.
TL:DR - I think some of the larger instances are overloaded its not just user counts and traffic, but also all the backend requests to sync with other instances which may be silently failing/timing out.
OK I was able to resolve this issue on my end, I had setup an internal consul based url during setup and after changing the config it looks like the old url was still lingering in config/db in a bunch of the json responses from the test urls in the documentation. After dropping everything from postgres and re-initializing I seem to be as far as everyone else (some can subscribe, others stuck pending, and no comments :D )