lemmy.one has been down for a while now. I’ve personally noticed that it hasn’t been working for over 24 hours, some comments suggest it may be longer than that:

lemmy.one is one of the larger Lemmy instances, with around 7.3k users.

The site does load, however it displays a Lemmy error. This shows that the Lemmy UI is still there and working, but the Lemmy back-end server process is not.

Based on this, it doesn’t look similar to the disappearance of vlemmy.net and it’s most likely just a technical issue.

So what is going on?

My guess is there are 3 likely scenarios for what went wrong:

  • The Lemmy server process crashed (it’s known to happen from time to time)
  • This is an update gone wrong (the latest Lemmy update in particular is tricky as it requires a DB migration).
  • Some other mysterious event messed up the DB

This is a very long time to be struggling with a technical issue and the lack of communication is concerning.

It could be that:

  • The admin is just not available to deal with it right now due to real life reasons.
  • The DB is now a mess and the Lemmy server process can’t be restarted.

I expect they have regular DB back-ups like everyone else and would’ve rolled back to a previous snapshot by now. I think the most likely scenario is that the admin just isn’t available to deal with it.

To fix this requires actual server access, so even though lemmy.one had several Lemmy admins, it’s unlikely that all of them have the required access to deal with this issue.

It would be interesting to hear if anyone managed to get in touch with lemmy.one’s admin.

Edit: It’s back: https://lemdit.com/post/294679

  • @lemannEnglish
    3•1 year ago

    I’m locked out of my .one account too :(. It’s been down since around Thursday for me, same as the user in the pic. The admins, or at least Jonah seems pretty hands-off, so I just think they aren’t aware the instance is down…

    When you hit the API directly it indicates some kind of DB issue: https://lemmy.one/api/v3/community/list?sort=New

    Error occurred while creating a new object: the database system is not yet accepting connections

    If you refresh that page, the error occasionally changes to this

    Error occurred while creating a new object: the database system is in recovery mode

    Edit: Hopefully off the back of this incident they set up some monitoring, whether its zabbix/prtg/uptimerobot or something. Having the DB suddenly conk out while everything else hums away smells a lot like full storage IMO