Hey yall, just wanted to chime in with a quick update on why we were down and have gone down a few times these past few days.

So… for those who don’t know a bit on the backend way of how lemmy works right now, it’s a lil more than a mess. Lemmy itself uses very little computing resources, you could probably run a decent instance on a toaster (I wouldn’t suggest it though) but the problem that arises rapidly is the way photos are managed. The photo database downloads images both that are uploaded and federated. There is an option to disable caching for federated NSFW posts by unchecking enable NSFW within the instance admin panel. That’s the only option related disabling federated images as of right now.

You see where I’m going with this, with this instance being as large as it is? Our photo database has been filling up pretty rapidly. We are attempting to troubleshoot solutions to prevent caching all federated content. Unfortunately as our backend team is attempting to figure that out, our pictures database just keeps ballooning more and more.

We have deleted some stuff in hopes to give more time, but we unfortunately have no choice but to migrate to object storage ASAP. Tomorrow there will likely be downtime as we migrate things over, and unfortunately there is no telling how long that will take.

Thank you for being patient and understanding while we navigate this. If you want to see updates to our status when we go down, check out our mastodon page here or come to our matrix public operations channel for updates (link in the sidebar.)

Things should work today though. Emphasis on should.

We are working on open collective still, but hosting this instance does cost money. If you are able to, donations are greatly appreciated. Especially as we will have to factor in object storage into our costs now.

  • @[email protected]
    link
    fedilink
    English
    641 year ago

    Yeahhhhh you don’t want to store images in a database. That will kill the database in a hurry. Store references to object storage.

      • @[email protected]
        link
        fedilink
        English
        181 year ago

        MSSQL has had blob storage for a while and I don’t know anyone that uses it. Their Filestream stuff was interesting by letting you store it in the FS but be indexes via the DB, but come-on, use a damn pointer people.

        • @[email protected]
          link
          fedilink
          English
          21 year ago

          My company stores some 80+ TB in a series of MSSQL databases. Yes, you read that right. In. Each DB is allowed to grow to 250GB, then a new one is created and written to while the old one becomes read only.

          • @[email protected]
            link
            fedilink
            English
            21 year ago

            That is horrific. In my day job, I stored images in a database in my first job, learnt my lesson and now have an allergic reaction just thinking about it.

      • @[email protected]
        link
        fedilink
        English
        71 year ago

        That is interesting. Oh well, I’ve met developers who didn’t always make the best design decisions for the backend until their ass has had chunks removed.

      • @[email protected]
        link
        fedilink
        English
        51 year ago

        You haven’t clarified yet as to whether or not you were storing the actual image data in the database…

  • @[email protected]
    link
    fedilink
    English
    291 year ago

    Just remove direct uploaded pictures, there are a number of image hosting sites that already exist that are both setup for this kind of thing and that people already use and link to.

    This is already the case for videos for the very same reason.

    • @[email protected]
      link
      fedilink
      English
      361 year ago

      The problem with a 3rd party hosting site is that there’s no guarantee they suddenly won’t go out of business, or hide or delete all their nsfw content like imgur and Tumblr and many others in the past.

      Just hosting images for another site like this one isn’t profitable because they can’t sell ads for the hotlinked images served to users of that other site.

      It is a conundrum to find a working long term solution.

      • @[email protected]
        link
        fedilink
        English
        16
        edit-2
        1 year ago

        One of the worst things when it comes to porn is videos or pictures being deleted. Especially frustrating to browse old forums and none of the download links work or none of the images show. Or when pornhub deleted nearly 80% of their videos. There are videos out there which I am still looking for. Probably will never find again. Partly the reason I am a hoarder.

        • @[email protected]
          link
          fedilink
          English
          91 year ago

          I completely agree.

          The models retire, the photographer or rigger retire, or shut down because of piracy.

          Torrents and download links disappearing.

          Fuck pornhub for multiple reasons.

          It’s such a shame.

          Support the artists while you can, because that’s what they are, and preserve their works for posterity.

        • Arobanyan
          link
          fedilink
          English
          41 year ago

          Yeah that was such a heartbreak, like over half the website just disappeared over false allegations and so much really popular amateur content gone forever was deleted for no reason in the purge.

      • @[email protected]
        link
        fedilink
        English
        111 year ago

        It really isn’t a problem when we’re talking about these kinds of images. This isn’t a type of content that if it disappears in 12 months anybody is going to feel great loss for. It’s not instructions on how to maintain some esoteric computer system, or some sort of critical cultural records. It’s a transient commodity, with a steady stream being generated all the time. Nobody is going back and browsing the stuff that was posted 5 years ago.

        Anyone who wants to keep such images for extended periods is able to save them locally to their computer as they get posted.

        • @Dark_Blade
          link
          English
          161 year ago

          You underestimate the average porn addict. People were downloading years of porn they’d ‘saved’ on Reddit while making their escape.

          • @[email protected]
            link
            fedilink
            English
            61 year ago

            The save option is still there.

            Also, This lemmy instance is actually more likely to fail right now than most of these established image sharing sites.

          • @Blamemeta
            link
            English
            01 year ago

            I did that. I have like 5 comics from when imgur started deleting porn.

      • Kyle
        link
        fedilink
        English
        41 year ago

        The only real solution is to pay to host images. Nobody will accept though so we end up with the issues you describe.

  • @[email protected]
    link
    fedilink
    English
    131 year ago

    Please tell me you’re not putting the images actually /IN/ the database? Images/files don’t belong in the fucking database(we have had customers try to do this and it’s the biggest headache to get fixed). You only put LINKS to files in the database.

  • Altima NEO
    link
    fedilink
    English
    101 year ago

    Do what needs to be done, boss! We’ll be here waiting for you.

    • @[email protected]
      link
      fedilink
      English
      151 year ago

      Bad idea having the photos and content stored in your paostgres DB. Better on a separate product like HCP or some other object storage platform

  • @hohoho
    link
    English
    81 year ago

    Oooh neat! Which object storage are you guys using? Ceph? MinIO? Something else? Would love to discuss gory details!

  • Nato Boram
    link
    fedilink
    English
    61 year ago

    It would be interesting if hosting images via IPFS was an option, but that would require pretty heavy changes on Lemmy’s side.

  • @[email protected]
    link
    fedilink
    English
    61 year ago

    I can’t believe you’re only just moving to object storage! Offload that stuff as much as possible (this is probably a grandmother suck eggs situation, I know).

    When you say “photo database”, do you mean Lemmy literally stores the binary image files in the db, or just the storage area for images (ofc linked to from the db)?

    • @[email protected]OPM
      link
      fedilink
      English
      61 year ago

      By default lemmy as a software, the whole photo database is stored directly into a file folder that links itself to the drive. The entire photos. And of course the drive just rapidly inflates like a balloon since it stores not just local images but federated images

  • db0
    link
    fedilink
    English
    6
    edit-2
    1 year ago

    Note if you’re migrating pictrs you don’t need a whole lemmy downtime. Just pictrs

      • db0
        link
        fedilink
        English
        21 year ago

        Also it will be way more time than you think. The migration is slooooooow.

        • @[email protected]M
          link
          fedilink
          English
          31 year ago

          25GB in 4 hours. Looks like it will take a few days 😕 I asked it to the Matrix chat, hopefully they have a solution.

  • @[email protected]
    link
    fedilink
    English
    51 year ago

    Thanks for the update and the openness. Hope you are able to make it work with a minimum of fuss, and that all the users understand. Good luck!

  • @[email protected]
    link
    fedilink
    English
    51 year ago

    You might want to look at storj.io for object storage. We’ve been using it in production at work and are really satisfied. It’s priced pretty well, highly available, high throughput, multi regional by default and s3 compatible.

  • Bootywind
    link
    fedilink
    2
    edit-2
    1 year ago

    I’m locked out of my NSFW account because of the 2FA. I toggled the setting on/off yesterday trying to fix the repeated login issue and was able to log in afterwards without. Now today it’s blocking access. @yay