Is it safe for data integrity to use a “non ECC mini pc” that runs docker containers from the volumes of a “NAS with ECC ram”?

Or does the mini pc also require ECC ram for the data integrity?

Sorry if it is a noob question.

  • @PriorProject
    link
    English
    7
    edit-2
    1 year ago

    The answers in this thread are surprisingly complex, and though they contain true technical facts, their conclusions are generally wrong in terms of what it takes to maintain file integrity. The simple answer is that ECC ram in a networked file server can only protect against memory corruption in the filesystem, but memory corruption can also occur in application code and that’s enough to corrupt a file even if the file server faithfully records the broken bytestream produced by the app.

    • If you run a Postgres container, and the non-ecc DB process bitflips a key or value, the ECC networked filesystem will faithfully record that corrupted key or value. If the DB bitflips a critical metadata structure in the db file-format, the db file will get corrupted even though the ECC networked filesystem recorded those corrupt bits faithfully and even though the filesystem metadata is intact.
    • If you run a video transcoding container and it experiences bitflips, that can result in visual glitches or in the video metadata being invalid… again even if the networked filesystem records those corrupt bits faithfully and the filesystem metadata is fully intact.

    ECC in the file server prevents complete filesystem loss due to corruption of key FS metadata structures (or at least memory bit-flips… but modern checksumming fs’s like ZFS protect against bit-flips in the storage pretty well). And it protects from individual file loss due to bitflips in the file server. It does NOT protect from the app container corrupting the stream of bytes written to an individual file, which is opaque to the filesystem but which is nonetheless structured data that can be corrupted by the app. If you want ECC-levels of integrity you need to run ECC at all points in the pipeline that are writing data.

    That said, I’ve never run an ECC box in my homelab, have never knowingly experienced corruption due to bit flips, and have never knowingly had a file corruption that mattered despite storing and using many terabytes of data. If I care enough about integrity to care about ECC, I probably also care enough to run multiple pipelines on independent hardware and cross-check their results. It’s not something I would lose sleep over.

    • TheHolm
      link
      fedilink
      English
      11 year ago

      Using iSCSI will protect you to some degree from bit flips in client cache memory.

      • @[email protected]
        link
        fedilink
        English
        1
        edit-2
        1 year ago

        It might, I’m not familiar enough with it to have an opinion. I think the post was meant to be about remote filesystems, though, not block devices.

        Well, it doesn’t say at all, but it seems most likely to me 🙂

  • @[email protected]
    link
    fedilink
    English
    2
    edit-2
    1 year ago

    First, why is every post on this forum -1? Somebody must be holding a grudge.

    Second: it doesn’t matter. ECC just prevents bit flips in RAM, once data leaves a system it’s irrelevant whether it had ECC or not.

    I’ve been running servers of various kinds for decades. There is a difference between running servers on hardware with ECC vs none, but it’s not a big deal. Unless you’re running, like, banking software or something where accuracy or uptime is critical…I wouldn’t sweat it. You may just have to reboot cuz of a kernel panic once or twice a year.

  • @SheeEttin
    link
    English
    11 year ago

    Define “safe”. It’s not going to burn your house down if you don’t use ECC.

  • ZytaZiouZ
    link
    English
    11 year ago

    Disclaimer: I am an absolute amateur with self hosting. That being said, in my opinion containers/services you want to run can be split into two categories for this: things that benefit from data integrity and things that don’t.

    Examples: a Plex/Jellyfin server that is only reading memory does not benefit at all, as it doesn’t handle storing the data at all. PiHole has no need for ecc. Security camera DVR software can benefit from ecc, but in my opinion could be run without it for home use. Whatever NAS software you are running can benefit the most from ecc.

    If your NAS has enough ecc ram to run the docker containers or vms that can benefit, I personally see no harm in using a second pc for everything else, as long as it is cheaper than adding more RAM/better CPU, and running two systems does not dramatically increase electricity used.

  • poVoq
    link
    fedilink
    English
    01 year ago

    While ECC memory is nice to have, potential data integrity issues can be mitigated against by using a file system with sufficient redundancy and checksum error correction like zfs or btrfs.

    Just run a regular scrub for errors to be auto-corrected from the extra copies.

    • @chripede
      link
      English
      11 year ago

      If you were to use ZFS you really should consider ECC. Filesystem checksumming doesn’t do you any good if it’s calculated wrong because of mem error.

    • @[email protected]OP
      link
      fedilink
      English
      1
      edit-2
      1 year ago

      That would be the plan, the NAS with ECC would run zfs with weekly scrubs (4 to 6 drives)

      Edit: now running ECC on devices with critical data or databases