edit: you are right, it’s the I/O WAIT that it destroying my performance:
%Cpu(s): 0,3 us, 0,5 sy, 0,0 ni, 50,1 id, 49,0 wa, 0,0 hi, 0,1 si, 0,0 st
I could clearly see it using nmon > d > l > - such as was suggested by @SayCyberOnceMore. Not quite sure what to do about it, as it’s simply my sdb1 drive which is a Samsung 1TB 2.5" HDD. I have now ordered a 2TB SSD and maybe I am going to reinstall from scratch on that new drive as sda1. I realize that’s just treating the symptom and not the root cause, so I should probably also look for that root cause. But that’s for another Lemmy thread!

I really don’t understand what is causing this. I run a few very small containers, and everything is fine - but when I start something bigger like Photoprism, Immich, or even MariaDB or PostgreSQL, then something causes the CPU load to rise indefinitely.

Notably, the top command doesn’t show anything special, nothing eats RAM, nothing uses 100% CPU. And yet, the load is rising fast. If I leave it be, my ssh session loses connection. Hopping onto the host itself shows a load of over 50,or even over 70. I don’t grok how a system can even get that high at all.

My server is an older Intel i7 with 16GB RAM running Ubuntu22. 04 LTS.

How can I troubleshoot this, when ‘top’ doesn’t show any culprit and it does not seem to be caused by any one specific container?

(this makes me wonder how people can run anything at all off of a Raspberry Pi. My machine isn’t “beefy” but a Pi would be so much less.)

  • @[email protected]
    link
    fedilink
    English
    49
    edit-2
    1 year ago

    It’s sounds like it could be an IO wait issue, system load will climb a ton without showing much CPU usage.

    Make sure you’re not running out of RAM and going into swap space, it doesn’t sound like it though.

    iotop might show something useful. And in htop you can add the 'PERCENT_IO_DELAY" column which can be useful.

    • @PriorProject
      link
      English
      19
      edit-2
      1 year ago

      My money is also on IO. Outside of CPU and RAM, it’s the most likely resource to get saturated (especially if using rotational magnetic disks rather than an SSD, magnetic disks are going to be the performance limiter by a lot for many workloads), and also the one that OP said nothing about, suggesting it’s a blind spot for them.

      In addition to the excellent command-line approaches suggested above, I recommend installing netdata on the box as it will show you a very comprehensive set of performance metrics without having to learn to collect each one on the CLI. A downside is that it will use RAM proportional to the data retention period, which if you’re swapping hard will be an issue. But even a few hours of data can be very useful and with 16gb of ram I feel like any swapping is likely to be a gross misconfiguration rather than true memory demand… and once that’s sorted dedicating a gig or two to observability will be a good investment.

    • @[email protected]
      link
      fedilink
      English
      81 year ago

      And I know OP mentioned not using much ram, but almost everytime I see a server load that high, it’s usually because the server is swapping heavily causing the iowait.

      • @[email protected]
        link
        fedilink
        English
        21 year ago

        Does top show unpaged memory too? I’ve had an application with a memory leak before that would fill up unpaged memory and it would look like nothing was using ram when I looked in the task manager, even though usage was 99%.

    • thelastknowngod
      link
      fedilink
      English
      31 year ago

      Yep. IO.

      OP, this might be overkill for you but it might be worth standing up a grafana/prometheus stack… You’d be able to see this stuff a lot faster and potentially narrow in on a root cause.

      • @PlutoniumAcidOP
        link
        English
        11 year ago

        That is definitely an interesting idea! Much, much better than the stupid dashdot container I am running now :-D

  • Neuromancer
    link
    fedilink
    English
    171 year ago

    run top and paste the output the top portion of the screen.

    I would suspect it is IO wait. You can get into disk contention if you have multiple containers fighting for disk. You will notice the IO queue is building up and that is shows you are waiting for IO transactions.

    %Cpu(s): 67.4 us, 13.0 sy, 0.0 ni, 19.4 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st

    See the field labeled WA, that is wait time. Basically time you are waiting for IO to complete.

    If that is high, you can increase the cache used by Linux BUT if the system crash you are at risk of losing saves.

  • Black Xanthus
    link
    English
    12
    edit-2
    1 year ago

    The last time I saw this was on a slow-failing HDD.

    Check a quick fsck might get you a few answers. You can find more info in the Linux manual. It could just be one or two bad blocks that you can recover and fix the problem (though, ofc, it’s time to backup your data).

    The other, slightly unusual time I’ve seen it is with mixed RAM. 16gb made of 2x6g and then 2x4gb did some real odd things to the system. If it’s not the disk, and your box will boot with one stick of ram, try it to see if it fixes the issue. It could be that your RAM speeds are off (or your like me and just put two sticks you had lying around, and it basically worked until it didn’t).

    An outlier, that I’ve not seen on modern machines is io/wait for a CD-ROM to spin up, even if your not accessing the CD-ROM. Normally caused by bad cabling. Based on the age of your machine, this is unlikely, but it might be worth unplugging devices to see if one is bad and not reporting properly.

    This is, if course, assuming dmsg is empty

    Final thought: see if your running SELinux. If you are, turn it off and try again. Those policies are complex, and something installed in a non-standard place could be causing SELinux to slow IO as it fills your logs with warnings.

    Hope that helps,

    • ActuallyRuben
      link
      fedilink
      English
      21 year ago

      To add on to this, if you’re using some random RAM stick picked out of the gutter, then it might be worth it to run memtest86+. Bad RAM sectors can give some weird unpredictable issues.

    • @PlutoniumAcidOP
      link
      English
      11 year ago

      Do not run fsck on a mounted device

      So how do I run this on /dev/sda? I can’t very well unmount the OS drive…

  • @[email protected]
    link
    fedilink
    English
    111 year ago

    “load” is not “CPU usage.” It’s “system usage” and includes disk and network activity. Including swapping if you’re low on memory.

    vmstat can tell you what your disk io looks like. Iotop can help with narrowing it down to a process.

    • @[email protected]
      link
      fedilink
      English
      61 year ago

      It’s a bit more complicated than that. System load is a count of how many processes are in an R state (either "R"unning or "R"eady). If a process does disk I/O or accesses the network, that is not counted towards load, because as soon as it makes a system call, it’s now in an S (or D) state instead of an R state.

      But disk I/O does affect it, which makes it a bit tricky. You mentioned swapping. Swapping’s partner in crime, memory-mapped files, also contribute. In both of those cases, a process tries to access memory (without making a system call) that the kernel needs to do work to resolve, so the process stays in an R state.

      I can’t think of a common situation where network activity could contribute to load, though. If your swap device is mounted over NFS maybe?

      Anyway, generally load is measuring CPU usage, but if you have high disk usage elsewhere (which is not counted directly) and are under high memory pressure, that can contribute to load. If you’re seeing a high load with low CPU utilization, that’s almost always due to high memory pressure, which can cause both swapping and filesystem cache drops.

  • @[email protected]
    link
    fedilink
    English
    71 year ago

    many people aren’t running containers on RBpi … while feasible, it was notoriously poor until the 8GB pi4, and still is easily bounded by SD card I/o. are there docker stats so you can see the disk + net I/o of each container?

  • SayCyberOnceMore
    link
    fedilink
    English
    61 year ago

    Install nmon - it’s a CLI tool to show system load Run it and press ‘d’ to show disk usage, then ‘l’ to show a longterm graph, then ‘-’ to speed it up. If your storage is the issue you’ll see it here - and potentially which drive(s) are affected.

  • @[email protected]
    link
    fedilink
    English
    51 year ago

    Immich and photoprism do AI detection and sorting right? Until they have scanned through all current photos you are going to have a lot of system load. And my 4 gig pi usually jusy runs at 1 gig of memory and low load…no Immich, but running Openmediavault for DAAP and SAMBA, and DejaDup and Syncthing, homeassistant, CUPS, etc.

    • @[email protected]
      link
      fedilink
      English
      11 year ago

      I ran immich on a server that’s substantially faster than a raspberry pi, and after about a day and a half it keeps me from getting in with ssh. Even locally, I have to wait for a long time to get a login prompt.

  • wwwwhatever
    link
    fedilink
    English
    31 year ago

    Check the wa level in top, if it is high the system is waiting for hardware to process stuff. If it is high, check with atop of disks are red.

    In such cases I almost always see some hardware but failing, networkcard or switch falling, harddisk/NFS stuff falling, memory falling. Hope this helps

  • @[email protected]
    link
    fedilink
    English
    31 year ago

    I’d try each application one by one. Maybe write a script to monitor load and stop the program if it goes past your desired threshold and notify you.

    It could also be a setting in some app like photoprism or immich … I think one of them uses tensorflow to classify images. That would increase the load if thats running in the background.

    Maybe try them with an empty directory so there is no data to process and see if you encounter the error. Then add some data and see how the load is.