Hello everyone,
I have an issue with my desktop, which I really don’t understand.
It is a desktop PC, with a Nvidia GPU, which I use for development (of AI, but like the image analysis part, not the slop part). I usually let it always run, and my previous machine was working fine like this.
However, on this machine, I get crashes after either a few days or a few weeks.
These “crashes” are weird : the machine is still up (and I can ping it), however I can’t ssh into it, I can’t connect to cockpit either, and can’t login physically on it (the display is frozen).
The only way to get it back up is to forcefully shut it down, wait a little and then powr it back up; but this isn’t ideal, since it interrupts computations etc
The only crashes shown in journalctl are related to irrelevant (I believe) programs : KDEConnect, DrKronqi, etc; nothing right before the crash.
Do you have any idea what could cause this ?
I tried logging temperatures, but nothing abnormal showed up. I ran memtest86+, nothing showed up either. Smart tells me everything is alright for my drives as well.
Right now I’m thinking about changing the alimentation or the motherboard, but frankly I don’t know what’s going on and it’s really puzzling me.
Well drkronqi is a crash handler, and KDE connect has its fingers in multiple pies? You say theyre not relevant but who do you know when the crash is, if you only notice it when you try to use the PC and its unresponsive? I wouldn’t dismiss the timing of these crashes if you can’t say exactly when it stopped functioning. They may even by symptoms if not the actual cause.
I’d start by increasing the logging levels to make them more detailed. You change the kernel and system logs to make them more detailed - how will differ slightly on your distro but essentially you want debug level system logs and lowest severity kernel logs. You can also setup some extra logging such as outputting system memory and vram usage to a dedicated file every second or so. This will.give you more information to hunt for the culprit.
As others have said, my first thought from your useage and description is a memory issue. That could be RAM or vram - especially with the AI useage. I’d start by logging both to their own files; out of memory can crash without much trace if the logs aren’t tracking the right things.
Also whatever process you are using - your AI tool for example - get that logging too.