[Solved] Trouble setting systemd timer

calm.like.a.bomb · edit-2 7 months ago

[Solved] Trouble setting systemd timer

@somethingsomethingidk · 7 months ago

Can you share your service file?

calm.like.a.bomb · 7 months ago

[Unit]
Description=Run rclone up for b2

[Service]
Type=oneshot
ExecStart=/zet/Users/radu/backup/rclone_up.sh

[Install]
WantedBy=rclone-up.timer

@somethingsomethingidk · 7 months ago

You could try setting TimeoutStopSec=“infinity” for the service. There may be a default timeout for services and its killing rclone before it can finish because the oneshot type is considered “starting” until the program exits.

@[email protected] · 7 months ago

TimeoutStopSec applies to the ExecStop command, TimeoutStartSec would be the culprit here. I’m not sure why there would be a default timeout of specifically 1:39 minutes though.

@somethingsomethingidk · 7 months ago

Yeah you’re right, it’s time for bed lol

The default start timeout is disabled by default for oneshot.

@[email protected] · edit-2 7 months ago

Does your script fork at some point (and might exit before the rsync job is completed)? ~~Because then you need to use Type=forking instead of simple or oneshot, otherwise systemd will start trying to clean up child processes when the script exits.~~

Edit: Actually considering the time span involved Type=forking will not solve your issue because it will timeout, if this is the problem you need to change your script to not do that.

calm.like.a.bomb · 7 months ago

No, my script doesn’t fork and I don’t think rclone does that either.

Here’s the script (pretty simple):

#!/bin/bash

repos=(fotorepo multirepo persorepo appconfigs)

if pidof -x rclone >/dev/null; then
  echo "Process already running"
  exit 1
fi

for repo in "${repos[@]}"; do
    inidate=$(date +'%Y%m%d_%H%M%S')
    /usr/bin/rclone -v --log-file=/backup/borg/logs/${repo}_b2sync_${inidate}.log sync /backup/borg/${repo} b2:${repo}
    if [[ $? -eq 0 ]]; then
      MSGDATE=$(date +'%d/%m/%y %T')
      mesaj="[${MSGDATE}] Upload for ${repo} was successful."
      curl -H "Title: B2 Upload" -H "Tags: arrow_double_up" -d "${mesaj}" "${URL}"
      #sendmsg "[${MSGDATE}] Upload for <b>${repo}</b> was <b><u>successful</u></b>."
    else
      MSGDATE=$(date +'%d/%m/%y %T')
      mesaj="[${MSGDATE}] Upload for ${repo} has failed. Check the logs."
      curl -H "Title: B2 Upload" -H "Tags: warning" -H "prio:high" -d "${mesaj}" "${URL}"
      #sendmsg "[${MSGDATE}] Upload for <b>${repo}</b> has <b><u>failed</u></b>. Check the logs."
    fi
    enddate=$(date +'%Y%m%d_%H%M%S')
    mv /backup/borg/logs/${repo}_b2sync_${inidate}.log /backup/borg/logs/${repo}_b2sync_${enddate}.log
done

@[email protected] · edit-2 7 months ago

Indeed, that all looks fairly innocuous. Just in case, you are sure that you didn’t just accidentially kill or killall rclone or bash?

Perhaps wrapping the script in strace might help debug where the offending signal is coming from.

calm.like.a.bomb · 7 months ago

Just in case, you are sure that you didn’t just accidentially kill or killall rclone or bash?

No. The process runs at night. Only if my dog started learning Linux and tested something! That makes me wonder…

@[email protected] · 7 months ago

I think I read something about there being timeouts for slow jobs, which backups definitely are. Really unsure though, it has been a while. Try spawning independent worker programs!

calm.like.a.bomb · 7 months ago

I just read a bit some internet posts and the documentation. I set TimeoutSec=infinity inside the service and I set the timer to run it now. I’ll see if that helps.

@sir_pronoun · 7 months ago

Is it on the horizon for cron to be deprecated?

calm.like.a.bomb · 7 months ago

I don’t think so, but I wanted to try systemd timers.

@sir_pronoun · 7 months ago

Sure, not trying to second guess you. What do you think of them so far? Is it worth the effort to learn them?

calm.like.a.bomb · 7 months ago

Yes, for sure. Timers are more versatile than what cron jobs can do.

@[email protected] · edit-2 7 months ago

Here is my template

sudo cat > /etc/systemd/user/rsync-backup.service <<EOF
[Unit]
Description=do rsync backups with some conditions
# After=network-online.target

[Service]
Type=oneshot
# require a power connection (optional)
# ExecStartPre=sh -c '[ $(cat /sys/class/power_supply/AC/online) = 1 ]'

# require battery over 40%
# ExecStartPre=sh -c '[ $(cat /sys/class/power_supply/BAT0/capacity) -ge 40 ]'

# require the connected network to NOT be "metered"
# ExecStartPre=sh -c '! $(nmcli -t -f GENERAL.METERED dev show | grep -q 'yes')'

ExecStart=/home/user/.local/bin/rsync-backup
# you might add everything you need
# ExecStart=/path/to/something/else

# delete old logs (disabled for testing)
# ExecStartPost=rm -f /var/log/rsync-backups.log
# log the updates
# ExecStartPost=sh -c 'echo "Last backup: $(date)" > /var/log/rsync-backup.log'
# write errors to log
StandardError=file:/var/log/rsync-backups.log

# GUI message
#ExecStartPost=/usr/bin/notify-send -t 0 -a "Backup" "rsync backup finished." "$(output of some command if you want infos about the backup)"

# run with low priority, when idling
# Nice=15
IOSchedulingClass=idle

# when conditions were not met, try again after 15 minutes
# Restart=on-failure
# RestartSec=900

[Install]
WantedBy=multi-user.target
EOF

Timer file:

sudo cat > /etc/systemd/user/rsync-backup.timer <<EOF
[Unit]
Description=do rsync backups with some conditions

[Timer]
OnCalendar=daily
Persistent=true
EOF

(I think the unit is needed)

That is a slightly modified variant of my automatic rpm-ostree system updates which took an hour or so with the help of ChatGPT and a lot of testing around.

Systemd services are lit.

If you add a “repeat when conditions are not met” you need another timer to start it. Like 2 loops, one big loop to start the process, and one small loop to keep trying until conditions are met. I do that with my system updates to prevent them being done

with low battery (or even using an AC requirement)
over a metered network
when the system is busy

calm.like.a.bomb · 7 months ago

I can’t see anywhere that a [Timer] section can be added to a service unit. I tried it and it doesn’t work:

Apr 11 13:16:15 computer systemd[2593025]: /home/clmbmb/.config/systemd/user/test.service:10: Unknown section 'Timer'. Ignoring.

The service/script ran as it should, but the Timer section is ignored, as you can see.

@[email protected] · 7 months ago

Okay good notice. You should put that in a deparate .timer file then

calm.like.a.bomb · 7 months ago

Thanks for the suggestion. I’ll look into this too.

@[email protected] · 7 months ago

deleted by creator

@[email protected] · 7 months ago

Nothing wrong with learning new tricks, but it’s worth mentioning on the side that sometimes a cron job is the right tool.