I have a high memory program (ML) that I want to run in a tmux session. However, if the process is OOM killed, the tmux session is also shut down. I partially solved it (using the below run-bg), but it still kills the tmux window it runs in (I suspect because of the PTY, but maybe not). How can I run this program, still have the stdin and stdout be from and to the tmux window, but not have tmux window exit when the program is OOM killed?
run-bg (what I actually use to run the high-memory process):
#!/bin/bash
unit="bg-$(date +%s%N)"
#get the environment args from the executing process
env_args=()
while IFS='=' read -r key val; do
# Append each --setenv=KEY=VALUE (properly shell-escaped)
env_args+=( "--setenv=${key}=$(printf '%q' "$val")" )
done < <(env)
# Start the command as a transient user service
systemd-run --user --pty \
--unit="$unit" \
--slice=runbg.slice \
--property=MemoryMax=18G \
--property=MemoryLimit=14G \
--property=Delegate=yes \
--working-directory="$PWD" \
--expand-environment=yes \
"${env_args[@]}" \
"$@"
My tmux config:
set-option -g default-command 'bash -c run-bg-tmux'
run-bg-tmux (this allows for the tmux windows to be separate):
#!/bin/bash
unit="job-$(date +%s%N)"
systemd-run --user --scope \
--slice=tmuxjobs.slice \
--unit $unit \
--property=MemoryMax=32G \
--property=Delegate=yes \
bash
screenan option?