Fix bash: fork: retry: Resource temporarily unavailable

Quick answer: raise the process limits — check ulimit -u, systemd's TasksMax, and /sys/fs/cgroup/pids.max, then bump whichever one is being hit.

I got a call last month from a client running a PHP-FPM box on Ubuntu 22.04. Their SSH sessions were spitting out bash: fork: retry: Resource temporarily unavailable every few seconds, cron jobs had stopped firing, and the monitoring agent couldn't even start. First instinct from their dev was "we're out of RAM, add more." Wrong. free -h showed 40% free. The kernel wasn't short on memory — it was short on processes. Every fork() call was returning EAGAIN, which the shell reports as "resource temporarily unavailable." The system had hit its per-user or per-cgroup process cap, and nothing could spawn a new child.

This is one of those errors that looks like a memory problem and isn't. The kernel has three or four separate ceilings on how many processes and threads can exist at once, and any of them can trip first. You have to check all of them, because fixing the wrong one wastes an hour. The usual culprits: ulimit -u (RLIMIT_NPROC), systemd's TasksMax on the unit, the cgroup v2 pids.max file, and kernel.pid_max or kernel.threads-max if you're unlucky enough to have a genuinely huge process count.

Find out which limit you're hitting

Before changing anything, get the actual numbers. Run these on the box while the error is happening.

# how many processes this shell can fork
ulimit -u

# how many processes are actually running for this user
ps -eLf | wc -l

# system-wide
cat /proc/sys/kernel/pid_max
cat /proc/sys/kernel/threads-max

# if you're on systemd + cgroup v2, check the unit
systemctl show your-service.service -p TasksMax -p TasksCurrent

# and the cgroup file directly
cat /sys/fs/cgroup/pids.max
cat /sys/fs/cgroup/pids.current

Whichever current number is hugging its max is your problem. In my client's case, TasksCurrent was 4,912 against a TasksMax of 4,915. PHP-FPM had 200 workers, each spawning short-lived threads, plus Node on the same host, plus a stuck backup job that wasn't reaping children. Classic slow fork bomb.

The fix, step by step

Raise the shell limit for your session. ulimit -u 8192 takes effect immediately for that shell. Not persistent, but it lets you get work done while you fix the real cap.
Make it stick for users. On most distros, edit /etc/security/limits.conf (or a file in /etc/security/limits.d/) and add:
```
*    soft    nproc    8192
*    hard    nproc    16384
root soft    nproc    unlimited
root hard    nproc    unlimited
```
Then loginctl terminate-user $USER or reboot. ulimit changes don't apply to existing sessions.
Fix the systemd unit. If your service runs under systemd, TasksMax overrides everything. Edit the unit or drop-in:
```
sudo systemctl edit your-service.service
```
Add:
```
[Service]
TasksMax=infinity
LimitNPROC=infinity
```
Then sudo systemctl daemon-reload && sudo systemctl restart your-service.
Bump the kernel ceiling if needed. Only do this if pid_max is actually the wall. On a 64-bit box the default (usually 32768 or 4194304 depending on distro) is fine for 99% of setups.
```
sudo sysctl -w kernel.pid_max=4194304
# persist
echo 'kernel.pid_max=4194304' | sudo tee /etc/sysctl.d/99-pids.conf
```
Find and kill the runaway. Raising limits without fixing the source is dumb — you'll just hit the higher wall tomorrow. Look for processes with huge child counts:
```
ps -eo user,pid,ppid,nlwp,cmd --sort=-nlwp | head -20
```
A PHP-FPM or Java process with 3,000 threads isn't working, it's leaking. Restart it and check the config.

If that doesn't work

Sometimes the cap isn't the cap you think. Things to try, in order:

Check TCP connections too. If the process spawning children is a web server, you might actually be hitting net.ipv4.ip_local_port_range exhaustion, which looks similar in logs. ss -s tells you. Bump with sysctl -w net.ipv4.ip_local_port_range="10240 65535".
Look at PID namespace limits. Containers (Docker, Podman, k8s) have their own cgroup pids.max. docker run --pids-limit=0 disables it. In k8s, it's the --pod-max-pids kubelet flag. A container hitting this shows the exact same error and none of the host-level knobs will help.
Check for a real fork bomb. :(){ :|:& };: is the famous one, but bash scripts with unbounded recursion or a cron job that overlaps with itself do the same thing more slowly. pstree -p will show the tree shape. If one parent has hundreds of children all doing the same thing, kill the parent.
Memory overcommit settings. Rare, but if vm.overcommit_memory=2 is set, malloc-like fork behavior can fail even with free RAM. sysctl vm.overcommit_memory — 0 or 1 is what you want.

Preventing it from coming back

The real fix isn't a bigger number. It's knowing what your process count should be and alerting when it drifts. Set up a Prometheus or Zabbix check on node_processes_state or just a cron running ps -eLf | wc -l and emailing you if it crosses 70% of your configured cap. The client above never would have noticed PHP-FPM slowly leaking workers over two weeks if they'd had that. Also: cap PHP-FPM's pm.max_children, cap Node's thread pool with UV_THREADPOOL_SIZE, and stop running 11 different services under one systemd unit. Each one wants its own TasksMax, and when they share, they fight.