Debugging and Troubleshooting

Learning Focus

By the end of this lesson you will have a reliable, repeatable debugging workflow for any systemd.path issue — from "it never triggers" to "it loops infinitely" to "it worked yesterday but not today."

Debugging Workflow

Follow this step-by-step procedure every time a path unit misbehaves. Do not skip steps — most issues are caught in the first three.

Step 1 — Validate Unit File Syntax

step-1-verify.sh
sudo systemd-analyze verify /etc/systemd/system/myjob.path
sudo systemd-analyze verify /etc/systemd/system/myjob.service

If there are syntax errors, the output tells you the exact line and directive:

example-error.txt
/etc/systemd/system/myjob.path:5: Unknown key 'Watchh' in section 'Path'

No output means no errors.

Step 2 — Reload Unit Files

step-2-reload.sh
sudo systemctl daemon-reload

The #1 Beginner Mistake

If you edit a unit file and forget to run daemon-reload, systemd still uses the old version. This causes "I changed the config but nothing happened" confusion.

Step 3 — Check Path Unit Status

step-3-check-path.sh
systemctl status myjob.path --no-pager

What to look for:

Status	Meaning	Action
`active (waiting)`	Watch is registered, waiting for events	Good — proceed to step 4
`inactive (dead)`	Watch is not running	`sudo systemctl start myjob.path`
`failed`	Unit failed to start	Check the error message in the status output
Not found	Unit file not found	Check the file path and name

Step 4 — Test the Service Manually

Before trusting the path trigger, verify the service works on its own:

step-4-test-service.sh
sudo systemctl start myjob.service
journalctl -u myjob.service -n 30 --no-pager

If the service fails here, the problem is in the script, permissions, or dependencies — not the path unit. Fix the service first.

Step 5 — Trigger the Path Unit

Create the condition that should trigger the path unit:

step-5-trigger.sh
# For DirectoryNotEmpty:
touch /path/to/watched/dir/testfile.txt

# For PathExists:
touch /path/to/signal/file

# For PathChanged:
echo "test" >> /path/to/watched/file

# For PathModified:
echo "test" >> /path/to/watched/file

Step 6 — Check Both Logs

step-6-check-logs.sh
# Path unit logs (when did it trigger?)
journalctl -u myjob.path --since '1 hour ago' --no-pager

# Service logs (what happened when it ran?)
journalctl -u myjob.service --since '1 hour ago' --no-pager

# Combined view
journalctl -u myjob.path -u myjob.service --since '1 hour ago' --no-pager

Troubleshooting Matrix

Use this matrix to jump directly to the root cause:

Symptom	Likely Cause	Fix
Path unit `active (waiting)` but never triggers	Watched path doesn't exist and `MakeDirectory=` is not set	Add `MakeDirectory=yes` or create the directory manually
Path unit `active (waiting)` but pre-existing file doesn't trigger	Using `PathChanged=` which only fires on future changes	Switch to `PathExists=` for pre-existing files
Service loops infinitely	`PathExists=` / `DirectoryNotEmpty=` and script doesn't remove the trigger	Add `rm` or `mv` for the trigger file in the script
`PathChanged=` never fires	Editor uses swap/rename (vim default)	Try `PathModified=` or watch the directory instead of the file
Service runs but `ExecStart` fails	Wrong path, permissions, or user	Use absolute paths; set `User=`; run `chmod +x` on script
`systemctl status` shows `failed`	Unit file syntax error	Run `systemd-analyze verify` on both files
Excess triggers overwhelm the system	Hot directory with rapid file creation	Add `StartLimitBurst=` and `StartLimitIntervalSec=`
Ran out of inotify watches	Too many path units or other inotify consumers	Increase `fs.inotify.max_user_watches`
User path unit not running after logout	Linger not enabled	Run `sudo loginctl enable-linger "$USER"`
Service fails with "Permission denied"	Running as wrong user or missing file permissions	Check `User=`, `Group=`, and file ownership/mode
Unit loads but `Triggers:` shows wrong service	`Unit=` directive points to non-existent service	Check the `Unit=` value or use matching base names
Path unit shows `inactive (dead)` after reboot	`WantedBy=` not set or unit not enabled	Add `WantedBy=paths.target` and `systemctl enable`
"Start request repeated too quickly"	Rate limit hit	Increase limits or fix the root cause of rapid triggers
Service succeeds but script output is missing	Wrong `StandardOutput` setting	Check `StandardOutput=` — default is `journal`

Common Failure Scenarios — Deep Dive

Scenario 1: "It Never Triggers"

Symptoms: Path unit is active (waiting) but dropping a file doesn't trigger the service.

Diagnostic steps:

diagnose-no-trigger.sh
# 1. Confirm the watch is active and the correct path is being watched
systemctl status myjob.path --no-pager

# 2. Confirm the watched path exists
ls -la /path/to/watched/dir/

# 3. Confirm the service name matches
systemctl show -p Triggers myjob.path

# 4. Check inotify watch count
cat /proc/sys/fs/inotify/max_user_watches
find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | wc -l

Common causes:

Directory doesn't exist and MakeDirectory= is not set.
Watching a symlink target instead of the symlink itself.
NFS/CIFS mount — inotify doesn't work over network filesystems.
inotify watches exhausted — see the inotify section below.

Scenario 2: "It Loops Infinitely"

Symptoms: The journal fills with repeated execution logs, CPU spikes, and the service never stops running.

Emergency fix:

emergency-stop.sh
# Stop the path unit immediately
sudo systemctl stop myjob.path

# Check how many times the service ran
journalctl -u myjob.service --since "10 minutes ago" | grep -c "Started"

Root cause: Your script uses PathExists= or DirectoryNotEmpty= but does NOT delete/move the trigger file.

Permanent fix: Add file cleanup to your script:

fix-loop.sh
# For PathExists:
rm /path/to/trigger/file

# For DirectoryNotEmpty:
rm -f /path/to/queue/*   # or mv to done/

Scenario 3: "Permission Denied"

Symptoms: Service starts but immediately fails with exit code 126 or 13.

Diagnostic steps:

diagnose-permissions.sh
# 1. Check who the service runs as
systemctl show -p User -p Group myjob.service

# 2. Check file permissions on the script
ls -la /usr/local/bin/process.sh

# 3. Check if the script is executable
file /usr/local/bin/process.sh

# 4. Check permissions on watched directories
ls -la /var/www/drop/

# 5. Try running the script as the service user
sudo -u www-data /usr/local/bin/process.sh

Common fixes:

fix-permissions.sh
# Make the script executable
sudo chmod +x /usr/local/bin/process.sh

# Fix ownership of the watched directory
sudo chown -R www-data:www-data /var/www/drop/

# Fix the User/Group in the service file
# User=www-data
# Group=www-data

Scenario 4: "Works Manually But Not Via Path"

Symptoms: Running sudo systemctl start myjob.service works, but the path trigger doesn't activate it.

Diagnostic steps:

diagnose-manual-only.sh
# 1. Confirm the path unit is actually running
systemctl status myjob.path --no-pager

# 2. Confirm the base names match
ls /etc/systemd/system/myjob.*

# 3. Check the Triggers field
systemctl show -p Triggers myjob.path

# 4. Manually trigger and watch the journal in real time
journalctl -u myjob.path -u myjob.service -f &
touch /path/to/trigger

Common causes:

Base names don't match — the .path is my-job.path but the .service is myjob.service.
Unit= directive points to a non-existent or misspelled service name.
Wrong directive — using PathChanged= for a file that you're creating (not modifying).

Scenario 5: "Start Request Repeated Too Quickly"

Symptoms: Journal shows start request repeated too quickly for myjob.service and the unit enters failed state.

Diagnostic steps:

diagnose-rate-limit.sh
# 1. Check the failure state
systemctl status myjob.service --no-pager

# 2. See how many times it started
journalctl -u myjob.service --since "5 minutes ago" | grep -c "Started"

# 3. Check current rate limits
systemctl show -p StartLimitBurst -p StartLimitIntervalSec myjob.service

Fixes:

fix-rate-limit.sh
# 1. Reset the failed state
sudo systemctl reset-failed myjob.service

# 2. Increase rate limits in the service file
# [Unit]
# StartLimitBurst=20
# StartLimitIntervalSec=60

# 3. Or fix the root cause — why is the directory getting so many files so fast?

inotify Watch Limits

Understanding the Limit

Each path unit uses at least one inotify watch. The system-wide default limit is usually 8192 watches. Other tools also consume watches:

IDE file watchers (VSCode, IntelliJ)
Docker
inotifywait scripts
Backup tools

Checking Current Usage

check-inotify.sh
# Current limit
cat /proc/sys/fs/inotify/max_user_watches

# Current usage (approximate)
find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | wc -l

# Which processes are using inotify
for pid in $(find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | cut -d/ -f3 | sort -u); do
    echo "PID $pid: $(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')"
done

Increasing the Limit

increase-inotify.sh
# Check current value
cat /proc/sys/fs/inotify/max_user_watches
# Default: 8192

# Increase temporarily (lost on reboot)
echo 65536 | sudo tee /proc/sys/fs/inotify/max_user_watches

# Increase permanently
echo "fs.inotify.max_user_watches=65536" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Verify
cat /proc/sys/fs/inotify/max_user_watches

Signs of Exhausted Watches

Path units show active (waiting) but never fire.
journalctl -u myjob.path shows inotify errors.
dmesg shows inotify: inotify_init: inotify instances limit reached.

Environment Differences

systemd's Environment Is NOT Your Shell

A common trap: your script works perfectly when run manually but fails inside systemd. This is because systemd services run in a minimal environment:

Feature	Your Shell	systemd Service
`PATH`	Full path with user bins	Minimal: `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin`
Home dir	`$HOME` is set	May not be set
Working dir	Your current dir	`/` (or `WorkingDirectory=`)
Shell config	`.bashrc` loaded	Not loaded
Environment vars	All your exports	Only explicit `Environment=`

Fixes:

fix-environment.service
[Service]
# Set PATH explicitly
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
# Or load from a file
EnvironmentFile=/etc/default/myapp
# Always use absolute paths in ExecStart
ExecStart=/usr/local/bin/process.sh
# Set working directory
WorkingDirectory=/var/www/html

Network Filesystem Warning

NFS, CIFS, and Network Mounts

inotify does not work over network filesystems (NFS, CIFS/SMB, SSHFS). If your watched directory is on a network mount:

PathChanged= and PathModified= will never fire.
PathExists= and DirectoryNotEmpty= may work, but only because systemd polls them internally.

Workaround: Use a .timer unit to poll network-mounted directories instead of a .path unit:

[Timer]
OnCalendar=*:0/5    # Check every 5 minutes
Persistent=true

Diagnostic Commands Reference

Purpose	Command
Verify unit syntax	`sudo systemd-analyze verify /etc/systemd/system/myjob.{path,service}`
Check path status	`systemctl status myjob.path --no-pager`
Check service result	`systemctl status myjob.service --no-pager`
View unit file content	`systemctl cat myjob.path`
Show all unit properties	`systemctl show myjob.path`
Show what the path triggers	`systemctl show -p Triggers myjob.path`
Follow live service logs	`journalctl -u myjob.service -f`
Errors only	`journalctl -u myjob.service -p err --no-pager`
Combined path+service logs	`journalctl -u myjob.path -u myjob.service -f`
Logs since specific time	`journalctl -u myjob.service --since "2 hours ago"`
Reset failed state	`sudo systemctl reset-failed myjob.service`
Security audit	`systemd-analyze security myjob.service`
inotify watch count	`cat /proc/sys/fs/inotify/max_user_watches`
List all path units	`systemctl list-units --type=path --all`

Key Takeaways

Follow the 6-step debugging workflow every time — most issues are caught in steps 1–3.
The #1 mistake is forgetting daemon-reload after editing a unit file.
Infinite loops are caused by PathExists=/DirectoryNotEmpty= without file cleanup.
systemd's environment is minimal — always use absolute paths and explicit Environment=.
inotify doesn't work on network filesystems — use timers for NFS/CIFS mounts.
systemd-analyze verify catches syntax errors before they cause runtime failures.

What's Next

Study Cases — real-world scenarios where systemd.path solves complex automation problems.

Debugging Workflow​

Step 1 — Validate Unit File Syntax​

Step 2 — Reload Unit Files​

Step 3 — Check Path Unit Status​

Step 4 — Test the Service Manually​

Step 5 — Trigger the Path Unit​

Step 6 — Check Both Logs​

Troubleshooting Matrix​

Common Failure Scenarios — Deep Dive​

Scenario 1: "It Never Triggers"​

Scenario 2: "It Loops Infinitely"​

Scenario 3: "Permission Denied"​

Scenario 4: "Works Manually But Not Via Path"​

Scenario 5: "Start Request Repeated Too Quickly"​

inotify Watch Limits​

Understanding the Limit​

Checking Current Usage​

Increasing the Limit​

Signs of Exhausted Watches​

Environment Differences​

systemd's Environment Is NOT Your Shell​

Network Filesystem Warning​

Diagnostic Commands Reference​

Key Takeaways​

What's Next​

Debugging Workflow

Step 1 — Validate Unit File Syntax

Step 2 — Reload Unit Files

Step 3 — Check Path Unit Status

Step 4 — Test the Service Manually

Step 5 — Trigger the Path Unit

Step 6 — Check Both Logs

Troubleshooting Matrix

Common Failure Scenarios — Deep Dive

Scenario 1: "It Never Triggers"

Scenario 2: "It Loops Infinitely"

Scenario 3: "Permission Denied"

Scenario 4: "Works Manually But Not Via Path"

Scenario 5: "Start Request Repeated Too Quickly"

inotify Watch Limits

Understanding the Limit

Checking Current Usage

Increasing the Limit

Signs of Exhausted Watches

Environment Differences

systemd's Environment Is NOT Your Shell

Network Filesystem Warning

Diagnostic Commands Reference

Key Takeaways

What's Next