Study Cases

Learning Focus

These study cases show how systemd timers solve real-world automation problems at scale. Each case includes the problem, architecture, complete unit files, scripts, and lessons learned.

Study Case 1: WordPress VPS — Complete Backup Pipeline

Problem

A WordPress hosting company needs a comprehensive backup pipeline for each VPS:

Database export at 2:30 AM.
Full site archive at 2:45 AM (after DB export completes).
Upload archive to S3 at 3:00 AM.
Purge backups older than 14 days at 4:00 AM.
All jobs must catch up after downtime, prevent overlap, and alert on failure.

Architecture

Unit Files

/etc/systemd/system/wp-db-export.service
[Unit]
Description=WordPress database export
After=network-online.target
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
User=www-data
Group=www-data
ExecStart=/usr/bin/flock -n /var/lock/wp-backup.lock /usr/local/bin/wp db export /mnt/backups/db-latest.sql --path=/var/www/html
RuntimeMaxSec=30m
StandardOutput=append:/var/log/wp-backup.log
StandardError=append:/var/log/wp-backup.log
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/log /var/lock /mnt/backups

/etc/systemd/system/wp-db-export.timer
[Unit]
Description=WordPress DB export daily at 02:30

[Timer]
OnCalendar=02:30
Persistent=true
RandomizedDelaySec=3m

[Install]
WantedBy=timers.target

/etc/systemd/system/wp-site-archive.service
[Unit]
Description=WordPress full site archive
After=wp-db-export.service
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
User=www-data
ExecStart=/usr/local/bin/wp-site-archive.sh
RuntimeMaxSec=1h
StandardOutput=append:/var/log/wp-backup.log
StandardError=append:/var/log/wp-backup.log
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/log /mnt/backups /var/www/html

/etc/systemd/system/wp-site-archive.timer
[Unit]
Description=WordPress site archive daily at 02:45

[Timer]
OnCalendar=02:45
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/wp-upload-s3.service
[Unit]
Description=Upload backup archive to S3
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
EnvironmentFile=/etc/default/wp-s3-config
ExecStart=/usr/local/bin/rclone copy /mnt/backups/wp-site-latest.tar.gz remote:backups/
RuntimeMaxSec=1h
StandardOutput=append:/var/log/wp-backup.log
StandardError=append:/var/log/wp-backup.log

/etc/systemd/system/wp-upload-s3.timer
[Unit]
Description=Upload backup to S3 daily at 03:00

[Timer]
OnCalendar=03:00
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/wp-backup-prune.service
[Unit]
Description=Purge backup files older than 14 days

[Service]
Type=oneshot
ExecStart=/usr/bin/find /mnt/backups -name "*.sql" -o -name "*.tar.gz" -mtime +14 -delete
StandardOutput=append:/var/log/wp-backup.log

/etc/systemd/system/wp-backup-prune.timer
[Unit]
Description=Purge old backups daily at 04:00

[Timer]
OnCalendar=04:00
Persistent=true

[Install]
WantedBy=timers.target

Deployment

deploy-backup-pipeline.sh
sudo systemctl daemon-reload
for t in wp-db-export wp-site-archive wp-upload-s3 wp-backup-prune; do
    sudo systemctl enable --now "${t}.timer"
done
systemctl list-timers --no-pager | grep wp-

Lessons Learned

Staggered schedules (02:30, 02:45, 03:00, 04:00) ensure each stage completes before the next starts.
flock on the DB export prevents overlap with manual backups.
OnFailure= on every service ensures alerts on any pipeline stage failure.
Persistent=true on all timers ensures nothing is skipped during maintenance windows.

Study Case 2: Multi-Site Fleet Management

Problem

A hosting company manages 50 WordPress sites across 10 servers. Each server needs:

WP-CLI cron runner every 15 minutes for all sites.
Nightly backup of all sites.
Weekly optimization of all sites.

Template Units

Using systemd template units (@) to handle multiple sites:

/etc/systemd/system/wp-cron@.service
[Unit]
Description=WordPress cron for %i

[Service]
Type=oneshot
User=www-data
ExecStart=/usr/local/bin/wp cron event run --due-now --path=/var/www/%i
StandardOutput=append:/var/log/wp-cron.log
StandardError=append:/var/log/wp-cron.log

/etc/systemd/system/wp-cron@.timer
[Unit]
Description=WordPress cron for %i every 15 minutes

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/wp-backup@.service
[Unit]
Description=Nightly backup for %i
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
User=www-data
EnvironmentFile=/etc/wp-sites/%i.env
ExecStart=/usr/bin/flock -n /var/lock/wp-backup-%i.lock /usr/local/bin/wp-backup-site.sh %i
RuntimeMaxSec=2h
StandardOutput=append:/var/log/wp-backup/%i.log
StandardError=append:/var/log/wp-backup/%i.log
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/log/wp-backup /var/lock /mnt/backups /var/www/%i

/etc/systemd/system/wp-backup@.timer
[Unit]
Description=Nightly backup for %i

[Timer]
OnCalendar=02:15
Persistent=true
RandomizedDelaySec=30m
FixedRandomDelay=true

[Install]
WantedBy=timers.target

Adding a New Site

add-site.sh
#!/usr/bin/env bash
SITE="$1"       # e.g., site1.example.com

# Create site-specific config
sudo tee "/etc/wp-sites/${SITE}.env" > /dev/null <<EOF
SITE_DIR=/var/www/${SITE}
DB_NAME=wp_${SITE//[.-]/_}
EOF

# Create log directory
sudo mkdir -p /var/log/wp-backup
sudo touch "/var/log/wp-backup/${SITE}.log"

# Enable timers
sudo systemctl daemon-reload
sudo systemctl enable --now "wp-cron@${SITE}.timer"
sudo systemctl enable --now "wp-backup@${SITE}.timer"

echo "Timers enabled for $SITE"

usage.sh
sudo bash add-site.sh site1.example.com
sudo bash add-site.sh site2.example.com
sudo bash add-site.sh site3.example.com

Fleet Jitter Visualization

With RandomizedDelaySec=30m and FixedRandomDelay=true:

backup-schedule-across-sites.txt
site1.example.com  →  02:15 + 3m12s  =  02:18:12
site2.example.com  →  02:15 + 17m45s =  02:32:45
site3.example.com  →  02:15 + 8m30s  =  02:23:30
site4.example.com  →  02:15 + 25m55s =  02:40:55
site5.example.com  →  02:15 + 12m18s =  02:27:18

Each site gets a stable, unique offset based on its instance name.

Lessons Learned

Template units (@) let you manage unlimited sites with one pair of unit files.
FixedRandomDelay=true with 30 minutes spread ensures backups don't overload storage I/O.
Per-site log files (/var/log/wp-backup/%i.log) simplify debugging.
Per-site lock files (/var/lock/wp-backup-%i.lock) prevent per-site overlap.

Study Case 3: SaaS Application — Billing Cycle Automation

Problem

A SaaS application needs automated billing tasks:

Daily: Generate usage reports for active customers.
Monthly (1st): Calculate invoices from usage data.
Monthly (3rd): Send invoice emails.
Quarterly: Generate compliance reports.
End of month: Archive billing data.

Architecture

Timer Units

/etc/systemd/system/billing-usage.timer
[Unit]
Description=Collect daily usage data for billing

[Timer]
OnCalendar=23:00
Persistent=true
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target

/etc/systemd/system/billing-invoice.timer
[Unit]
Description=Calculate monthly invoices on the 1st

[Timer]
OnCalendar=*-*-01 06:00:00
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/billing-email.timer
[Unit]
Description=Send invoice emails on the 3rd

[Timer]
OnCalendar=*-*-03 09:00:00
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/billing-compliance.timer
[Unit]
Description=Quarterly compliance report

[Timer]
OnCalendar=quarterly
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/billing-archive.timer
[Unit]
Description=Month-end billing data archival

[Timer]
OnCalendar=*-*~1 22:00:00
Persistent=true

[Install]
WantedBy=timers.target

Lessons Learned

OnCalendar=*-*~1 for last day of month — native systemd feature, no shell tricks needed.
Persistent=true on all billing timers ensures no invoices are missed during maintenance.
Staggered daily schedules (usage at 23:00, invoices on the 1st, emails on the 3rd) give time for human review between stages.
Quarterly timer uses the quarterly keyword for simplicity.

Study Case 4: Server Infrastructure Maintenance Calendar

Problem

An ops team manages 20 production servers and needs a structured maintenance calendar:

Every 5 minutes: Health check and uptime monitoring.
Hourly: Log aggregation.
Daily at 3 AM: Security updates check.
Weekly (Sunday 2 AM): Full system backup.
Monthly (1st): Kernel and package update report.
Quarterly: Certificate rotation and audit.

Orchestration Script

/usr/local/bin/setup-maintenance-calendar.sh
#!/usr/bin/env bash
set -euo pipefail

# Timers to create
declare -A TIMERS=(
    ["health-check"]="*:0/5"
    ["log-aggregate"]="hourly"
    ["security-check"]="03:00"
    ["system-backup"]="Sun 02:00"
    ["update-report"]="monthly"
    ["cert-rotation"]="quarterly"
)

for name in "${!TIMERS[@]}"; do
    schedule="${TIMERS[$name]}"

    # Create timer
    cat > "/etc/systemd/system/${name}.timer" <<EOF
[Unit]
Description=Scheduled ${name}

[Timer]
OnCalendar=${schedule}
Persistent=true
RandomizedDelaySec=5m
FixedRandomDelay=true

[Install]
WantedBy=timers.target
EOF

    # Create service
    cat > "/etc/systemd/system/${name}.service" <<EOF
[Unit]
Description=Run ${name}
OnFailure=alert-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/${name}.sh
RuntimeMaxSec=1h
StandardOutput=append:/var/log/${name}.log
StandardError=append:/var/log/${name}.log
NoNewPrivileges=true
PrivateTmp=true
EOF

    echo "Created ${name}.timer (${schedule})"
done

systemctl daemon-reload

for name in "${!TIMERS[@]}"; do
    systemctl enable --now "${name}.timer"
done

echo "All maintenance timers enabled:"
systemctl list-timers --no-pager | grep -E "$(echo "${!TIMERS[@]}" | tr ' ' '|')"

Lessons Learned

Scripted timer creation ensures consistency across 20 servers.
FixedRandomDelay=true on all timers prevents synchronized load across the fleet.
OnFailure= on all services feeds into a centralized alerting system.
One script to deploy, one script to verify — operational simplicity.

Study Case 5: Monitoring Stack — Metric Collection Pipeline

Problem

A monitoring stack collects metrics at different frequencies:

Every 10 seconds: CPU, memory, disk I/O.
Every 1 minute: Network traffic, open connections.
Every 5 minutes: Application-level metrics (response times, queue depth).
Every 15 minutes: Aggregate and push to external monitoring service.

High-Frequency Timer (10 Seconds)

For sub-minute intervals, use OnUnitActiveSec=:

/etc/systemd/system/metrics-fast.timer
[Unit]
Description=Collect fast metrics every 10 seconds

[Timer]
OnBootSec=5s
OnUnitActiveSec=10s
AccuracySec=1s

[Install]
WantedBy=timers.target

/etc/systemd/system/metrics-fast.service
[Unit]
Description=Fast metric collection (CPU, memory, disk)

[Service]
Type=oneshot
ExecStart=/usr/local/bin/collect-fast-metrics.sh

Medium-Frequency Timer (1 Minute)

/etc/systemd/system/metrics-medium.timer
[Unit]
Description=Collect medium-frequency metrics

[Timer]
OnCalendar=minutely
AccuracySec=5s

[Install]
WantedBy=timers.target

Aggregation and Push Timer (15 Minutes)

/etc/systemd/system/metrics-push.timer
[Unit]
Description=Aggregate and push metrics every 15 minutes

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/metrics-push.service
[Unit]
Description=Push aggregated metrics to monitoring service

[Service]
Type=oneshot
EnvironmentFile=/etc/default/monitoring
ExecStart=/usr/local/bin/push-metrics.sh
RuntimeMaxSec=2m
StandardOutput=append:/var/log/metrics-push.log

Lessons Learned

Monotonic timers (OnUnitActiveSec=10s) handle sub-minute intervals that OnCalendar cannot express.
AccuracySec=1s is essential for high-frequency metrics — the default 1-minute accuracy is too coarse.
Combine monotonic (fast collection) + calendar (periodic push) for a multi-tier metrics pipeline.
Keep fast-path scripts extremely lightweight — 10-second intervals leave no room for slow scripts.

Study Case Summary

Case	Key Pattern	Timer Type	Key Learning
1. Backup Pipeline	Staggered schedules	Calendar	Order stages by time; `flock` + `OnFailure`
2. Fleet Management	Template units (`@`)	Calendar	`FixedRandomDelay` for per-site jitter
3. Billing Cycle	Month-end (`~`)	Calendar	Last-day-of-month with `-~1`
4. Maintenance Calendar	Scripted deployment	Calendar	Consistent across fleet with one script
5. Monitoring Stack	Sub-minute intervals	Monotonic	`OnUnitActiveSec=10s` with `AccuracySec=1s`

What's Next

Cheatsheet and Quiz — one-page cheat sheet, hands-on lab, and 10-question quiz.

Study Case 1: WordPress VPS — Complete Backup Pipeline​

Problem​

Architecture​

Unit Files​

Deployment​

Lessons Learned​

Study Case 2: Multi-Site Fleet Management​

Problem​

Template Units​

Adding a New Site​

Fleet Jitter Visualization​

Lessons Learned​

Study Case 3: SaaS Application — Billing Cycle Automation​

Problem​

Architecture​

Timer Units​

Lessons Learned​

Study Case 4: Server Infrastructure Maintenance Calendar​

Problem​

Orchestration Script​

Lessons Learned​

Study Case 5: Monitoring Stack — Metric Collection Pipeline​

Problem​

High-Frequency Timer (10 Seconds)​

Medium-Frequency Timer (1 Minute)​

Aggregation and Push Timer (15 Minutes)​

Lessons Learned​

Study Case Summary​

What's Next​

Study Case 1: WordPress VPS — Complete Backup Pipeline

Problem

Architecture

Unit Files

Deployment

Lessons Learned

Study Case 2: Multi-Site Fleet Management

Problem

Template Units

Adding a New Site

Fleet Jitter Visualization

Lessons Learned

Study Case 3: SaaS Application — Billing Cycle Automation

Problem

Architecture

Timer Units

Lessons Learned

Study Case 4: Server Infrastructure Maintenance Calendar

Problem

Orchestration Script

Lessons Learned

Study Case 5: Monitoring Stack — Metric Collection Pipeline

Problem

High-Frequency Timer (10 Seconds)

Medium-Frequency Timer (1 Minute)

Aggregation and Push Timer (15 Minutes)

Lessons Learned

Study Case Summary

What's Next