hpr_documentation/ccdn/CDN_Update_Script.md

# CDN Update Script

This document describes a reference implementation of the `cdn-update.sh` automation script for the Community CDN architecture.

The script is designed to:

* Download the signed control file
* Verify its signature
* Validate JSON syntax
* Generate fail2ban configuration
* Generate nginx configuration
* Select an available origin
* Synchronize content using rsync
* Export Prometheus metrics
* Fail safely when configuration validation fails

---

# Reference Script

```bash
#!/usr/bin/env bash

set -euo pipefail

###############################################################################
# Configuration
###############################################################################

BASE="/var/lib/cdn"

CONFIG_DIR="${BASE}/config"
CONTENT_DIR="${BASE}/content"
METRICS_DIR="${BASE}/metrics"

CONTROL_URL="https://control.example.org/hpr.ccdn.settings.json"
SIG_URL="https://control.example.org/hpr.ccdn.settings.json.minisig"

PUBKEY="RWQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

CONTROL_FILE="${CONFIG_DIR}/hpr.ccdn.settings.json"
SIG_FILE="${CONFIG_DIR}/hpr.ccdn.settings.json.minisig"

NGINX_GEN="/etc/nginx/conf.d/cdn-generated.conf"

FAIL2BAN_JAIL="/etc/fail2ban/jail.d/cdn-generated.local"

METRICS_FILE="${METRICS_DIR}/cdn.prom"

TMPDIR="$(mktemp -d)"

###############################################################################
# Metrics helper
###############################################################################

metric_write() {
    cat > "${METRICS_FILE}" <<EOF
cdn_last_sync_timestamp ${LAST_SYNC_TIMESTAMP:-0}
cdn_sync_success ${SYNC_SUCCESS:-0}
cdn_sync_duration_seconds ${SYNC_DURATION:-0}
cdn_invalid_requests_total ${INVALID_REQUESTS:-0}
cdn_active_origin{origin="${ACTIVE_ORIGIN:-none}"} 1
EOF
}

###############################################################################
# Cleanup
###############################################################################

cleanup() {
    rm -rf "${TMPDIR}"
}

trap cleanup EXIT

###############################################################################
# Download control file
###############################################################################

echo "Downloading control file..."

curl -fsSL \
    -o "${TMPDIR}/hpr.ccdn.settings.json" \
    "${CONTROL_URL}"

curl -fsSL \
    -o "${TMPDIR}/hpr.ccdn.settings.json.minisig" \
    "${SIG_URL}"

###############################################################################
# Verify signature
###############################################################################

echo "Verifying signature..."

minisign \
    -Vm "${TMPDIR}/hpr.ccdn.settings.json" \
    -P "${PUBKEY}" \
    -x "${TMPDIR}/hpr.ccdn.settings.json.minisig"

###############################################################################
# Validate JSON
###############################################################################

jq empty "${TMPDIR}/hpr.ccdn.settings.json"

install -m 0644 \
    "${TMPDIR}/hpr.ccdn.settings.json" \
    "${CONTROL_FILE}"

install -m 0644 \
    "${TMPDIR}/hpr.ccdn.settings.json.minisig" \
    "${SIG_FILE}"

###############################################################################
# Load values
###############################################################################

mapfile -t ORIGINS < <(
    jq -r '.origins[]' "${CONTROL_FILE}"
)

RSYNC_INTERVAL=$(
    jq -r '.rsync_interval_hours' "${CONTROL_FILE}"
)

MAXRETRY=$(
    jq -r '.fail2ban.maxretry' "${CONTROL_FILE}"
)

FINDTIME=$(
    jq -r '.fail2ban.findtime' "${CONTROL_FILE}"
)

BANTIME=$(
    jq -r '.fail2ban.bantime' "${CONTROL_FILE}"
)

###############################################################################
# Generate fail2ban configuration
###############################################################################

echo "Generating fail2ban configuration..."

ADMIN_IPS=$(
    jq -r '.admin_ips[]?' "${CONTROL_FILE}" \
        | tr '\n' ' '
)

cat > "${FAIL2BAN_JAIL}" <<EOF
[nginx-invalid]
enabled = true

maxretry = ${MAXRETRY}
findtime = ${FINDTIME}
bantime = ${BANTIME}

ignoreip = 127.0.0.1 ${ADMIN_IPS}
EOF

systemctl reload fail2ban

###############################################################################
# Select active origin
###############################################################################

ACTIVE_ORIGIN=""

for ORIGIN in "${ORIGINS[@]}"
do
    if ssh \
        -o BatchMode=yes \
        -o ConnectTimeout=5 \
        "${ORIGIN}" \
        true
    then
        ACTIVE_ORIGIN="${ORIGIN}"
        break
    fi
done

if [ -z "${ACTIVE_ORIGIN}" ]
then
    echo "No origin available"
    exit 1
fi

###############################################################################
# Generate nginx configuration
###############################################################################

echo "Generating nginx config..."

EXT_REGEX=$(
    jq -r '.allowed_extensions[]' "${CONTROL_FILE}" \
    | paste -sd'|' -
)

cat > "${NGINX_GEN}" <<EOF
autoindex off;

location ~ ^/eps/hpr[0-9]{4}/hpr[0-9]{4}\.(${EXT_REGEX})\$ {
    root ${CONTENT_DIR}/public_html;
}

location = /robots.txt {
    root ${CONTENT_DIR}/public_html;
}

location = /favicon.ico {
    root ${CONTENT_DIR}/public_html;
}

location / {
    access_log /var/log/nginx/invalid_requests.log;
    return 404;
}
EOF

nginx -t
systemctl reload nginx

###############################################################################
# Rsync
###############################################################################

echo "Starting rsync..."

START_TIME=$(date +%s)

if rsync \
    -az \
    --delete-delay \
    rsyncuser@"${ACTIVE_ORIGIN}":/srv/content/ \
    "${CONTENT_DIR}/"
then

    END_TIME=$(date +%s)

    LAST_SYNC_TIMESTAMP="${END_TIME}"
    SYNC_DURATION=$((END_TIME - START_TIME))
    SYNC_SUCCESS=1

else

    LAST_SYNC_TIMESTAMP=$(date +%s)
    SYNC_DURATION=0
    SYNC_SUCCESS=0

fi

###############################################################################
# Invalid request metric
###############################################################################

INVALID_REQUESTS=$(
    wc -l \
    < /var/log/nginx/invalid_requests.log \
    || echo 0
)

###############################################################################
# Write metrics
###############################################################################

metric_write

echo "Update complete"
```

---

# Recommended Enhancements

## Atomic Configuration Updates

Generate temporary configuration files first and only replace active files after validation succeeds.

Example workflow:

1. Generate configuration in a temporary directory.
2. Run `nginx -t`.
3. Replace production configuration.
4. Reload nginx.

This prevents broken configuration from affecting service availability.

---

## SSH Host Key Pinning

Use a dedicated `known_hosts` file.

Example:

```bash
ssh \
  -o UserKnownHostsFile=/etc/cdn/known_hosts \
  -o StrictHostKeyChecking=yes
```

This protects against origin impersonation.

---

## Restrict Synchronized Content

Limit rsync to approved file types.

Example:

```bash
rsync \
  -az \
  --delete-delay \
  --include='*/' \
  --include='*.mp3' \
  --include='*.ogg' \
  --include='*.opus' \
  --include='*.txt' \
  --include='*.json' \
  --exclude='*' \
  rsyncuser@origin:/srv/content/ \
  /var/lib/cdn/content/
```

This prevents accidental synchronization of unexpected files.

---

## Cached Control Files

If the control server is temporarily unavailable:

* Continue serving content
* Continue using the last verified control file
* Retry on the next scheduled execution

Nodes should never accept an unsigned replacement file.

---

## Version-Based Synchronization

Store the last control file version.

Only perform a full synchronization when:

* The control file version changes
* `force_full_rsync` is enabled

This reduces unnecessary origin traffic.

---

## Fail2ban Dynamic Ban Includes

Generate a separate include file for:

* Immediate IP bans
* Dynamic blocklists

Avoid rewriting the primary jail configuration on every update.

---

# Example Cron Schedule

```cron
*/5 * * * * /usr/local/bin/cdn-update.sh
```

This provides:

* Control file refresh every 5 minutes
* Automatic failover detection
* Automatic configuration updates
* Regular synchronization scheduling

---

# Operational Flow

1. Download control file.
2. Verify minisign signature.
3. Validate JSON.
4. Generate fail2ban configuration.
5. Generate nginx configuration.
6. Select active origin.
7. Synchronize content.
8. Export Prometheus metrics.
9. Exit successfully.

If any validation step fails, the script exits without modifying the running configuration.

This fail-closed behavior helps ensure that only authenticated, valid configuration changes are applied to CDN nodes.