hpr_documentation/ccdn/ccdn_docker_image_requirements.md

# Community Content Delivery Network Docker nodes

## Background

We wish to deploy a Community CDN that is easy to deploy and maintain.

This can be achieved by providing a Community Managed docker nodes that is regularly updated.

Changes can be applied to the GitTea, and the mirror nodes should pick up the changes.

## Removal of node

- Remove from central DNS
- Remove from monitoring
- Remove from allow lists
- Remove from authorized_keys

## 

- Addition of new nodes
- Removal of old nodes




Path Validation Strategy

This is where your design differs from a normal web server.

You know:

episode numbers
valid extensions
exact file list

Therefore every request can be validated.

Option 1 (recommended)

Allow nginx to serve files normally.

If file does not exist:

error_page 404 = @invalid_request;

location @invalid_request {
    access_log /var/log/nginx/invalid-paths.log;
    return 404;
}

This scales well even with tens of thousands of files.

Prevent Directory Browsing

autoindex off;

location / {
    try_files $uri =404;
}

No listings.

No traversal.

No guessing directories.







thank you - I have refined the requirements now so ignore what I said earlier.

We are building a private community content delivery network, with nodes that serves files that typically change once a day.

## Hardware Requirements

Requirements for Hosting

- 24/7 Home Service
- fixed IP address
- unlimited bandwidth
- fast > 500mb/sec upload
- large > 5T of storage
- permission from your ISP to run a web server
- Contact information know to the Janitors
- Optional: UPS

## Software updates


The nodes should ideally run on both [Docker](https://en.wikipedia.org/wiki/Docker_%28software%29) and [Podman](https://en.wikipedia.org/wiki/Podman)

It should run Debian `debian:stable-slim` 

All software MUST be running verifiable Free Libre Open Source software with links to the license.

The nodes should have as little utilities as possible to do it's job and in order to reduce the attack surface.

The amount of traffic is about 5Mb/sec with a load of 2 TPS

There will be many nodes, that will come and go over time.

The nodes will be added to DNS so the load can be shared.

The origin server requirements will be dictated by the needs of the nodes.

Nodes will monitor a rss control channel from the origin server for instructions.

The RSS file will contain a link to a json file.

The json file will contain 

- the fail2ban settings, list of ip addresses, maxretry, findtime and bantime
- force full rsync Boolean flag - false
- hours between rsync eg every 3 hours
- list of origin servers
- list of Admin IP addresses
- list of IP addresses to ban immediately 
- list of useragents to ban immediately 

If the origin server is not available then the nodes should connect to the next origin server in the list.

nodes will synchronize the files from a origin server using rsync over ssh.

A full rsync is sufficient to ensure files are copied correctly.

A full rsync will be done several times a day, eg every 3 hours - from settings

atomic updates are not required as only new files are added.

There are thousands of files

The files will come from a well known directory eg `public_html/eps/hpr${ep_num}/hpr${ep_num}*.${extension}`

The `ep_num` will be a digit from 0001 to 9999

The `extension` will be from a well know list from the origin updated perhaps once a year

nodes will then serve the files using `nginx`

`nginx` must restrict access to files outside the well known files

A list of well known files that clients require (favorite icon, robots.txt, etc) will be available from the origin server.

The files we wish to serve are all Creative Commons Licensed so can be shared.

Random browsing is not allowed.

All file paths are known, and are provided by the origin server.

There should never be any request for a file outside the well known paths.

Any attempt to access any file outside the known paths should be logged.

Based on a configuration provided by the origin server, the server will run failtoban

repeat offenders will be banned for 7 days.

The Admin IP addresses should never be banned

traffic should be secured with let's encrypt of equivalent tls transport

clients can request the files without tls

The nodes public ip address will be placed on a allow list for ssh

Monitoring should be available using Prometheus or similar.















                    +----------------+
                    | Origin Server A|
                    +--------+-------+
                             |
                    RSS Control Feed
                             |
                    JSON Configuration
                             |
          +------------------+------------------+
          |                                     |
          v                                     v

+----------------+                   +----------------+
| Edge Node 1    |                   | Edge Node 2    |
|                |                   |                |
| nginx          |                   | nginx          |
| rsync          |                   | rsync          |
| fail2ban       |                   | fail2ban       |
| node-agent     |                   | node-agent     |
| prometheus exp.|                   | prometheus exp.|
+--------+-------+                   +--------+-------+
         |                                     |
         +----------------+--------------------+
                          |
                        DNS RR
                          |
                       Clients