2026-08-07_13-20-43_CEST

2026-08-07 13:20:43 +02:00
parent a06b950661
commit 75440a83b0
5 changed files with 922 additions and 4 deletions
@@ -0,0 +1,213 @@
+# Community Content Delivery Network Docker nodes
+
+## Background
+
+We wish to deploy a Community CDN that is easy to deploy and maintain.
+
+This can be achieved by providing a Community Managed docker nodes that is regularly updated.
+
+Changes can be applied to the GitTea, and the mirror nodes should pick up the changes.
+
+## Removal of node
+
+- Remove from central DNS
+- Remove from monitoring
+- Remove from allow lists
+- Remove from authorized_keys
+
+## 
+
+- Addition of new nodes
+- Removal of old nodes
+
+
+
+
+Path Validation Strategy
+
+This is where your design differs from a normal web server.
+
+You know:
+
+episode numbers
+valid extensions
+exact file list
+
+Therefore every request can be validated.
+
+Option 1 (recommended)
+
+Allow nginx to serve files normally.
+
+If file does not exist:
+
+error_page 404 = @invalid_request;
+
+location @invalid_request {
+    access_log /var/log/nginx/invalid-paths.log;
+    return 404;
+}
+
+This scales well even with tens of thousands of files.
+
+Prevent Directory Browsing
+
+autoindex off;
+
+location / {
+    try_files $uri =404;
+}
+
+No listings.
+
+No traversal.
+
+No guessing directories.
+
+
+
+
+
+
+
+thank you - I have refined the requirements now so ignore what I said earlier.
+
+We are building a private community content delivery network, with nodes that serves files that typically change once a day.
+
+## Hardware Requirements
+
+Requirements for Hosting
+
+- 24/7 Home Service
+- fixed IP address
+- unlimited bandwidth
+- fast > 500mb/sec upload
+- large > 5T of storage
+- permission from your ISP to run a web server
+- Contact information know to the Janitors
+- Optional: UPS
+
+## Software updates
+
+
+The nodes should ideally run on both [Docker](https://en.wikipedia.org/wiki/Docker_%28software%29) and [Podman](https://en.wikipedia.org/wiki/Podman)
+
+It should run Debian `debian:stable-slim` 
+
+All software MUST be running verifiable Free Libre Open Source software with links to the license.
+
+The nodes should have as little utilities as possible to do it's job and in order to reduce the attack surface.
+
+The amount of traffic is about 5Mb/sec with a load of 2 TPS
+
+There will be many nodes, that will come and go over time.
+
+The nodes will be added to DNS so the load can be shared.
+
+The origin server requirements will be dictated by the needs of the nodes.
+
+Nodes will monitor a rss control channel from the origin server for instructions.
+
+The RSS file will contain a link to a json file.
+
+The json file will contain 
+
+- the fail2ban settings, list of ip addresses, maxretry, findtime and bantime
+- force full rsync Boolean flag - false
+- hours between rsync eg every 3 hours
+- list of origin servers
+- list of Admin IP addresses
+- list of IP addresses to ban immediately 
+- list of useragents to ban immediately 
+
+If the origin server is not available then the nodes should connect to the next origin server in the list.
+
+nodes will synchronize the files from a origin server using rsync over ssh.
+
+A full rsync is sufficient to ensure files are copied correctly.
+
+A full rsync will be done several times a day, eg every 3 hours - from settings
+
+atomic updates are not required as only new files are added.
+
+There are thousands of files
+
+The files will come from a well known directory eg `public_html/eps/hpr${ep_num}/hpr${ep_num}*.${extension}`
+
+The `ep_num` will be a digit from 0001 to 9999
+
+The `extension` will be from a well know list from the origin updated perhaps once a year
+
+nodes will then serve the files using `nginx`
+
+`nginx` must restrict access to files outside the well known files
+
+A list of well known files that clients require (favorite icon, robots.txt, etc) will be available from the origin server.
+
+The files we wish to serve are all Creative Commons Licensed so can be shared.
+
+Random browsing is not allowed.
+
+All file paths are known, and are provided by the origin server.
+
+There should never be any request for a file outside the well known paths.
+
+Any attempt to access any file outside the known paths should be logged.
+
+Based on a configuration provided by the origin server, the server will run failtoban
+
+repeat offenders will be banned for 7 days.
+
+The Admin IP addresses should never be banned
+
+traffic should be secured with let's encrypt of equivalent tls transport
+
+clients can request the files without tls
+
+The nodes public ip address will be placed on a allow list for ssh
+
+Monitoring should be available using Prometheus or similar.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                    +----------------+
+                    | Origin Server A|
+                    +--------+-------+
+                             |
+                    RSS Control Feed
+                             |
+                    JSON Configuration
+                             |
+          +------------------+------------------+
+          |                                     |
+          v                                     v
+
+----------------+                   +----------------+
+| Edge Node 1    |                   | Edge Node 2    |
+|                |                   |                |
+| nginx          |                   | nginx          |
+| rsync          |                   | rsync          |
+| fail2ban       |                   | fail2ban       |
+| node-agent     |                   | node-agent     |
+| prometheus exp.|                   | prometheus exp.|
+--------+-------+                   +--------+-------+
+         |                                     |
+         +----------------+--------------------+
+                          |
+                        DNS RR
+                          |
+                       Clients
+
+