HPR/hpr_documentation
6
0
Fork 2
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: HPR:main
Branches Tags
HPR:main
sgoti:main
..
compare: sgoti:main
Branches Tags
sgoti:main
HPR:main

No commits in common. "main" and "main" have entirely different histories.
main ... main

2 changed files with 149 additions and 147 deletions
Show Stats Expand all files Collapse all files

288
ccdn/README.md
View File

@ -1,144 +1,146 @@
# Community Content Delivery Network (CCDN) # Community Content Delivery Network (CCDN)
_A location to track the deployment of the HPR Community Content Delivery Network, that provides a mirror network for our content._ A location to track the deployment of the HPR Community Content Delivery Network, that provides a mirror network for our content.
## Availability of HPR Content
Availability of HPR Content
The HPR site has traditionally been run on a single instance which makes the project vulnerable.
The HPR site has been traditionally been run on a single instance, which makes the project vulnerable.
We have experienced several occasions where we have suffered downtime, resulting from system outages, denial of service attacks, forced decommissioning, or increased costs.
We have experienced several times where we have suffered from issues resulting from system outages, denial of service attacks, forced decommissioning, or increased costs.
There is a clear need to host the content in multiple geographically distributed networks to increase reliability and redundancy.
There is a clear need to host the content in multiple geographically distributed networks to increase reliability and redundancy.
Applying a [Content Delivery Network](https://en.wikipedia.org/wiki/Content_delivery_network) in front of the provider addresses some but not all of these issues.
Applying a [Content Delivery Network](https://en.wikipedia.org/wiki/Content_delivery_network) in front of the provider addresses some but not all of these issues.
These large vendor solutions provide free tiers, but the long term business model shows that these are not sustainable.
These large vendor solutions provide free tiers, but the long term business model shows that these are not sustainable.
Additionally the algorithms used would flag behavior, considered normal for HPR contributors, as suspicious and would deny them access.
Additionally the algorithms used would flag behavior considered normal for HPR contributors, as suspicious and would deny them access.
## Looking to the past
At the dawn of the Internet, it was common for websites and services like DNS to be [mirrored](https://en.wikipedia.org/wiki/Mirror_site) by friends. # Looking to the past
This was for a long time not a viable option for HPR as the quantity of Audio Content was expensive to host and transfer, and was therefore beyond what a home user could reliably serve. At the dawn of the Internet, it was common for websites and services like DNS to be [mirrored](https://en.wikipedia.org/wiki/Mirror_site) by friends.
Over time, and in some locations members of our community have access to facilities that a few years ago would have been reserved for Internet Service Providers. This was for a long time not a viable option for HPR as the quantity of Audio Content was expensive to host and transfer, and was therefore beyond what a home user could reliably serve.
If you are one of the fortunate people, and would like to contribute hosting of a mirror of the HPR site and media, then please get in touch with _admin @ hackerpublicradio.org_ Over time, in some locations members of our community have access to facilities that a few years ago would have been reserved for Internet Service Providers.
If you are interested in helping hosting the HPR site and media, then please get in touch with _admin @ hackerpublicradio.org_
### Requirements for Hosting
- 24/7 Home Service ## Requirements for Hosting
- fixed IP address
- unlimited bandwidth - 24/7 Home Service
- fast > 500mb/sec upload - fixed IP address
- large > 3T of storage - unlimited bandwidth
- permission from your ISP to run a web server - fast > 500mb/sec upload
- Contact information know to the Janitors - large > 1T of storage
- Optional: [UPS](https://en.wikipedia.org/wiki/Uninterruptible_power_supply) - permission from your ISP to run a web server
- Contact information know to the Janitors
<!-- - Optional: [UPS](https://en.wikipedia.org/wiki/Uninterruptible_power_supply)
## Mediation for the Internet Archive Outage <!--
Links media files will be updated to point to a redirect service running on the HPR Hub ## Mediation for the Internet Archive Outage
eg: `https://hub.hackerpublicradio.org/redirect.php?id=9999`
Links media files will be updated to point to a redirect service running on the HPR Hub
This will maintain a list of HPR mirrors and for now do a simple random redirect eg: `https://hub.hackerpublicradio.org/redirect.php?id=9999`
- https://hpr.nyc3.cdn.digitaloceanspaces.com/ → alpha.nj.us.na.mirror.hackerpublicradio.org This will maintain a list of HPR mirrors and for now do a simple random redirect
- https://188.212.114.84/HPR/ → alpha.
- Internet Archive - DOWN - https://hpr.nyc3.cdn.digitaloceanspaces.com/ → alpha.nj.us.na.mirror.hackerpublicradio.org
- https://188.212.114.84/HPR/ → alpha.
Maxmind GeoIP free edition has: - Internet Archive - DOWN
Geolocation codes Maxmind GeoIP free edition has:
Our data includes codes that can be used to identify the continent, country, subdivision, and postal or metro code area of the geolocation of the IP address. The codes follow these conventions: Geolocation codes
continent
Our data includes codes that can be used to identify the continent, country, subdivision, and postal or metro code area of the geolocation of the IP address. The codes follow these conventions:
a two-character continent code, as follows: continent
AF - Africa a two-character continent code, as follows:
AN - Antarctica
AS - Asia AF - Africa
EU - Europe AN - Antarctica
NA - North America AS - Asia
OC - Oceania EU - Europe
SA - South America NA - North America
country the two-character ISO 3166-1 country code OC - Oceania
subdivision the region-portion of the ISO 3166-2 code for the region SA - South America
country the two-character ISO 3166-1 country code
So I will use that. subdivision the region-portion of the ISO 3166-2 code for the region
Ken Fallon (PA7KEN, G5KEN)
Although parsing is better with https://github.com/maxmind/mmdbinspect/ So I will use that.
I created a new documentation repo but am keeping the old one around for now as a work in progress Ken Fallon (PA7KEN, G5KEN)
Although parsing is better with https://github.com/maxmind/mmdbinspect/
I created a new documentation repo but am keeping the old one around for now as a work in progress
To that end I'm removing port 80 from `borg` and 443 from another server to point to the new server ``
I plan to update the feeds, and the site to point to To that end I'm removing port 80 from `borg` and 443 from another server to point to the new server ``
That will redirect to one of the mirrors, currently only `vger.mirror.hackerpublicradio.org` but then the IA once it's back, and also
I plan to update the feeds, and the site to point to
For any of that I need media files - so this is fix now, check later. That will redirect to one of the mirrors, currently only `vger.mirror.hackerpublicradio.org` but then the IA once it's back, and also
We have an account on `rsync.net` which I think we should use privately to push to from the `hpr_generator/static` and the `static/static` For any of that I need media files - so this is fix now, check later.
We have an account on `rsync.net` which I think we should use privately to push to from the `hpr_generator/static` and the `static/static`
# Origne Server
Where is the source of truth ? As in where will the mirrors rsync the files from ? # Origne Server
This will need to be RW to the processes generating tools, but RO to the admins volunteers of the mirror ccdn network. Where is the source of truth ? As in where will the mirrors rsync the files from ?
Using `rsync.net` is not ideal as we only have one account with RW access. This will need to be RW to the processes generating tools, but RO to the admins volunteers of the mirror ccdn network.
> What is currently stored on `rsync.net`? Is it just the media files or the html, extended show notes, and related show images, or both? Using `rsync.net` is not ideal as we only have one account with RW access.
For now it makes sense to have this on `borg`. > What is currently stored on `rsync.net`? Is it just the media files or the html, extended show notes, and related show images, or both?
For now it makes sense to have this on `borg`.
So for now it will be on borg. There are many disadvantages to this, single point of contact, same backup disk as the rsync source, and if I get ddosed were down. However for now that seems to be the way to go
This also means a change to how we send out files. The end point is no longer the IA but having it on the ccdn.
I want to be able add the encoded media, and the transcripts to the assets table as part of their generation So for now it will be on borg. There are many disadvantages to this, single point of contact, same backup disk as the rsync source, and if I get ddosed were down. However for now that seems to be the way to go
This also means a change to how we send out files. The end point is no longer the IA but having it on the ccdn.
# Are files available I want to be able add the encoded media, and the transcripts to the assets table as part of their generation
eps valid=1 # Are files available
show needs to be in eps table
eps valid=1
show needs to be in eps table
# Hosting a complete copy
Where does a complete copy of the website that is easy to download to another computer live? # Hosting a complete copy
> What does hosting a complete copy of the website mean? Is this the static site (html, css, images [host and episode], media [ogg, mp3, spx, vtt, srt, txt]) and dynamic hub stuff? Where does a complete copy of the website that is easy to download to another computer live?
## Internet Archive (IA) > What does hosting a complete copy of the website mean? Is this the static site (html, css, images [host and episode], media [ogg, mp3, spx, vtt, srt, txt]) and dynamic hub stuff?
What files associated with an episode are allowed to be stored on IA? ## Internet Archive (IA)
* full show notes? What files associated with an episode are allowed to be stored on IA?
* Other associated example files?
* images * full show notes?
* audio * Other associated example files?
* images
What are the standard/best practices for organizing files on IA? * audio
If we can store all show related files on IA, is that what we want to do? What are the standard/best practices for organizing files on IA?
Should IA be the main storage of a shows assets?
If we can store all show related files on IA, is that what we want to do?
Should IA be the main storage of a shows assets?
## git repository
As Ken as mentioned before, a git repository could be used to allow for an easy way to download and keep updated a complete copy of the website (perhaps without the audio files). This could be achieved relatively simply. ## git repository
We could also take advantage of GitLab, GitHub or other git hosting providers as mirrors. As Ken as mentioned before, a git repository could be used to allow for an easy way to download and keep updated a complete copy of the website (perhaps without the audio files). This could be achieved relatively simply.
## Docker image We could also take advantage of GitLab, GitHub or other git hosting providers as mirrors.
The image would not come with the static html, but would be set up to run the site-generator and associated update scripts on a regular basis. ## Docker image
Another way would be to have the image automatically rsync the website to initialize and then update on a regular basis. The image would not come with the static html, but would be set up to run the site-generator and associated update scripts on a regular basis.
Another way would be to have the image automatically rsync the website to initialize and then update on a regular basis.
--> -->

8
developer_information.md
View File

@ -16,7 +16,7 @@ Some things we can change without discussion but other things we need to get inp
Changes can take a long time, community approval can take several months, while other changes require a lot of work from volunteers who are focused on other priorities. Changes can take a long time, community approval can take several months, while other changes require a lot of work from volunteers who are focused on other priorities.
We allow redistribution by releasing all our content under a [Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/). In the same vein all our code is released under [aGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html), or other [OSI approved licenses](https://opensource.org/licenses). We allow redistribution by releasing all our content under a [Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/). In the same vein all our code is released under [aGPLv3](https://www.gnu.org/licenses/agpl-3.0.en.html), or other [OSI approved licenses](https://opensource.org/licenses) .
We do not track statistics to the detriment of our prime directive. We do not track statistics to the detriment of our prime directive.
@ -36,7 +36,7 @@ That said, we move with the times when there is a clear advantage to do so.
We run up to date patched stable software. We run up to date patched stable software.
We have a long tradition of supporting and sharing hacker culture. Any identified vulnerabilities are fixed with credit if requested. We have a long tradition of supporting and sharing hacker culture. Any identified vulnerability are fixed with credit if requested.
We use [RSS](https://www.rssboard.org/rss-specification) as a delivery mechanism, which is by default fault tolerant. We use [RSS](https://www.rssboard.org/rss-specification) as a delivery mechanism, which is by default fault tolerant.
@ -46,7 +46,7 @@ All our code is on [GitTea](https://repo.anhonesthost.net/HPR), please clone loc
[Our database](https://hackerpublicradio.org/hpr.sql) is updated frequently, please copy locally. [Our database](https://hackerpublicradio.org/hpr.sql) is updated frequently, please copy locally.
Our media is served from our [Community Content Delivery Network (CCDN)](https://repo.anhonesthost.net/HPR/hpr_documentation/ccdn/). Our media is served from our [Community Content Delivery Network (CCDN)](https://repo.anhonesthost.net/HPR/hpr_documentation/ccdn/)
Bug reports, and patches are welcome from anyone without commitment. Bug reports, and patches are welcome from anyone without commitment.
@ -61,5 +61,5 @@ In order to contribute you need to [create an account](https://repo.anhonesthost
Once you have set up your account, you will need to set up your local Once you have set up your account, you will need to set up your local
development environment. [Instructions here](set-up-development-environment.md) development environment. [Instructions here](set-up-development-environment.md)
Changes can be submitted as described in [hpr3797 :: How to submit changes to HPR](https://hackerpublicradio.org/eps/hpr3797/index.html). Changes can be submittted as described in [hpr3797 :: How to submit changes to HPR](https://hackerpublicradio.org/eps/hpr3797/index.html).