HPR/hpr_documentation
8
0
Fork 2
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Update queue documentation

Browse Source
This commit is contained in:
Ken Fallon
2026-05-27 16:48:05 +02:00
parent aa3a681e5c
commit b9a461eb1b
2 changed files with 218 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files

289
queue.md
View File

@@ -1,12 +1,15 @@
# The queue
# The Queue
The queue is necessary as "Hacker Public Radio ... releases shows every weekday Monday to Friday."
## Why release like this ?
Back in 2010, shows were getting released on an ad hoc basis. Some days there were as many as two or even three shows released per day.
More often than not though, there was no show at all. As time went on the period between no shows increased.
After a time it was unclear if HPR was alive. This was reflected in a steady decline of subscribers.
Back in 2010, shows were getting released on an ad hoc basis.
Some days there were as many as two or even three shows released per day.
More often than not though, there was no show at all.
As time went on the period between no shows increased.
After a time it was unclear if HPR was alive.
This was reflected in a steady decline of subscribers.
One of the policies that we decided on was that "one show a day - every day, builds trust and retains listeners."
This has turned out to be true based on the steadily increasing subscriber base, and since then lots of research has been done that supports it.
@@ -22,117 +25,251 @@ For those that prefer to have the shows as soon as they are posted, we also have
## Scheduling
Hacker Public Radio releases shows on a very regular schedule and ideally the flow of shows in would match the flow of shows out.
> HPR can only control the release rate (fixed at 5 shows a week), and has no control over the rate at which shows are submitted.
We see however that the
Sometimes there are very few shows in the queue resulting in problems for the admins in getting shows to air.
![Diagram of a ball placed in an unstable equilibrium](images/Diagram_of_a_ball_placed_in_an_unstable_equilibrium.OrdinaryArtery.CC-BY-SA_4.0.png)
Other times there are a lot of shows in the queue resulting in hosts waiting a long time for their show to air.
<a href="https://commons.wikimedia.org/wiki/File:Diagram_of_a_ball_placed_in_an_unstable_equilibrium.svg">OrdinaryArtery</a>, <a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, via Wikimedia Commons
https://davecturner.github.io/2016/12/01/rate-limiting.html
Despite the problems we haven't missed a day since 2010-09-21, and sometimes that is the only thing that keeps the Janitors going.
![](images/rate-limiting.David_Turner.CC-BY-SA_4.0.png)
Copyright © David Turner<br>This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
Copyright © David Turner<br>This work is licensed under a
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
## History of scheduling on HPR
Over the years we tried various different methods to ensure one show every weekday Monday to Friday.
<ul>
<li>To begin with hosts signed up for a particular day based on a published [calendar](https://lists.hackerpublicradio.com/pipermail/hpr/2008-January/000000.html).</li>
<ul>
<li>This failed as hosts could not meet the scheduled time, or found the pressure too much.</li>
</ul>
</li>
<li>Then shows were released that were available on the FTP server.
<ul>
<li>This lead to arguments as there appeared to be no order to the when the shows were posted.</li>
</ul>
</li>
<li>Then we switched to a first come first out method.
<ul>
<li>This lead to having too many shows from the one host or on one topic, where a hosts posted a series of shows at the one time.</li>
<li>This also lead to arguments about time critical shows been left too late.</li>
</ul>
</li>
<li>Then we went through several mechanisms of rules, first in first out, except for new hosts, interviews, etc, etc.
<ul>
<li>Needed to be updated too frequently</li>
<li>It was not clear what the rules were, and shows came out in a seemingly in random order.</li>
<li>The Janitors were been accused of favouring other peoples shows</li>
</ul>
</li>
<li>Pick a slot from a schedule where the hosts themselves got to pick a slot
<ul>
<li>Works well but can lead to <a href="https://en.wikipedia.org/wiki/Mechanical_equilibrium">unstable equilibrium</a> between supply and delivery.</li>
</ul>
</li>
<li>Current: Schedule with free slots been filled from a reserve pool.
<ul>
<li></li>
</ul>
</li>
</ul>
## Scheduling Rules
<ol>
<li>You must have your audio recording ready to upload <strong>before</strong> you pick a slot.</li>
<li>All hosts must leave at least 9 slots (approximately two weeks) between their shows.</li>
<li>New hosts, Interviews, and other time critical shows should use the first free slot.
Otherwise, when the queue is filling up then leave some slots free for new contributors by selecting a slot in the first empty week.</li>
</ol>
</p>
We have a <a href="https://repo.anhonesthost.net/HPR/hpr_documentation/raw/branch/main/workflow/SchedulingGuidlinesFlowChart.odg">flow diagram</a> if that helps.
---
# Updated version of [hpr4195 :: Hacking HPR Hosts](https://hackerpublicradio.org/eps/hpr4195/index.html)
> Social Engineering more contributions to HPR by picking when to publish your show
Hosted by Ken Fallon on Friday, 2024-08-30 is flagged as Clean and is released under a CC-BY-SA license.
Tags: HPR, Queue, Scheduling, Buffering.
## There’s no Business like …
While we may be "dedicated to sharing knowledge", we are competing for the time and attention of our Audience.
Therefore we are in the Entertainment Business.
The clue is in this statement from the about page.
<blockquote>
Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases shows every weekday Monday through Friday.
</blockquote>
Let’s compare that to others.
Any event promoter needs to provide the Who, What, Where, and When, to their potential audience.
<ul>
<li>U2, UV Achtung Baby Live playing the Las Vegas Sphere, from 23 to the 30 June 2024.</li>
<li>Richard III in the Globe Theatre, All Summer. (The resident players is implied)</li>
<li>BBC News at Ten. (The what and when are in the name, BBC News Team is implied, as is Daily)</li>
<li>Season 2 of Firefly, returns to Netflix in the Fall.</li>
</ul>
A theater will have an address and a schedule for when the events occur.
On TV and radio they have predefined channel locations, and often have 24x7 schedule of programs.
For Hacker Public Radio (HPR) our "venue" or channel is our RSS Feed, and our schedule is a show every weekday Monday through Friday.
A podcast production enterprise, like the NPR, BBC, etc have permanent staff who’s day job is to come in and create content.
Other approaches used by Netflix, or Disney+, etc is to commission external parties to record unique content.
They might also just purchase in shows.
Regardless of the approach, they all have a mechanism to meet the production schedule.
Unlike other podcasters, HPR has no control of our supply chain.
We do still have a contract to deliver one "product" a day Monday to Friday,
but we also have no control over our distribution channels.
I think it’s important to understand just how much energy goes into managing this balancing act.
It’s the absolute <strong>core</strong> of the project, and is what takes up most of our time and energy.
# Feeding the Queue
We have to feed the queue.
## Control of our supply chain
<blockquote>
A supply chain, .. is a complex logistics system that consists of facilities that convert raw materials into finished products
and distribute them to end consumers or end customers.
Meanwhile, supply chain management deals with the flow of goods within the supply chain in the most efficient manner.
</blockquote>
<a href="https://en.wikipedia.org/wiki/Supply_chain">From Wikipedia, the free encyclopedia</a>
Our supply chain is entirely dictated by the generous hosts who donate their time to recording a show.
Therefore the Janitors have no control over when shows are sent in.
As Janitors, we can only contact the community to remind them to send in shows.
We need your help to manage this.
### Boom/Bust Supply
![Diagram of a ball placed in an unstable equilibrium](images/Diagram_of_a_ball_placed_in_an_unstable_equilibrium.OrdinaryArtery.CC-BY-SA_4.0.png)
<a href="https://commons.wikimedia.org/wiki/File:Diagram_of_a_ball_placed_in_an_unstable_equilibrium.svg">OrdinaryArtery</a>,
<a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, via Wikimedia Commons
Usually there is a burst of contributions after a "Call for Shows",
which is itself as a result of a lull in the amount of contributions.
This leads to boom and bust/saw tooth delivery of shows.
There is a painful behaviour that the Janitors observe after a "Call for Shows".
<ul>
<li>There is a burst of contributions all taking the first available slots.</li>
<li>The queue quickly fills up the upcoming weeks.</li>
<li>It takes time for the "Call for Shows" to get to everyone.</li>
<li>A potential host is late hearing the "Call for Shows" and sees a full queue, resulting in them not submitting a show.</li>
<li>Worse is that it instills the feeling of HPR "<a href="https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf">Crying Wolf</a>",
incorrectly assuming the subsequent "Call for Shows" can safely be ignored.</li>
<li>After a few weeks our queue is empty and we need to put out another "Call for Shows", which that host ignores.</li>
</ul>
The timely delivery of shows is an inherent challenge with volunteer contributions.
Fortunately this is a well understood problem known as <a href="https://en.wikipedia.org/wiki/Queueing_theory">Queueing theory</a>,
and we have implemented the <a href="https://hackerpublicradio.org/contribute.html#schedule_reserve_pool">Reserve Pool</a>,
as a means to regulate/<a href="https://en.wikipedia.org/wiki/Data_buffer">buffer</a> the incoming delivery with outgoing supply.
<blockquote>
The reserve queue is intended only to be used in the cases where there is still a gap in the schedule one week prior to release.
This was known as the emergency queue, but now can also be used when the hosts don’t care when the shows are scheduled.
They will be used on a first in first out basis, when there is no conflict with the scheduling guidelines.
These shows contain a message alerting listeners to the fact that we had free slots that were not filled.
</blockquote>
### Scheduling Rules
When you are contributing a show, you decide when to post your show.
The choice of slot may even encourage others to submit a show themselves.
[hpr4195 :: Hacking HPR Hosts](https://hackerpublicradio.org/eps/hpr4195/index.html)
Our observations show that there is a <a href="https://en.wikipedia.org/wiki/Goldilocks_principle">Goldilock Zone</a>
where there are just the right amount of free slots to encourage contributions.
Can I ask you to do something for me.
Please leave 2 weeks between the shows unless it's an interview or a urgent show.
I know you have thoughts on this but filling the next free slot is not the best approach for keeping the queue filled.
#### Too Many free slots
I go into the logic behind this in this show https://hackerpublicradio.org/eps/hpr4195/index.html. Essentially its
managing unstable equilibrium
<https://en.wikipedia.org/wiki/Mechanical_equilibrium>.
When there are too many free slots some people get disheartened and don’t want to contribute to a dying project.
Having no open slots means that hosts think that the project is healthy
and they don't send in a show.
On the other hand too many free slots can send regular hosts into a panic to fill them.
We all suffer from this, and it can lead to burnout.
Having too many open slots gives the feeling that HPR is failing as a
project so they don't send in a show.
Fortunately we now have the <a href="https://hackerpublicradio.org/contribute.html#schedule_reserve_pool">Reserve Pool</a>,
where they janitors can post their backup shows when they are needed.
If you already posted shows, then don't be afraid to leave empty slots.
You already have done your bit, it's up to the rest of them to help out.
The idea that some shows are sub par because they are rushed in to fill free slots can now be put to rest.
All the shows in the <a href="https://hackerpublicradio.org/contribute.html#schedule_reserve_pool">Reserve Pool</a>
are there because the Host did not feel the need to rush the shows out.
Over the last year we have been implementing the reserve queue to fill
empty slots and it's been working very well. We have seen several hosts
send in a show as a result of the announcement we add reminding people
that "HPR needs listeners to contribute shows in order to survive".
#### Too many free slots
On the other hand seeing too many free slots some people get disheartened that their show won’t be aired for weeks,
so end up not recording a show in the first place.
Scheduling Guidelines
#### Hacking Human Behaviour
1. New hosts, Interviews, and other time critical shows should use the
first free slot.
2. Always try and fill any free slots that are available in the
*upcoming two weeks*.
3. When the queue is filling up then leave some slots free for new
contributors.
4. Post non urgent shows into the first empty week.
5. If you are uploading a series of shows then post them one every two
weeks.
6. If you have a non urgent show that is timeless, then add it to the
Reserve Queue <https://hackerpublicradio.org/about.html#reserve_queue>.
So the HPR Community can influence the supply chain by been smart about how we schedule the shows.
https://repo.anhonesthost.net/HPR/hpr_documentation/raw/branch/main/workflow
/SchedulingGuidlinesFlowChart.odg
When you upload consider the <a href="https://hackerpublicradio.org/contribute.html#scheduling_rules">Scheduling Rules</a> when picking your slot.
This way <strong>your</strong> (person reading this) actions,
give the HPR Community complete control over the supply of shows in a general sense.
Remember the HPR Community have not missed a day since September 2009.
# Distribution Channels
Getting our podcast distributed is no problem what so ever.
## History of scheduling
While NPR, BBC, Netflix, Disney+, etc can afford to record unique content, unique content is very, very expensive.
Amazon, Apple and Spotify may have the resources to do this,
but they and others, make use of the freely available content to inflate their inventory of content.
To begin with hosts signed up for a particular day based on a published [calendar](https://lists.hackerpublicradio.com/pipermail/hpr/2008-January/000000.html).
This failed as hosts could not meet the scheduled time, or found the pressure too much.
We provide these platforms with our feed and our blessing. So long as they adhere to the
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License</a>.
Our prime directive at work.
Then shows were released that were available on the FTP server.
This lead to arguments as there appeared to be no order to the when the shows were posted.
We publish a RSS feed, and that feed is then picked up by others such as
<a href="https://archive.org/details/hackerpublicradio">Archive.org</a>,
<a href="https://music.amazon.fr/podcasts/9d9e6211-ff78-4501-93b6-6a9e560c4dbd/hacker-public-radio">Amazon Music</a>,
<a href="https://podcasts.google.com/feed/aHR0cDovL2hhY2tlcnB1YmxpY3JhZGlvLm9yZy9ocHJfcnNzLnBocA">Google Podcasts</a>,
<a href="https://www.iheart.com/podcast/256-hacker-public-radio-30994513/" target="_blank">iHeart Radio</a>,
<a href="https://podcasts.apple.com/us/podcast/hacker-public-radio/id281699640">iTunes</a>,
<a href="https://www.listennotes.com/de/podcasts/hacker-public-radio-hacker-public-radio-mNH-jsI7LcJ/">Listen Notes</a>,
<a href="https://www.mixcloud.com/hackerpublicradio/">MixCloud</a>,
<a href="https://player.fm/series/hacker-public-radio">PlayerFM</a>,
<a href="https://www.podchaser.com/podcasts/hacker-public-radio-76781">Podchaser</a>,
<a href="https://nl.radio.net/podcast/hacker-public-radio">Radio.net</a>,
<a href="https://open.spotify.com/show/7e2hYcnHj9vKgUzsIOf4r3">Spotify</a>,
<a href="https://toppodcast.com/podcast_feeds/hacker-public-radio/">Top Podcasts</a> and now for some reason
<a href="https://www.imdb.com/title/tt30528560">IMDB</a>.
Then we switched to a first come first out method.
This lead to having too many shows from the one host or on one topic, where a hosts posted a series of shows at the one time.
This also lead to arguments about time critical shows been left too late.
We have no control over what they do with the feed, how often the use it, if they cache it, if they use the images from it, if the show the
explicit tag, or as in this case, if they <a href="https://repo.anhonesthost.net/HPR/hpr_hub/issues/11">display the host</a> or not.
The followed many attempts to
That’s why you can help by taking up the mop and becoming the Janitor for your Distribution Channel.
---
, and arguments abotu
, and seemingly in random order.
So we need some way to control the uppredictable nature of getting shows, with the steady
We haven't missed a day since 2010-09-21 and sometimes that is the only thing that keeps the Janitors going.
Summary:
Fact:
HPR can only control the release rate and has no control over the
rate at which shows are submitted.
Sometimes there are very few shows in the queue resulting in problems
for the admins in getting shows to air.
Other times there are a lot of shows in the queue resulting in hosts
waiting a long time for their show to air.
# Mail list Discussions on the topic
On 8/28/21 08:12, e8hffff wrote:
> It's painful to only have one show a week day, and that's double pain if only

10
workflow/processing_a_show.md
View File

@@ -17,8 +17,8 @@ The files are downloaded by trusted Janitors that have the ability to `scp`/`rsy
The directory structure is based on a combination of fields separated by the underscore (`_`) delimiter.
- Upload date and time `UTC_TIMESTAMP()` at reservation time.
- The requested `episode number` or `9999` if the reserve queue is to be used.
- The requested `epidode date` or `1970-01-01` if the reserve queue is to be used.
- The requested `episode number` or `9999` if the reserve pool is to be used.
- The requested `epidode date` or `1970-01-01` if the reserve pool is to be used.
- The random unique key for this request.
```
@@ -30,9 +30,9 @@ The upload will produce at a minimum a `shownote.json` file. It may also include
Addition files and images may be provided by the host, eg: images, scripts, pdf documents etc.
Shows destined for the reserve queue are moved from the upload directory and placed in the reserve directory using the script [rename-reserve.bash](https://repo.anhonesthost.net/HPR/hpr-tools/src/branch/main/workflow/rename-reserve.bash).
Shows destined for the reserve pool are moved from the upload directory and placed in the reserve directory using the script [rename-reserve.bash](https://repo.anhonesthost.net/HPR/hpr-tools/src/branch/main/workflow/rename-reserve.bash).
This is run manually by the Janitors as it checks to see if a url to the show was provided, and attempts to download the linked file. When new hosts submit a show directly to the reserve queue, the Janitors will resubmit it to the first available slot in the normal queue. This is because new hosts need to have an entry created in the `hosts` table, but also because it gives the community an opportunity to welcome the new host.
This is run manually by the Janitors as it checks to see if a url to the show was provided, and attempts to download the linked file. When new hosts submit a show directly to the reserve pool, the Janitors will resubmit it to the first available slot in the normal queue. This is because new hosts need to have an entry created in the `hosts` table, but also because it gives the community an opportunity to welcome the new host.
It renames the directory structure based on a combination of fields separated by the underscore (`_`) delimiter.
@@ -45,7 +45,7 @@ It renames the directory structure based on a combination of fields separated by
```
2339680845_987_4bd713699e5bc0978d5fef85a60f09bc7f70ef3488624_Emperor_Ming_Top_tips_for_time_travel
```
Reserve shows are downloaded and submitted by the Janitors on behalf of the host. This follows the normal posting process where the host and Janitors are cc'd on the notification emails. The only difference is that the audio is edited to include a notification that the show is from the reserve queue, and that it is the Janitors that upload the show via the supplied link.
Reserve shows are downloaded and submitted by the Janitors on behalf of the host. This follows the normal posting process where the host and Janitors are cc'd on the notification emails. The only difference is that the audio is edited to include a notification that the show is from the reserve pool, and that it is the Janitors that upload the show via the supplied link.
## Adding the host to the `hosts` table