hpr-knowledge-base/hpr_transcripts/hpr4688.txt

Episode: 4688
Title: Downloading Podcasts with a Shell Script
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4688/hpr4688.mp3
Transcribed: 2026-07-31 16:16:47 (official HPR transcript)

---

This is Hacker Public Radio Episode 4688, for 2026-07-22
Today's show is entitled, "Downloading Podcasts with a Shell Script"
The host is Whiskeyjack and the duration is 00:30:19
The flag is Clean, and the license is CC-BY-SA
The summary is "The basic principles of downloading podcasts using your own shell scripts."
In this episode, I will describe techniques for downloading podcasts using basic shell commands
such as WGet.
I will illustrate this using a bash script that can be used to download HPR Podcast.
Even if you do not have any interest in downloading your podcasts using this method, you
may find some of the methods useful or interesting.
It is the principles that are discussed here that are important, rather than the implementation.
I realize that there are already a number of different podcasts download programs available,
including at least one written in bash.
However, you may feel that none of these suit how you wish to do things and want to create
your own system tailored to your specific needs.
If so, then I hope that the following is of some use to you.
If not, then you may still find some of the things discussed here to still be of interest.
Some of the subjects I cover include using WGet to a user-defined file name, parsing XML
with XMLLint, using I notify weight to trigger an action when a file is created or modified,
using notify send to send a message to the notification area, and a way of allowing
a Cron job to send a message to the user interface.
Background There has been an ongoing discussion in comments to some HPR episodes about
problems downloading HPR podcast episodes.
Apparently some people have been experiencing problems with the way the episode URLs are structured.
I am afraid that I don't fully understand the nature of these problems, so I will be addressing
that problem directly.
Instead, I will present a bash script that I have written which can be used to download HPR
podcasts.
This bash script can be run using Cron to automatically fetch new HPR podcasts and save
them to a designated directory.
This is a simplified version of a script that I have used for years to download HPR
and other podcasts.
I won't try to read the full bash script out in this podcast, as that would be a bit
dull to listen to.
I will instead describe what each section does and why I chose to do things that way.
Perhaps other people can offer suggestions of better ways to do things.
I will post the full bash script in the show notes.
Fetching podcasts.
The standard way of distributing podcasts is to publish an RSS feed containing URL links to the
audio files.
RSS is a very long established and widely supported mechanism for this and other purposes.
An RSS feed is basically an XML document which can be accessed over HTTP.
These URLs contained in the RSS XML document can then be used to download the actual audio
files such as MP3 or Ag files.
Basically, what we need to do is the following.
Download the RSS XML document.
Extract the URL links to the audio files.
Compare the lists of these links to a previously saved list and see which ones are new
and which ones are ones that we previously downloaded.
Make a list of the new URLs.
Go through this list of new URLs and download each of the new audio files.
Check to see that we actually received the new audio file.
Add the URLs of the files we successfully downloaded to our saved list of podcasts URLs.
In addition to this, we would like to have the above-happened automatically in the background
without our having to take any action on our own.
We may wish to receive a notification of one new podcast has arrived however.
We would probably also wish to receive notification of any errors or failures.
Fetching podcasts, the preliminaries, our desire to be able to run the script automatically
imposes some requirements on our solution.
Dischedual the script, we will use Cron.
Cron is a Linux facility to run scripts on a schedule.
One of the side effects of using Cron however is that we need to specify the full path to
locations where we intend to keep any data files plus also the full path to where we intend
to put the downloaded podcasts.
So the first thing we need to do in our script is to specify a number of different values
for things like file locations as well as things like the URL for the HPR RSS feed and several
other things.
I will skip over the details of these, although I may make reference to them later.
Get the RSS data.
The first thing of real substance to do is to fetch the Cron RSS feed data.
I have put this in a bash function called, get RSS URL data.
The contents of this function are a one-liner but with a number of elements chained together
through pipes.
Now loading the RSS XML document.
First we use WGet, which is a standard command on most Linux distros.
We specify four things.
First we set a timeout, I have chosen 20 seconds.
Next we set a retry limit, I have chosen 3.
Then we specify the output of WGet is sent to standard out rather than saved as a file.
This is done by using the dash capital O option followed by a space and then a dash.
The capital O option is usually used to specify a file to save the output to, but when
used with a dash causes the output to go to standard out.
Then we specify the URL of the HPR RSS feed, contents of the XML document.
This gives us the HPR RSS XML document.
There are about 5,000 lines in this RSS document.
Most of these lines are the show note, which are also included in the feed.
Extracting the podcast episode URLs.
There are only 10 lines of the document to contain information that we are interested in
however.
These lines are enclosed in and closure XML tags.
We just need to find those lines and separate out the URLs.
Standard command line tools.
There are two ways that we can do this.
One is to use a combination of grip, said, and cut.
Grab can find the lines containing the enclosure tags.
Said and cut can extract the URL from the surrounding extraneous data.
However, this method does not discriminate between real enclosure tags in the data portion
of the RSS feed and enclosure tags in the show notes which are included in the feed from
episodes such as this one.
This may be an acceptable problem in practical terms, but we can do better.
Using an XML parser.
The other method is to actually parse the XML document.
There are at least two command line XML parsers that I am aware of.
These are XML lint and XML starlet.
I have used XML lint in this example.
I have not used XML starlet, so I can't offer any comment on how easy or difficult to use
it is.
I won't give a detailed explanation of all the things that XML lint can do.
It has many features, most of which, as the name suggests, have to do with finding formatting
problems with the XML itself.
Describing everything it can do would be at least one episode in itself.
I will instead just give the particular command used and explain each element of it.
In this example, assume that we are piping the output of WGET directly into XML lint.
The complete command is XML lint that is XML lint that is XML lint.
Space--X-Path that is XP-A-T-H space double quote slash slash channel that C-H-A-N-N-E-L slash item slash
enclosure slash ampersand URL double quote space-space pipe symbol space cut that C-U-T space
dash D single quote double quote single quote space dash F2.
In this example, XML lint is the name of the command.
slash dash X-Path tells it to parse the document according to the string which follows.
slash slash channel slash item slash enclosure slash ampersand URL tells it to find a series
of tags in the hierarchy of channel followed by item followed by enclosure and then extract
the URL attribute from the enclosure tag.
The dash which follows it tells it to look for input from standard in rather than from
a file.
The result is a string which has the URL attribute name and equal sign and the URL that we want
enclosed in quotes.
To get just the URL itself, we pipe the output from XML into cut using the double quote
characters as the limiters.
We then save the result in a temporary file.
Finding the new episodes.
Next, we wish to find the new podcast episodes.
Each H-Pierre episode is identified by a unique URL.
This means that if we save the URLs of episodes that we have already downloaded, we just
have to look for the URLs that do not appear in this saved list.
The easiest way to do this is to take our two lists of URLs, sort each into temporary
files and compare the sorted URLs using the Com command.
This is simple but has a drawback.
Some podcasts occasionally change distributors.
When they do this, the old podcasts are republished with new URLs and you end up downloading
a lot of old episodes over again.
With H-Pierre, we could get around this by extracting just the file name and looking for
that instead of the full URL.
I will however leave that problem as an exercise for the student and just accept that if
the URL format changes, we may end up downloading old episodes over again.
Since the feed has a maximum of only 10 episodes in and however, that isn't really that big
of a problem.
It would be more of a problem with podcasts which have very large numbers of episodes in
their feed, but the solutions of those will be feed specific.
Downloading the new podcasts, we should now have a list of URLs for the new podcasts we
do not already have.
Typically, this should be only one file, but there could be several, or even as many
as 10, if we have not turned on our computer in a while.
Therefore, we need to iterate through the file of new podcasts URLs and download each
one.
Before we do that however, we should check to see if there is in fact anything new to download.
To do this, simply use WC-L to count the number of lines in the list of new URLs and
saves a resulting number.
If this number is zero, there is nothing to download and we can skip the download step.
As an additional check, we should see if the number of downloads exceeds some threshold
value that we wish to set.
This is not a major problem with HPR, but some podcasts have hundreds of files in their
RSS feed rather than just the most recent ones.
If we do exceed our download limit, then we need to log in error and skip downloading.
Assuming there are no problems so far, however, the first thing we need to do is to extract
the name of the audio file from the URL.
We can do that using the base name command.
We will use this to specify the name that we use when we save the audio file.
HPR has a very well-formed file name.
Some podcasts do not however, and for those you would need to construct some sort of suitable
name using either information found in the URL or simply creating a name using a timestamp.
Next, we download the audio file using WGet.
This is similar to how we downloaded the RSS feed, but with a few changes.
One is that I have increased the timeout to 90 seconds.
This may not have been necessary, but it seemed like a good idea.
The next is that when specifying the output file name using dash capital O, we use the
file name we extracted from the URL.
The third is that we specify a destination directory using the dash capital P option.
After WGet has finished, including any retry said it had to do, we next check that the expected
new file is both present and not empty.
We can do this using an if statement, wished with the dash S option.
If the file was found and not zero, that we add that URL to a temporary list of downloaded
URLs.
If the file was not present or with zero length, we output an error message to an error
log.
I will come back to this point later.
Next, if there is more than one podcast to download, we sleep for three seconds.
While not strictly necessary, it is considered to be polite to not hammer a server repeatedly,
but rather to put a small delay between file downloads.
After we have downloaded all the audio files in our list, we can add the list of URLs
for the files downloaded to the permanent list.
While we are at it, we should use tail to trim the permanent log to keep it from growing
indefinitely.
This limit should be several times bigger than the number of files in the RSS feed.
In this case, I selected 50.
Finally, we write any errors to the permanent error log and also write those same errors
to another file used to signal errors for display to the user.
We have now successfully downloaded, at least one HPR podcast.
To notify the user of events, it would be convenient to be informed of new podcast downloads
when they occur and also to be notified of any errors.
One of the limitations of crime jobs is that they cannot access user interface.
This means that we cannot readily send a message directly to the notification system to inform
the user of the presence of due podcasts or of errors.
I notify wait.
The solution to this is to use I notify wait to monitor particular files and directories
for changes.
The man page for I notify wait states the following.
I notify wait efficiently waits for changes the files using Linux's I notify interface.
It is suitable for waiting for changes to files from shell scripts.
It can either exit once an event occurs or continually execute an output events as the occur.
End of quote.
In many Linux test rows, I notify wait is provided by the I notify tool package that is
I and O T I F Y dash T O O L S.
I won't go over all the features of I notify wait.
Instead, I would just describe how to use it for our purposes here.
I notify wait modes.
I should point out first though that I notify wait operates in two different modes.
In the normal default mode, it exits after being triggered by an event and it must be
reestablished again in order to resume monitoring.
In monitor mode, which is enabled by using the dash M option, it runs indefinitely, responding
to events.
I will use the default mode here.
The man page for I notify wait provides a simple example that we could copy and modify
for our purposes.
A great minute examples that you will find are based on this example.
However, it doesn't quite do what we want, so we need to change a few things.
Pod fetch notify.
The first shell script is one which monitors for the arrival of new podcasts and sends
a notification to the user.
I will call this Pod fetch notify.
The complete scripts are in the show notes, I will just provide a brief description here.
Setting up event watches using I notify wait.
Descript is enclosed in a while loop, which runs indefinitely.
In the first line inside the while loop, we call I notify wait.
I notify wait will then block until the event it is told to look for occurs.
In short, execution of the script will wait there until an event occurs.
The names of the events are listed in the man file.
In this case, we are looking for modify, create, and move to.
The latter one being spelled M-O-V-E-D underscore T-O.
Each of these does pretty much as you would expect, reacting to modifying an existing file,
creating a new file, or moving a file to that directory.
Problems when using text editors, I should point out that if you are testing a script
which uses I notify wait, then modifying a file with a text editor may not produce the
results that you may think it would.
Instead, it treats us as a new file with the same name with the original file being erased.
Since I notify wait, attaches itself to the I node rather than the file name, it sees
the file that the text editor changed as being a new file.
If you wish to test this realistically, then use echo to overwrite the file by using I-O
readirection, capturing output.
In my example, I captured the output from standard out into a variable, but I don't do anything
with it.
If you wish to, for example, display the name of the newly downloaded podcast file, then
use the dash dash format option along with an appropriate formatting code.
There are details about this in the man page.
On the next line, we captured the exit code using dollar sign question mark, responding
to exit codes.
If the exit code was zero, then a monitor event was triggered and there should be a new
podcast in the directory.
In this case, we display a message indicating that a new podcast has arrived.
I will describe how to send notifications shortly.
If the exit code was not zero, then an error occurred.
An example of such an error would be if the directory were not present when monitoring
was started.
In this case, we display a message indicating that a fatal error has occurred and then
exit.
Delay for more podcasts.
Finally, we use sleep to wait for some arbitrary period of time to prevent notifications
from being triggered multiple times if several podcasts were being downloaded in succession.
In this case, I chose to wait for 60 seconds.
We have now completed the process and can return to the top of the loop and resume waiting
using I notify wait.
Sending notifications to the user.
I mentioned above about sending notification messages to the user.
In the Nome desktop, notification messages appear from the center of the top bar in a list.
Other desktops or operating systems may have something similar.
To send a notification message to the notification area, you use the notify send command.
This is n-o-t-i-f-y-s-e-n-d.
Simply follow notification send with a quoted string and it will be displayed in the notification
area.
What fetch error notify?
The second shell script is one which notify the user of errors.
I will call this pod fetch error notify.
With this shell script, we set up a watch on a file which contains any error messages from
pod fetch.
This script is very similar to pod fetch notify.
The exceptions are, with I notify wait, we only monitor for modify.
There is no sleep command at the end of the loop.
Instead, we sleep for a few seconds just after getting the exit code from I notify wait.
This helps prevent problems caused by race conditions.
Next, we check the I notify wait exit code.
If it was zero, then we read the error report file and send notification message to the
user containing that error message.
If it was not zero, then we check to make sure that the directory that should contain
the error log exists.
If it does not exist, then we send a notification message to that effect to the user and
terminate the script.
If the directory exists, then we check to see if the error message file used for signaling
exists.
If the file does not exist, then we create it.
One of the reasons for an I notify wait error is that if this file that it is told to monitor
it does not exist, it cannot set up a watch condition.
By creating the file, we correct the cause of the error and allow I notify wait to operate
normally.
Finally, we increment an error counter and check to see if the limit is exceeded.
If there are excessive errors, then sending notification message to the user and exit.
The reason for this is to give the user an indication that the error notifications are
not working for some reason, and there may be a problem that needs looking into.
The error counter is reset every time the I notify exit status is okay, so occasional unexpected
glitches should be something that is ignored.
Of course, podcast fetching errors are something that will probably happen only very rarely
if at all.
So this final step may be seen as an unnecessary embellishment, installing the scripts.
Next I will describe how to install and prepare the scripts to run.
We need to perform the following steps.
First, we need to create a directory to hold the scripts and their associated data files.
Next, we need to create a directory to hold the downloaded podcasts.
Then we must copy the scripts to these directories and make them executable.
Then we must edit the scripts to have the file path in the script, match the locations
of the new directories that we created.
Then we need to install XMLLint or alternatively modify the download script to comment
out the use of XMLLint and enable the alternative method using Grip and set instead.
Then we need to run each script manually from the command line to check for errors.
If podfetch ran correctly, it should download the most recent 10 podcasts during this test.
Adding podfetch to the content.
The above describes how to run the scripts manually.
In order to fetch podfetch cast automatically, we need to add the podfetch script to the
Cron schedule to do this open a terminal.
Type CronTab that CR-O-N-T-A-B-Space-E and then press Return.
A text editor should open up containing the CronTab file.
On to Buntu, this editor is GNU-Dano.
Enter the appropriate Cron parameters.
I will provide an example here for running at 12 pass the hour every three hours.
This is one, two, space, Asterisk, rash, slash, three.
Space, Asterisk, Asterisk, Asterisk, space, Asterisk,
space, Asterisk, Space, Slash, Home, slash, your user name slash the path to where you're
putting the files slash podfetch.sh.
I won't explain Cron in detail here.
The example that I have just given
should be good enough for most people.
The Asteris slash three parameter
will cause it to run every three hours.
The one-two parameter will cause it to run
12 minutes past the hour when it does run.
Checking every three hours should be good enough
for most people, but you can adjust that as you see fit.
I would recommend, however,
that you don't check more frequently than once per hour.
Checking more frequently than necessary,
puts extra load on the distribution servers.
It is very unlikely that you really do need each new podcast
the moment it is available.
I would also recommend changing the 12 parameter
to some other random minute value.
I would suggest avoiding on the hour or on the half hour
as a lot of other people are probably checking at those times
and it would be better to spread the load out more evenly over time.
The file path parameter should, of course,
match the actual path to wherever you have located
in the script, including the correct user name.
Making the notification scripts start automatically.
The two notifications scripts can be made to start automatically.
The exact method to do this may vary according to distribution
or desktop.
On Ubuntu, this is done using the start-up application
preferences GUI program, which should come already installed.
I won't go into details on this here.
As it should be fairly self-evident, how to use it once you see it.
What this program does is to create dot desktop files
in the dot config slash auto start directory in your home directory.
These dot desktop files are all run automatically on startup.
Once you have added the notification scripts,
you will need to log out and then log back in to make them active.
Conclusion.
In this episode, I explained how to write a set of simple shell scripts
to automatically download each new episode of HPR
as it comes out and to notify you of its arrival.
The download script described here is tailored specifically
for use with HPR only.
However, it was derived from a larger script
that downloaded other podcasts as well,
based on information read in from a text file.
If you are feeling ambitious,
you can add those features back into this
to handle all of the podcasts that you listen to.
In a comment to another episode of HPR,
I had said that I would cover ID3 tags in MP3 files,
but this episode is long enough now,
so I will leave that subject for later.
I look forward to seeing you again later
in another episode of HPR Public Radio.
You have been listening to the Hacker Public Radio podcast, at hackerpublicradio.org.
Today's show was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast, then visit the HPR site to find out how easy it really is.
Hosting for HPR has been kindly provided by anhonesthost.com, the Internet Archive, rsync.net, and the HPR Community Content Delivery Network.
Unless otherwise stated, today's show is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.