Database/hosts_list.tpl, Database/hosts_showcount.sqlite.sql: examples
of using 'query2tt2'
Database/query2csv, Database/query2json, Database/query2tt2: minor bug
fix relating to '-dbarg=ARG' option
Database/edit_episode: updated a message
Database/query2csv, Database/query2tt2: updated both scripts to handle
SQLite and MySQL databases; both take (almost) the same options;
both query a database in a similar way with arguments to match
placeholders; 'query2tt2' takes a TT² template and options to feed
to it; converted the database connection section to a function
'db_connect'; both have comprehensive POD documentation.
Community_News/.make_shownotes.cfg: more comments
Community_News/make_email: added colour test for pod2usage
Community_News/make_shownotes: fixed a bug when using
-lastrecording=DATETIME. Added reporting of expanded output file
names; tidying and updates to POD documentation.
Community_News/recording_dates.dat: added test entry for May 2025
Community_News/.make_email.cfg: Removal of the SQLite database
reference. No longer needed.
Community_News/make_email: A lot of updates relating to confusing
variable names. Rationalising the three code paths to compute the
recording date (-date=DATE), the review month (-month=DATE), and
the default using today's date to determine needed date
information. Removed all references to the database because we
don't need it any more. New date computation functions. A lot of
POD updates, including an EXAMPLES section.
Community_News/make_email: Many bugs relating to the review month versus
the recording date. Some of these must have been there for 12 years!
Rerwrote several parts and added a function for converting Perl date
formats. Added '-force' option to turn off checks when the script is
run in the month after a review month (not uncommon, but fairly
infrequent). Fix for writing bad records to the date cache file.
Updates to POD documentation.
Community_News/make_email_template.tpl: Added new shared variables for
the review year and month as opposed to the recording year and
month. Improved the text explaining the "handouts" versus the final
show notes. Removed the reference to the target show itself. This
used to be where the "handout" notes were written, then they were
replaced before show release. This isn't being done any more.
Community_News/recording_dates.dat: Updated with the correct recording
date (and time) for the review of shows in March 2025.
.make_email.cfg: New configuration file to simplify the original options
to 'make_email'
.make_shownotes.cfg: New configuration file to simplify the original
extremely obscure options to 'make_shownotes'
collect_HPR_database: Script to simplify the collection and setup
of MySQL dumps from the HPR server and conversion to a SQLite
database.
make_email: Many changes to make the script simpler to use. It looks for
all files in the same directory as the script. Reduced the number of
options and added a new configuration file. Now reads and writes
a date cache file (defined in the configuration file) where it
writes the date and time of the next recording. Now uses a local
SQLite database rather than linking to the live HPR database (more
secure). Takes an output file name (with optional '%s'). Functions
for loading and updating the date cache (also used by
'make_shownotes'). Doesn't attempt to generate a real mail message,
just something that can be cut and pasted into a mail client.
make_email_template.tpl: TT2 template for generating the mail message.
This whole function was moved from the script itself to this
templating system, making it all a lot simpler.
make_meeting: Minor updates. This script is probably obsolete.
make_shownotes: Almost totally rewritten. It looks for all files in the
same directory as the script. Reduced the number of options and
added a new configuration file. Now reads a date cache
file (defined in the configuration file) where 'make_email' has
written the date and time of the next recording. Now generates
output files rather than writing to the live HPR database. These
files can be added to the database on the 'hub' using existing
workflow(s). One of the files generated is a stand-alone full HTML file
for circulation to volunteers recording the show. The others are the
HTML snippet to add to the database, and a JSON version for use in
the hub workflow. The full HTML gets the expanded comments and
contains markers of comments already read or missed last month. This
version computes the episode number and date which will be used to
post the resulting show (previously reserved slots were searched for
in the database). The extremely complex query that collects comments
has been thoroughly tested and enhanced and seems to be reliable.
Dropped the "Any Other Business" section (and all code relating to
it in the script and the template).
shownote_template.tpl: Soft link to the latest template. Doing this
needs consideration given that the configuration file could just
reference the appropriate file. This technique may just be
a nuisance.
shownote_template11.tpl: Previous template, updated for the last release
of 'make_shownotes'. Now replaced.
shownote_template12.tpl: New template without AOB capability.
InternetArchive/future_upload: now updates the state of shows
InternetArchive/reformat_html: new Perl script to reformat the HTML
originally found in the HPR database in the 'notes' field to the format
required in the 'description' field of an item on the IA. It reads
from STDIN and writes to STDOUT.
Show_Submission/extract_images: some tidying, addition of more POD
documentation.
Show_Submission/query2csv, Show_Submission/query2json: these softlinks
were turned to hard links to make them more visible in the Git repo.
The files are in the 'Database/' directory.
Show_Submission/extract_images: removed unwanted modules, added POD
documentation, added 'coalesce' routine, adjusted to use the
absolute paths to the input files and generated image files. The
latter are always written to the directory where the HTML resides.
Corrected logic around overwriting image files and '--force'.
Still an 'alpha' version subject to more testing.
Show_Submission/extract_images: new script to read an HTML file looking
for 'data' scheme URIs (embedded images), extract them and modify
the HTML to reflect the new source of the image. At present it
writes a generated file name with a sequence number in it, but the
appropriate suffix/extension for the image type. This is an alpha
version which needs further work.
Show_Submission/parse_JSON: attempting to debug a JSON parsing failure.
Show_Submission/author_title.pl: Added the subtitle field taken from the
JSON into the YAML
Show_Submission/do_pandoc_assets: New script to process Markdown assets
files. Not in use.
Show_Submission/copy_shownotes: Changed the location of the function library
Show_Submission/do_brave: Updates to the way local stand-alone HTML is generated for
review purposes.
Show_Submission/do_index: Changed the location of the function library
Show_Submission/do_pandoc: Changed the location of the function library; now uses
'author_title.pl' to generate YAML for Pandoc
Show_Submission/do_parse: Trivial change
Show_Submission/do_pictures: Changed the location of the function library; better
handling of the show specification
Show_Submission/do_report: Changed the location of the function library
Show_Submission/do_update_reservations: Changed the location of the function library
Show_Submission/fix_relative_links: Added features 'say' and 'state'
Show_Submission/parse_JSON: New checks: notes too short, trailing spaces on title,
summary and tags (needing JSON changes). Check for Markdown in the
assets (see 'do_pandoc_assets'). New 'trim' function.
InternetArchive/recover_transcripts: Bash script to be run on 'borg'
which collects files missing on the IA ready for upload as part of
the missing asset repair process.
InternetArchive/repair_assets: Bash script to take assets from the IA
(after they had been repaired on 'borg') and copy them to the HPR
server for the notes to access. The local machine, where this was
run, was used to store files being uploaded. The planned script to
modify the notes to reflect the new file locations was never
finished. Notes were edited with Vim using a few macros.
InternetArchive/repair_item: Bash script which is best run on 'borg',
which repairs an IA item by comparing the files on the IA with the
files on 'borg' (or a local machine). These files are either in
'/data/IA/uploads/' or in the temporary file hierarchy used by
'recover_transcripts' (which calls it). Used after a normal IA
upload to check for and make good any missed file uploads (due to
timeouts, etc). Also used during asset repairs, but that project is
now finished.
InternetArchive/snapshot_metadata: Bash script which collects detailed
metadata from the IA in JSON format and saves it locally (run on
a local PC). Older shows on the IA often contained derivative files
which were identified by the script 'view_derivatives'. These files
were never needed, they were IA artefacts, so can be deleted (see
the script header for how).
InternetArchive/view_derivatives: Perl script to interpret a file of
JSON metadata from the IA for an HPR show in order to determine the
parent-child hierarchy of files where there may be derivatives. We
don't want IA-generated derivatives, but this process was hard to
turn off in earlier times. Generates a hierarchical report and
a list of unwanted derivatives (see 'snapshot_metadata' for more
details of how this was used).
InternetArchive/repair_assets: Accidentally reverset the "sanity check"
logic, so put it back the right way!
InternetArchive/view_derivatives: Started on the POD documentation but
didn't get very far.
InternetArchive/view_derivatives: New Perl script which Reads JSON
metadata from the IA and builds tree-like structures linking
original and derived files on the IA. It reports these trees and
saves a subset of derived files in an output file to be used for
deletion. In general we do not want derivatives, we make them
ourselves. Older software had no reliable way to prevent them.
InternetArchive/future_upload: Added logging and debugging
InternetArchive/ia_db.sql: Added new tables
InternetArchive/recover_transcripts: New script to run on 'borg' and
copy missing files from the backup disk to the IA
InternetArchive/repair_assets: More comments, including one about a bug in the design.
InternetArchive/repair_item: Fix relating to octal numbers (if there are
leading zeroes in a number). '_DEBUG' is now in the function
library. Added comments to explain obscure stuff.
InternetArchive/snapshot_metadata: New Bash script (to run on my
desktop) which collects metadata for a show and stores in in the
'~/HPR/IA/assets' directory. Runs 'view_derivatives' on it to find
derivative files for deletion.
InternetArchive/tidy_uploaded: Moves files and directories containing
uploaded files into a holding area for later backup. Added
debugging, logging and a 'force' mode.
InternetArchive/upload_manager: Manages 'ia.db' (on my workstation).
Needs many updates which have just started to be added.
InternetArchive/weekly_upload: Old script, now obsolete.
