Updates since 2024-06-15

Database/query2tt2: comment and documentation updates; use of Perl's
    try/catch.

InternetArchive/.make_metadata.cfg: added comments for readability

InternetArchive/make_metadata: bug fix needed now that all shows on the HPR server have
    a directory with assets under it.

InternetArchive/repair_assets: new Bash script in development. Collects
    assets from the IA and uploads them to a new directory on the HPR
    server. Will run 'fix_asset_links' (to repair asset links for their
    new directories) once it is ready.

InternetArchive/repair_item: Bash script which was originally written to
    run on 'borg' and upload files to a new IA item when the uploads
    timed out. Now enhanced to upload missing files recovered from the
    HPR backup disk, such as transcripts.

InternetArchive/repair_item

@@ -3,7 +3,7 @@
 #
 #         FILE: repair_item
 #
-#        USAGE: ./repair_item [-h] [-v] [-d {0|1}] [-D] [-l N] itemname
+#        USAGE: ./repair_item [-h] [-v] [-d {0|1}] [-D] [-l N] [-X] itemname
 #
 #  DESCRIPTION: Repairs an IA "item" (HPR show) if something has failed during
 #               the upload.
@@ -18,20 +18,32 @@
 #               temporarily on 'borg') and determines which have not been
 #               uploaded, then takes steps to perform the uploads.
 #
+#               Version 0.0.10 onwards has the capability to repair an IA item
+#               from the HPR backup disk. This seems to be necessary because
+#               the transcripts were not carried over (although we are
+#               adding them to the IA for new shows now, older ones were never
+#               copied), and there has been a case where none of the assets
+#               were on the IA. The method used it to place the backup files
+#               in the directory 'repairs' under the local IA or
+#               InternetArchive directory. The files are held in the hierarchy
+#               '$item/$item/'. The assets are in the lower directory and the
+#               source file is in the upper one. This emulates the placement
+#               on the IA itself.
+#
 #      OPTIONS: ---
 # REQUIREMENTS: ---
 #         BUGS: ---
 #        NOTES: ---
 #       AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
-#      VERSION: 0.0.9
+#      VERSION: 0.0.10
 #      CREATED: 2020-01-05 22:42:46
-#     REVISION: 2024-06-14 18:03:58
+#     REVISION: 2024-07-12 14:39:38
 #
 #===============================================================================
 
 #set -o nounset                              # Treat unset variables as an error
 
-VERSION="0.0.9"
+VERSION="0.0.10"
 
 SCRIPT=${0##*/}
 # DIR=${0%/*}
@@ -45,10 +57,12 @@ case $(hostname) in
     i7-desktop)
         BASEDIR="$HOME/HPR/InternetArchive"
         UPLOADS="$HOME/HPR/IA/uploads"
+        REPAIRS="$BASEDIR/repairs"
         ;;
     borg)
         BASEDIR="$HOME/IA"
         UPLOADS="/data/IA/uploads"
+        REPAIRS="$BASEDIR/repairs"
         ;;
     *)
         echo "Wrong host!"
@@ -185,7 +199,7 @@ _usage () {
     cat >$STDOUT <<-endusage
 ${SCRIPT} - version: ${VERSION}
 
-Usage: ./${SCRIPT} [-h] [-v] [-d {0|1}] [-D] [-l N] item
+Usage: ./${SCRIPT} [-h] [-v] [-d {0|1}] [-D] [-l N] [-X] item
 
 Attempts to repair an IA item where the upload has failed for some reason.
 
@@ -203,6 +217,12 @@ Options:
                         during one run of the script. The range is 1 to
                         $DEFLIMIT. This can be helpful when there are upload
                         problems.
+  -X                    Run in "extended" mode. In this mode the directory
+                        holding files to be added to the IA is '~/IA/repairs'
+                        and the files have most likely come from the HPR
+                        backup disk and aren't on the IA due some error. We
+                        want to use the capabilities of ${SCRIPT} to repair
+                        things and deal with the IA upload problems.
 
 Arguments:
     item                The item in the form 'hpr1234'
@@ -229,7 +249,7 @@ DEFLIMIT=20
 #
 # Process options
 #
-while getopts :d:Dhl:v opt
+while getopts :d:Dhl:vX opt
 do
     case "${opt}" in
         D) DEBUG=1;;
@@ -237,6 +257,7 @@ do
         h) _usage 0;;
         l) LIMIT=$OPTARG;;
         v) VERBOSE=1;;
+        X) EXTENDED=1;;
         *) echo "** Unknown option"
            _usage 1;;
     esac
@@ -264,6 +285,8 @@ if [[ $LIMIT -lt 1 || $LIMIT -gt $DEFLIMIT ]]; then
     _usage 1
 fi
 
+EXTENDED=${EXTENDED:-0}
+
 #
 # Should have one argument
 #
@@ -295,6 +318,22 @@ if ! ia metadata "$item" --exists > /dev/null 2>&1; then
     exit 1
 fi
 
+#
+# The -X (EXTENDED) mode is for when we have to upload files that have
+# mysteriously vanished from the IA. The directories here are equivalent to
+# those used by 'repair_assets'. There is a top-level directory the represents
+# the IA item, and below that a hierarchy defining placement under the item.
+# There is a 'repairs' directory per host in case we need to preair IA stuff
+# from elsewhere.
+#
+if [[ $EXTENDED -eq 1 ]]; then
+    coloured 'cyan' "Using 'Extended' mode"
+    if [[ ! -e $REPAIRS ]]; then
+        mkdir -p "$REPAIRS"
+    fi
+    UPLOADS="$REPAIRS/$item"
+fi
+
 #
 # Declarations
 #