paulj/hpr-tools
1
0
Fork 0
forked from HPR/hpr-tools
Code Pull Requests Activity

Updates for show "repair" processing

Browse Source 
InternetArchive/future_upload: Added logging and debugging

InternetArchive/ia_db.sql: Added new tables

InternetArchive/recover_transcripts: New script to run on 'borg' and
    copy missing files from the backup disk to the IA

InternetArchive/repair_assets: More comments, including one about a bug in the design.

InternetArchive/repair_item: Fix relating to octal numbers (if there are
    leading zeroes in a number). '_DEBUG' is now in the function
    library. Added comments to explain obscure stuff.

InternetArchive/snapshot_metadata: New Bash script (to run on my
    desktop) which collects metadata for a show and stores in in the
    '~/HPR/IA/assets' directory. Runs 'view_derivatives' on it to find
    derivative files for deletion.

InternetArchive/tidy_uploaded: Moves files and directories containing
    uploaded files into a holding area for later backup. Added
    debugging, logging and a 'force' mode.

InternetArchive/upload_manager: Manages 'ia.db' (on my workstation).
    Needs many updates which have just started to be added.

InternetArchive/weekly_upload: Old script, now obsolete.
This commit is contained in:
Dave Morriss
2024-08-22 13:13:38 +01:00
parent dc0f29e957
commit 19030fee71
9 changed files with 994 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
InternetArchive/repair_item
View File

@@ -6,19 +6,24 @@
# USAGE: ./repair_item [-h] [-v] [-d {0|1}] [-D] [-l N] [-X] itemname
#
# DESCRIPTION: Repairs an IA "item" (HPR show) if something has failed during
# the upload.
# the upload (and when recovering deleted files from the
# changeover to the HPR static site).
#
# The most common failures are caused by the file upload
# processes timing out and being aborted (by the 'ia' tool which
# performs the item creation and the uploads). This failure
# means that a show being processed on 'borg' does not get all
# of the components loaded to the IA.
# of the components loaded to the IA. This happens during the
# sequence of running the 'make_metadata' Perl script which
# generates a CSV file of show data, followed by 'ia metadata
# --spreadsheet=<CSV file>'. Failures in the second part cause
# it to be aborted
#
# This script looks at the files belonging to the show (stored
# temporarily on 'borg') and determines which have not been
# uploaded, then takes steps to perform the uploads.
#
# Version 0.0.10 onwards has the capability to repair an IA item
# Version 0.0.11 onwards has the capability to repair an IA item
# from the HPR backup disk. This seems to be necessary because
# the transcripts were not carried over (although we are
# adding them to the IA for new shows now, older ones were never
@@ -30,20 +35,24 @@
# source file is in the upper one. This emulates the placement
# on the IA itself.
#
# This script can be called directly to recover a new show which
# failed during creation/upload, or by 'recover_transcripts'
# which is repairing shows with missing assets.
#
# OPTIONS: ---
# REQUIREMENTS: ---
# BUGS: ---
# NOTES: ---
# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
# VERSION: 0.0.10
# VERSION: 0.0.11
# CREATED: 2020-01-05 22:42:46
# REVISION: 2024-07-12 14:39:38
# REVISION: 2024-07-20 17:06:10
#
#===============================================================================
#set -o nounset # Treat unset variables as an error
VERSION="0.0.10"
VERSION="0.0.11"
SCRIPT=${0##*/}
# DIR=${0%/*}
@@ -55,6 +64,7 @@ STDOUT="/dev/fd/2"
#
case $(hostname) in
i7-desktop)
# TODO: consider not allowing this to be run anywhere but on 'borg'
BASEDIR="$HOME/HPR/InternetArchive"
UPLOADS="$HOME/HPR/IA/uploads"
REPAIRS="$BASEDIR/repairs"
@@ -100,7 +110,7 @@ TMP1=$(mktemp) || { echo "$SCRIPT: creation of temporary file failed!"; exit 1;
trap 'cleanup_temp $TMP1' SIGHUP SIGINT SIGPIPE SIGTERM EXIT
# {{{ -- Functions -- Upload, exists_in, queued_tasks, _DEBUG, _usage
# {{{ -- Functions -- Upload, exists_in, queued_tasks, _usage
#=== FUNCTION ================================================================
# NAME: Upload
@@ -174,19 +184,6 @@ queued_tasks () {
return
}
#=== FUNCTION ================================================================
# NAME: _DEBUG
# DESCRIPTION: Writes a message if in DEBUG mode
# PARAMETERS: List of messages
# RETURNS: Nothing
#===============================================================================
_DEBUG () {
[ "$DEBUG" == 0 ] && return
for msg in "$@"; do
printf 'D> %s\n' "$msg"
done
}
#=== FUNCTION ================================================================
# NAME: _usage
# DESCRIPTION: Reports usage; always exits the script after doing so
@@ -297,10 +294,11 @@ fi
item="${1}"
#
# Ensure item spec is correctly formatted
# Ensure item spec is correctly formatted. Have to cater for leading zeroes
# being interpreted as octal.
#
if [[ $item =~ hpr([0-9]+) ]]; then
printf -v item 'hpr%04d' "${BASH_REMATCH[1]}"
printf -v item 'hpr%04d' "$((10#${BASH_REMATCH[1]}))"
else
coloured 'red' "Incorrect show specification: $item"
coloured 'yellow' "Use 'hpr9999' format"
@@ -310,7 +308,7 @@ _DEBUG "Parsed item: $item"
#
# It's possible that the show upload failed before anything was uploaded, even
# the metadata. It's never been seen, but it seems wise to cater for it.
# the metadata. It's rarely seen, but it seems wise to cater for it.
#
if ! ia metadata "$item" --exists > /dev/null 2>&1; then
coloured 'red' "This item is not apparently on the IA; can't continue"
@@ -323,7 +321,7 @@ fi
# mysteriously vanished from the IA. The directories here are equivalent to
# those used by 'repair_assets'. There is a top-level directory the represents
# the IA item, and below that a hierarchy defining placement under the item.
# There is a 'repairs' directory per host in case we need to preair IA stuff
# There is a 'repairs' directory per host in case we need to repair IA stuff
# from elsewhere.
#
if [[ $EXTENDED -eq 1 ]]; then