Updates for missing asset "repair"
InternetArchive/recover_transcripts: Bash script to be run on 'borg' which collects files missing on the IA ready for upload as part of the missing asset repair process. InternetArchive/repair_assets: Bash script to take assets from the IA (after they had been repaired on 'borg') and copy them to the HPR server for the notes to access. The local machine, where this was run, was used to store files being uploaded. The planned script to modify the notes to reflect the new file locations was never finished. Notes were edited with Vim using a few macros. InternetArchive/repair_item: Bash script which is best run on 'borg', which repairs an IA item by comparing the files on the IA with the files on 'borg' (or a local machine). These files are either in '/data/IA/uploads/' or in the temporary file hierarchy used by 'recover_transcripts' (which calls it). Used after a normal IA upload to check for and make good any missed file uploads (due to timeouts, etc). Also used during asset repairs, but that project is now finished. InternetArchive/snapshot_metadata: Bash script which collects detailed metadata from the IA in JSON format and saves it locally (run on a local PC). Older shows on the IA often contained derivative files which were identified by the script 'view_derivatives'. These files were never needed, they were IA artefacts, so can be deleted (see the script header for how). InternetArchive/view_derivatives: Perl script to interpret a file of JSON metadata from the IA for an HPR show in order to determine the parent-child hierarchy of files where there may be derivatives. We don't want IA-generated derivatives, but this process was hard to turn off in earlier times. Generates a hierarchical report and a list of unwanted derivatives (see 'snapshot_metadata' for more details of how this was used).
This commit is contained in:
@@ -5,23 +5,28 @@
|
||||
#
|
||||
# USAGE: ./snapshot_metadata episode_number
|
||||
#
|
||||
# DESCRIPTION: Collects metadata from the IA for a given show and stores it
|
||||
# in the cache.
|
||||
# DESCRIPTION: Collects JSON metadata from the IA for a given show and stores
|
||||
# it in the cache. Runs 'view_derivatives' on the JSON to
|
||||
# display the derivatives if any, and to save their names if
|
||||
# found, for deletion.
|
||||
# Deletion is performed thus (external to this script):
|
||||
#
|
||||
# cat assets/hpr$(./next_repair)/derived.lis | xargs ia delete hpr$(./next_repair) --no-backup
|
||||
#
|
||||
# OPTIONS: ---
|
||||
# REQUIREMENTS: ---
|
||||
# BUGS: ---
|
||||
# NOTES: ---
|
||||
# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
|
||||
# VERSION: 0.0.2
|
||||
# VERSION: 0.0.3
|
||||
# CREATED: 2024-08-16 20:36:51
|
||||
# REVISION: 2024-08-17 10:31:15
|
||||
# REVISION: 2024-10-02 17:40:13
|
||||
#
|
||||
#===============================================================================
|
||||
|
||||
set -o nounset # Treat unset variables as an error
|
||||
|
||||
VERSION="0.0.2"
|
||||
VERSION="0.0.3"
|
||||
|
||||
SCRIPT=${0##*/}
|
||||
# DIR=${0%/*}
|
||||
@@ -126,17 +131,19 @@ fi
|
||||
show="${1,,}"
|
||||
|
||||
#
|
||||
# Ensure show id is correctly formatted. We want it to be 'hpr1234'
|
||||
# Ensure show id is correctly formatted. We want it to be 'hpr1234' but we
|
||||
# allow the 'hpr' bit to be omitted, as well as any leading zeroes. We need to
|
||||
# handle the weirdness of "leading zero means octal" though, but we always
|
||||
# store it as 'hpr1234' once processed.
|
||||
#
|
||||
if [[ $show =~ (hpr)?([0-9]+) ]]; then
|
||||
printf -v show 'hpr%04d' "${BASH_REMATCH[2]}"
|
||||
printf -v show 'hpr%04d' "$((10#${BASH_REMATCH[2]}))"
|
||||
else
|
||||
coloured 'red' "Incorrect show specification: $show"
|
||||
coloured 'yellow' "Use 'hpr9999' or '9999' format"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
#-------------------------------------------------------------------------------
|
||||
# Setting up paths
|
||||
#-------------------------------------------------------------------------------
|
||||
|
Reference in New Issue
Block a user