Commit Graph

9 Commits

Author SHA1 Message Date
Dave Morriss
0ccbb6419a Adding new 'view_derivatives'
InternetArchive/view_derivatives: New Perl script which Reads JSON
    metadata from the IA and builds tree-like structures linking
    original and derived files on the IA. It reports these trees and
    saves a subset of derived files in an output file to be used for
    deletion. In general we do not want derivatives, we make them
    ourselves. Older software had no reliable way to prevent them.
2024-08-22 13:25:22 +01:00
Dave Morriss
19030fee71 Updates for show "repair" processing
InternetArchive/future_upload: Added logging and debugging

InternetArchive/ia_db.sql: Added new tables

InternetArchive/recover_transcripts: New script to run on 'borg' and
    copy missing files from the backup disk to the IA

InternetArchive/repair_assets: More comments, including one about a bug in the design.

InternetArchive/repair_item: Fix relating to octal numbers (if there are
    leading zeroes in a number). '_DEBUG' is now in the function
    library. Added comments to explain obscure stuff.

InternetArchive/snapshot_metadata: New Bash script (to run on my
    desktop) which collects metadata for a show and stores in in the
    '~/HPR/IA/assets' directory. Runs 'view_derivatives' on it to find
    derivative files for deletion.

InternetArchive/tidy_uploaded: Moves files and directories containing
    uploaded files into a holding area for later backup. Added
    debugging, logging and a 'force' mode.

InternetArchive/upload_manager: Manages 'ia.db' (on my workstation).
    Needs many updates which have just started to be added.

InternetArchive/weekly_upload: Old script, now obsolete.
2024-08-22 13:13:38 +01:00
Dave Morriss
dc0f29e957 Updates since 2024-06-15
Database/query2tt2: comment and documentation updates; use of Perl's
    try/catch.

InternetArchive/.make_metadata.cfg: added comments for readability

InternetArchive/make_metadata: bug fix needed now that all shows on the HPR server have
    a directory with assets under it.

InternetArchive/repair_assets: new Bash script in development. Collects
    assets from the IA and uploads them to a new directory on the HPR
    server. Will run 'fix_asset_links' (to repair asset links for their
    new directories) once it is ready.

InternetArchive/repair_item: Bash script which was originally written to
    run on 'borg' and upload files to a new IA item when the uploads
    timed out. Now enhanced to upload missing files recovered from the
    HPR backup disk, such as transcripts.
2024-07-16 21:39:28 +01:00
Dave Morriss
9203dc26e0 Added InternetArchive/function_lib.sh
InternetArchive/function_lib.sh: new file; subset of
    '~/bin/function_lib.sh' which is referred to in a number of scripts.
    It contains relevant functions such as 'yes_no' and 'define_colours'.
2024-06-15 18:47:24 +01:00
Dave Morriss
6805cd662b Updated 'repair_item'
InternetArchive/repair_item: originally planned in 2020 as a Bash script
    to find missing files in shows and then add them, it was not turned
    into the current form until May 2024. Now, with the heavy loading of
    the IA servers, normal uploads are timing out and being aborted.
    This script is more "determined" to upload files and usually
    successfully "repairs" shows that need it.
2024-06-15 17:14:22 +01:00
Dave Morriss
7a4290fcdd Attempting to generate fix_tags.bin 2024-06-14 23:13:45 +01:00
Dave Morriss
50edeccc88 Updates from previous repo
FAQ/FAQ.mkd, FAQ/Makefile: this version of the FAQ is now out of date
    and probably should be deleted.

InternetArchive/repair_item: script to upload missing shows after tie
    out errors during the normal upload; still under development.

InternetArchive/update_state: script to update show state in the
    'reservations' table in the database. Uses the CMS interface.

Link_Checker/scan_links: under development. Not currently usable.

Miscellaneous/fix_tags: audio metadata manipulation script. Recently
    added to this repo for convenience. Updates for 'experimental::try',
    the official Perl try/catch.

PostgreSQL_Database/add_hosts_to_show, PostgreSQL_Database/hpr_schema_2.pgsql,
    PostgreSQL_Database/nuke_n_pave.sh: an old experimental Pg database
    to take over from the previous MySQL version (from before 2023).
    Kept for reference; never implemented.
2024-06-14 16:00:04 +01:00
Dave Morriss
38abbcdd39 Moved project directories and files to an empty local repo 2024-06-04 16:35:44 +01:00
2d2b937a9b Initial commit 2024-06-04 15:06:29 +00:00