diff --git a/FAQ/FAQ.html b/FAQ/FAQ.html index 356ac17..e2487f8 100644 --- a/FAQ/FAQ.html +++ b/FAQ/FAQ.html @@ -1,66 +1,13 @@ - - - - - - - - Hacker Public Radio FAQ (draft) - - - - - - -
-
-

Hacker Public Radio FAQ (draft)

-

A collection of questions and answers with links to the HPR site

-

HPR Contributors

-
-
- -
-
-
-

Table of Contents

- -
-

Hacker Public Radio FAQ

+

Hacker Public Radio FAQ 2021-04-01

0.1 What is Hacker Public Radio?

    -

  • Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases shows every weekday Monday through Friday.

    • -

  • What differentiates HPR from other podcasts is that the shows are crowd sourced from the community — fellow listeners like you. There is no restriction on how long shows can be, nor on the topic you can cover as long as they are not spam and "are of interest to Hackers".

    -

    If you want to see what topics have been covered so far just have a look at our Archive. We also allow for a series of shows so that hosts can go into more detail on a topic.

    • +

  • Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases shows (episodes) every weekday Monday through Friday.

    • +

  • What differentiates HPR from other podcasts is that the shows are crowd sourced from the community — fellow listeners like you. There is no restriction on how long shows can be, nor on the topic you can cover, as long as they are not spam and "are of interest to Hackers".

    +

    If you want to see what topics have been covered so far just have a look at our Archive. We also allow for shows to be grouped into series so that hosts can go into more detail on a topic.

  • Look at the About HPR page for a full description of The HPR Community, Free Culture, Governance and our Patrons.

    • -

  • A recent question on the mailing list asks if HPR is a Content Publication Network like YouTube, or is it a Podcast that random people contribute to? Or to put it another way, is the host addressing the Internet in general or the HPR Audience?

    +

  • A question on the mailing list in 2020 asked if HPR is a Content Publication Network like YouTube, or is it a Podcast that random people contribute to? Or to put it another way, is the host addressing the Internet in general or the HPR Audience?

      -
    • The consensus of opinion is that HPR is a Podcast contributed to by the community.
      • +
    • The consensus of opinion is that HPR is a Podcast contributed to by the community, not a Content Publication Network.

0.2 What is a show?

@@ -79,7 +26,7 @@

  • Each show is expected to have certain mandatory attributes listed below.

  • HPR shows are usually expected to be informative or educational, and as such some sort of accompanying written material is highly desirable. This might consist of brief notes, links to relevant web sites, or possibly longer notes and examples. Photographs, diagrams and example files are also welcome if the host feels it helps to get the message across.

  • Shows are released under a Creative Commons Attribution-ShareAlike 3.0 Unported license. See Stuff you need to know for much more detail about this and related issues.

    • -

  • As a contributor you can choose when your show will be released. We only release one show a day, during week days, but if a slot is free you can claim it. Go to the calendar page to do this. You need to have your show ready to upload before you do this though.

    • +

  • As a contributor you can choose when your show will be released. We only release one show a day, during week days, but if a slot is free you can claim it. Go to the calendar page to do this. You need to have your show ready to upload before you reserve a slot though.

    • 0.3 Can I submit a show made for another podcast?

      @@ -105,30 +52,37 @@
    • Audio
    -

    0.6 Is it OK if I don't include notes with my show?

    +

    0.6 What are tags?

      -
    • No, please don't do that. It is important to provide good show notes for reasons described on the Picking a slot for your show page: +

    • We are trying to maintain a collection of tags to allow people to find shows on the same (or similar) topic.

      • +

    • Tags are simple words or short phrases describing aspects of a show such as what topics are covered in it. To get hints about what tags have already been used you can view all the current tags in alphabetical order on this page: Tag summary

      • +

    • Some examples are: android, creative commons and gaming.

      • +

    • Are there any tags that are best avoided?

      • +
    +

    0.7 Is it OK if I don't include notes with my show?

      -
    • They provide the listeners with more information on the topic
      • -
    • They ensure your show gets posted on time
      • +
    • Please don't do that. It is important to provide good show notes for reasons described on the Picking a slot for your show page: +
        +
      • They provide the listeners with more information on the topic your show is covering
        • +
      • They ensure your show gets posted on time (the volunteers don't have to make notes for you)
      • They encourage people to download your show
      • They allow your show to be indexed by search engines
      • They make your show accessible to the deaf and hard of hearing
    -

    0.7 How do I include photographs or other files with my show?

    +

    0.8 How do I include photographs or other files with my show?

    • TBA
    -

    0.8 What is and is not edited in a show after submission?

    +

    0.9 What is and is not edited in a show after submission?

    • We don't listen to the audio before posting a show to the website, so we don't change the content.

      • -

    • Unless you have done so and told us, we add an introduction to the audio announcing what it is. Similarly we add an outro. This is not editing since we don't change the content.

      • +

    • Unless you have done so and told us you have, we add an introduction to the audio announcing what it is. Similarly we add an outro. This is not editing since we don't change the content.

    • We may make changes to the show title, summary or tags. Usually these are minor changes such as spelling corrections. The intention is not any sort of censorship, merely to make these items more understandable or easier to index.

    • Depending on the form the show notes take we may modify these.

      • If what is submitted is plain text we convert it to Markdown in order to generate HTML.
        • -
      • If the notes are one of the permitted markup formats (mostly Markdown) we may adjust this to ensure that valid HTML is generated.
        • +
      • If the notes are one of the permitted markup formats (mostly Markdown flavours) we may adjust this to ensure that valid HTML is generated.
      • If the notes are HTML we run them through an HTML checker and correct any mistakes it highlights.
      • We may correct spelling, grammar and punctuation if this clarifies anything which might otherwise be unclear.
      • If despite the request for show notes of some kind, none are provided, we may add notes which help to clarify the content of the show. This may be done after the show is released (when we have had a chance to listen to it).
        • @@ -137,33 +91,33 @@

      • We may make changes to a show's metadata if the host requests it (see the item on fixing errors after upload).

      • If a show contains links to resources which disappear after a period of time, we may at some future time change the notes to refer to any Wayback Machine copy that can be found.

      -

      0.9 What are some common errors in show notes?

      +

      0.10 What are some common errors in show notes?

      It depends on the note format to some extent:

      -

      0.9.1 HTML notes

      +

      0.10.1 HTML notes

      • It's very easy to hand-write bad HTML. That's why we use an HTML checker on it, and fix the errors it reports.

      • A common fault is to write <code><pre> to define a pre-formatted block of text using a fixed-width font. However it's illegal because the <pre> tag can't be enclosed in other tags like <code>. Use <pre><code> instead.

      -

      0.9.2 Markdown notes

      +

      0.10.2 Markdown notes

      • This is quite a forgiving format. Some people embed HTML in the Markdown, and the HTML used suffers from some of the issues discussed in this document. These are much harder to spot. The HTML checker can't really help unless it's run on the output from Pandoc.
      -

      0.9.3 General errors

      +

      0.10.3 General errors

      • Spelling is often an issue in notes. As the notes are processed they are edited in the Vim editor with the spell check option turned on. In most cases spelling problems flagged by Vim are corrected.

      • Grammar and punctuation can also be issues. We do not have very effective grammar checking tools in the note preparation workflow. Modifications may be made to split up extra-long (often comma-spliced) sentences, rationalise the use of capital letters and similar.

      -

      0.10 If I notice an error in my show's details how can it be fixed?

      +

      0.11 If I notice an error in my show's details how can it be fixed?

      • The HPR administrators can make changes to show titles, summaries, notes and so forth. Ideally send an email to admin at hackerpublicradio.org explaining what the problem is and we'll fix it for you. We'll also ensure that the changes are propagated to the relevant page on archive.org.

      • Don't be tempted to send in your corrections as a comment. Comments are not propagated to archive.org, so people referring to that copy will not see the changes.

      -

      0.11 Does hobbypublicradio.com have anything to do with HPR?

      +

      0.12 Does hobbypublicradio.com have anything to do with HPR?

      -

      0.12 How can I talk to other HPR hosts and listeners?

      +

      0.13 How can I talk to other HPR hosts and listeners?

      • You have a number of choices:
          @@ -174,8 +128,3 @@
        • An HPR tradition exists where we record a 26-hour show at the turn of the year celebrating the arrival of the New Year around the world. We use Mumble to record the show and anyone can connect to the Mumble server and speak to other HPR listeners and hosts. The recording is used to make multiple shows which are released when they are ready.
      -
    -
    -
    - - diff --git a/FAQ/FAQ.mkd b/FAQ/FAQ.mkd index 0544549..e7a3fc9 100644 --- a/FAQ/FAQ.mkd +++ b/FAQ/FAQ.mkd @@ -1,9 +1,11 @@ -[%# FAQ.mkd 2020-09-26 14:26:49 version: 0.0.3 -%] +[%# FAQ.mkd <2021-04-01 15:17:14> version: 0.0.5 -%] +[% USE date -%] --- -title: Hacker Public Radio FAQ (draft) +title: Hacker Public Radio FAQ (draft) [% date.format(date.now,"%F") %] subtitle: A collection of questions and answers with links to the HPR site author: HPR Contributors ... +[%# {{{ Links %] [% about = "http://hackerpublicradio.org/about.php" -%] [% reqslot = "http://hackerpublicradio.org/request_a_slot.php" -%] [% needtoknow = "http://hackerpublicradio.org/stuff_you_need_to_know.php" -%] @@ -15,32 +17,34 @@ author: HPR Contributors [% maillist = "http://hackerpublicradio.org/maillist" -%] [% freenode = "http://webchat.freenode.net/?channels=oggcastplanet" -%] [% podrec = "http://hackerpublicradio.org/series.php?id=75" -%] +[% tags = "http://hackerpublicradio.org/tags.php" -%] +[%# }}} %] -# Hacker Public Radio FAQ {.unnumbered} +# Hacker Public Radio FAQ [% date.format(date.now,"%F") %] {.unnumbered} ## What is Hacker Public Radio? - Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases - shows every weekday Monday through Friday. + shows (episodes) every weekday Monday through Friday. - What differentiates HPR from other podcasts is that the shows are crowd sourced from the community — fellow listeners like you. There is no - restriction on how long shows can be, nor on the topic you can cover as long + restriction on how long shows can be, nor on the topic you can cover, as long as they are not spam and "are of interest to Hackers". If you want to see what topics have been covered so far just have a look at - our [Archive]([% index %]). We also allow for a [series]([% series %]) of - shows so that hosts can go into more detail on a topic. + our [Archive]([% index %]). We also allow for shows to be grouped into + [series]([% series %]) so that hosts can go into more detail on a topic. - Look at the [*About HPR*]([% about %]) page for a full description of *The HPR Community*, *Free Culture*, *Governance* and our *Patrons*. -- A recent question on the mailing list asks if HPR is a Content Publication +- A question on the mailing list in 2020 asked if HPR is a Content Publication Network like YouTube, or is it a Podcast that random people contribute to? Or to put it another way, is the host addressing the Internet in general or the HPR Audience? - The consensus of opinion is that HPR is a Podcast contributed to by the - community. + community, not a Content Publication Network. ## What is a show? @@ -81,7 +85,7 @@ author: HPR Contributors - As a contributor you can choose when your show will be released. We only release one show a day, during week days, but if a slot is free you can claim it. Go to the [*calendar*]([% calendar %]) page to do this. You need to have - your show ready to upload before you do this though. + your show ready to upload before you reserve a slot though. ## Can I submit a show made for another podcast? @@ -115,13 +119,30 @@ author: HPR Contributors - Tags - Audio +## What are tags? + +- We are trying to maintain a collection of tags to allow people to find shows + on the same (or similar) topic. + +- Tags are simple words or short phrases describing aspects of a show such as + what topics are covered in it. To get hints about what tags have already + been used you can view all the current tags in alphabetical order on this + page: [Tag summary]([% tags %]) + +- Some examples are: android, creative commons and + gaming. + +- Are there any tags that are best avoided? + ## Is it OK if I don't include notes with my show? -- No, please don't do that. It is important to provide good show notes for +- **Please don't do that**. It is important to provide good show notes for reasons described on the [*Picking a slot for your show*]([% reqslot %]) page: - - They provide the listeners with more information on the topic - - They ensure your show gets posted on time + - They provide the listeners with more information on the topic your show + is covering + - They ensure your show gets posted on time (the volunteers don't have to + make notes for you) - They encourage people to download your show - They allow your show to be indexed by search engines - They make your show accessible to the deaf and hard of hearing @@ -135,8 +156,8 @@ author: HPR Contributors - We don't listen to the audio before posting a show to the website, so we don't change the content. -- Unless you have done so and told us, we add an introduction to the audio - announcing what it is. Similarly we add an *outro*. This is not +- Unless you have done so and told us you have, we add an introduction to the + audio announcing what it is. Similarly we add an *outro*. This is not editing since we don't change the content. - We may make changes to the show title, summary or tags. Usually these are @@ -147,8 +168,8 @@ author: HPR Contributors - Depending on the form the show notes take we may modify these. - If what is submitted is plain text we convert it to Markdown in order to generate HTML. - - If the notes are one of the permitted markup formats (mostly Markdown) - we may adjust this to ensure that valid HTML is generated. + - If the notes are one of the permitted markup formats (mostly Markdown + flavours) we may adjust this to ensure that valid HTML is generated. - If the notes are HTML we run them through an HTML checker and correct any mistakes it highlights. - We may correct spelling, grammar and punctuation if this clarifies diff --git a/FAQ/Makefile b/FAQ/Makefile index 3b0f8f7..c17c3f8 100644 --- a/FAQ/Makefile +++ b/FAQ/Makefile @@ -1,31 +1,52 @@ # ============================================================================== -# FAQ Makefile 2021-01-30 11:00:47 +# FAQ Makefile 2021-04-01 09:52:16 # ============================================================================== # # Simple Makefile to rebuild the components in this file # -all : markdown + +MDFILE = FAQ.mkd +HTMLFILE = FAQ.html +DEVFILE = FAQ_dev.html +TPLFILE = FAQ.tpl +PHPFILE = FAQ.php + +all : markdown php # -# Find all *.mkd files in the current directory and turn them into a list of -# *.html files as a rule target +# Rules for converting the Markdown into two files, one for stitching into +# the PHP framework via FAQ.tpl and the other just a stand-alone HTML file for +# development. # -markdown: $(addsuffix .html,$(basename $(wildcard *.mkd))) +markdown: $(HTMLFILE) $(DEVFILE) + +$(HTMLFILE): $(MDFILE) + tpage $< | pandoc -f markdown-smart -t html5 \ + --number-sections --table-of-contents --toc-depth=4 -o $@ + +$(DEVFILE): $(MDFILE) + tpage $< | pandoc -f markdown-smart -t html5 --standalone \ + --template=hpr.html5 -c http://hackerpublicradio.org/css/hpr.css \ + --number-sections --table-of-contents --toc-depth=4 -o $@ # -# Rule to get from a Markdown (*.mkd) file to an HTML equivalent allowing for -# Template Toolkit stuff within the file +# When we generate the file FAQ.html (HTML fragment) we use tpage to make +# a complete standalone PHP page called FAQ.php # -%.html: %.mkd; tpage $< | pandoc -f markdown-smart -t html5 --standalone \ - --template=hpr.html5 -c http://hackerpublicradio.org/css/hpr.css \ - --number-sections --table-of-contents --toc-depth=4 -o $@ +.PHONY: php upload + +php: $(PHPFILE) + +$(PHPFILE): $(HTMLFILE) + tpage $(TPLFILE) > $(PHPFILE) # -# Upload the FAQ HTML to the server for development +# Upload the FAQ HTML+PHP to the server for development # -upload: - scp -P 22074 FAQ.html hpr@hackerpublicradio.org:www/ +upload: $(PHPFILE) + scp -P 22074 $(PHPFILE) hpr@hackerpublicradio.org:www/ +# ============================================================================== # # Use 'make sync' to copy updates to the visible project area # @@ -35,7 +56,7 @@ upload: sync: put fromdir = $(HOME)/HPR/FAQ/ -todir = $(HOME)/HPR/Projects/hpr-admin/FAQ/ +todir = $(HOME)/HPR/Projects/hpr-tools/FAQ/ filter = $(fromdir).rsync_export syncmsg = Updating git directory with updates from development version @@ -44,5 +65,5 @@ put: @echo "** $(syncmsg)" rsync -vaP --filter=". $(filter)" $(fromdir) $(todir) -.PHONY: upload sync put +.PHONY: sync put diff --git a/InternetArchive/ia.db b/InternetArchive/ia.db index 28fdfab..aa1d20e 100644 Binary files a/InternetArchive/ia.db and b/InternetArchive/ia.db differ diff --git a/InternetArchive/repair_item b/InternetArchive/repair_item index 3a9f89e..2debea4 100755 --- a/InternetArchive/repair_item +++ b/InternetArchive/repair_item @@ -23,15 +23,15 @@ # BUGS: --- # NOTES: --- # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com -# VERSION: 0.0.6 +# VERSION: 0.0.7 # CREATED: 2020-01-05 22:42:46 -# REVISION: 2024-05-10 12:39:52 +# REVISION: 2024-05-17 20:49:47 # #=============================================================================== #set -o nounset # Treat unset variables as an error -VERSION="0.0.6" +VERSION="0.0.7" SCRIPT=${0##*/} # DIR=${0%/*} @@ -190,8 +190,10 @@ Options: mode and the actions will be carried out. -D Run in debug mode where a lot more information is reported - -l N Control the number of shows that can be uploaded at - once. The range is 1 to $DEFLIMIT. + -l N Control the number of files that can be uploaded + during one run of the script. The range is 1 to + $DEFLIMIT. This can be helpful when there are upload + problems. Arguments: item The item in the form 'hpr1234' @@ -274,6 +276,16 @@ else fi _DEBUG "Parsed item: $item" +# +# It's possible that the show upload failed before anything was uploaded, even +# the metadata. It's never been seen, but it seems wise to cater for it. +# +if ! ia metadata "$item" --exists > /dev/null 2>&1; then + coloured 'red' "This item is not apparently on the IA; can't continue" + coloured 'yellow' "Try running the entire upload again from the start" + exit 1 +fi + # # Declarations # @@ -392,7 +404,7 @@ else else retries=0 - printf 'Uploading %s\n' "$file" + coloured 'blue' "Uploading $file" # # Run 'cmd'. If it succeeds then write to the log and loop for the @@ -414,9 +426,12 @@ else [ "$retries" -eq "$retry_threshold" ] && { ((failures++)) + [[ $VERBOSE -eq 1 ]] && \ + coloured 'blue' "Retry limit reached; abandoning this file" continue 2 } + [[ $VERBOSE -eq 1 ]] && coloured 'blue' "Pausing for $sleeptime and retrying" sleep $sleeptime done # until eval ... diff --git a/InternetArchive/update_state b/InternetArchive/update_state index 7b40ae3..3ae90af 100755 --- a/InternetArchive/update_state +++ b/InternetArchive/update_state @@ -25,9 +25,9 @@ # BUGS: --- # NOTES: --- # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com -# VERSION: 0.0.9 +# VERSION: 0.0.10 # CREATED: 2022-04-19 12:50:52 -# REVISION: 2023-10-21 22:51:42 +# REVISION: 2024-06-01 14:19:20 # #=============================================================================== @@ -37,7 +37,7 @@ SCRIPT=${0##*/} # DIR=${0%/*} # shellcheck disable=SC2034 -VERSION="0.0.9" +VERSION="0.0.10" STDOUT="/dev/fd/2" @@ -54,6 +54,8 @@ source "$LIB" # define_colours +# {{{ ---- Functions: ---- _usage _DEBUG + #=== FUNCTION ================================================================ # NAME: _usage # DESCRIPTION: Report usage @@ -73,13 +75,17 @@ processed. Options: -h Print this help - -D Enable DEBUG mode where a lot of information about the workins + -D Enable DEBUG mode where a lot of information about the working of the script is displayed - -d Dry-run mode. Reports what it will do but doesn't do it + -d Dry-run mode. Reports what it would do but doesn't do it -F Force the update(s) without checking the state of the show on the IA -l N Limit the number of shows processed to N -m Monochrome mode - no colours + -R Normally, if a show is not in the IA, the script retries + waiting for it to be uploaded (assuming it's being worked on + by the IA servers). Including -R limits the retries to one + which is useful when uploading multiple shows one at a time. Examples ./${SCRIPT} -h @@ -87,7 +93,10 @@ Examples ./${SCRIPT} -d ./${SCRIPT} -dm ./${SCRIPT} -Dd + ./${SCRIPT} -F ./${SCRIPT} -l1 + ./${SCRIPT} -m + ./${SCRIPT} -R ./${SCRIPT} endusage @@ -107,6 +116,8 @@ _DEBUG () { done } +# }}} + #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # @@ -114,10 +125,10 @@ _DEBUG () { # case $HOSTNAME in hprvps|marvin|borg) -# UPLOADS="/data/IA/uploads" +# UPLOADS="/data/IA/uploads" BASEDIR="$HOME/IA" ;; i7-desktop) -# UPLOADS="$HOME/HPR/IA/uploads" +# UPLOADS="$HOME/HPR/IA/uploads" BASEDIR="$HOME/HPR/IA" ;; *) echo "Wrong host!"; exit 1 ;; @@ -149,16 +160,17 @@ RETRIES=3 # # Option defaults # -COLOUR=1 # use colours by default -DRYRUN=0 # live mode by default +COLOUR=1 # use colours by default +DRYRUN=0 # live mode by default DEBUG=0 FORCE=0 +RETRYING=1 # retry if a show's not on the IA DEFLIMIT=20 # # Process options # -while getopts :hdDFl:m opt +while getopts :hdDFl:mR opt do case "${opt}" in h) _usage;; @@ -167,6 +179,7 @@ do F) FORCE=1;; l) LIMIT=$OPTARG;; m) COLOUR=0;; + R) RETRYING=0;; ?) echo "$SCRIPT: Invalid option; aborting"; exit 1;; esac done @@ -189,6 +202,10 @@ if [[ $FORCE -eq 1 ]]; then coloured 'yellow' "Forcing updates without checking the IA state" fi +if [[ $RETRYING -eq 0 ]]; then + coloured 'yellow' "Not retrying updates if the show is missing" +fi + # # Check the argument count after any options # @@ -220,7 +237,7 @@ _DEBUG "reservations = $reservations" # "queue" from the variable 'reservations' which contains lines returned from # querying the CMS status interface. # -count=0 +showcount=0 while read -r line; do if [[ $line =~ ^([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),.*$ ]]; then state="${BASH_REMATCH[5]}" @@ -232,7 +249,16 @@ while read -r line; do if [[ $state = 'MEDIA_TRANSCODED' ]]; then _DEBUG "show = $show, state = $state" - retry_count=$RETRIES + # + # If we're retrying (waiting for a show to be uploaded) then loop + # $RETRIES times, otherwise don't retry at all + # + if [[ $RETRYING -eq 1 ]]; then + retry_count=$RETRIES + else + retry_count=1 + fi + while [ $retry_count -gt 0 ]; do # # Look for the show on the IA. If not found we sleep 30 @@ -240,7 +266,8 @@ while read -r line; do # times, controlled by $RETRIES, then we give up this show. If # there are more shows then we keep going. # - if [ $FORCE -eq 1 ] || ia list "hpr$show" > /dev/null 2>&1; then + if [ $FORCE -eq 1 ] || ia metadata "hpr$show" --exists > /dev/null 2>&1; then +# if [ $FORCE -eq 1 ] || ia list "hpr$show" > /dev/null 2>&1; then command="${QUERY1}?ep_num=${show}&status=UPLOADED_TO_IA" command_bak="${QUERY1_BAK}?ep_num=${show}&status=UPLOADED_TO_IA" @@ -282,19 +309,23 @@ while read -r line; do ((retry_count--)) done - if [[ $retry_count -eq 0 ]]; then + # + # Are all retries done, and are we retrying anyway? + # + if [[ $retry_count -eq 0 && $RETRYING -eq 1 ]]; then coloured 'red' "Failed to update show $show; retry count reached" coloured 'yellow' "The command 'ia list hpr$show' repeatedly returned \"failure\"" coloured 'yellow' "Database updates not done" - coloured 'yellow' "Try again later with './update_state'" + coloured 'yellow' "Try again later with './${SCRIPT}'" fi # # Stop the loop if we have reached the limiting number # - ((count++)) - [[ $count -eq $LIMIT ]] && { + ((showcount++)) + [[ $showcount -eq $LIMIT ]] && { echo "Upload limit ($LIMIT) reached" + ((--showcount)) break } @@ -303,9 +334,9 @@ while read -r line; do done <<< "$reservations" if [[ $DRYRUN -eq 0 ]]; then - echo "Number of shows processed successfully: $count" + echo "Number of shows processed successfully: $showcount" fi exit -# vim: syntax=sh:ts=8:sw=4:ai:et:tw=78:fo=tcrqn21 +# vim: syntax=sh:ts=8:sw=4:ai:et:tw=78:fo=tcrqn21:fdm=marker diff --git a/Link_Checker/scan_links b/Link_Checker/scan_links index 8f08f1c..364aa9b 100755 --- a/Link_Checker/scan_links +++ b/Link_Checker/scan_links @@ -3,7 +3,7 @@ # # FILE: scan_links # -# USAGE: ./scan_links +# USAGE: ./scan_links [-help] [-[no]verbose] [-config=FILE] # # DESCRIPTION: Scan the notes in the database for links. Test each link to # see if it's available. Keep a record of the date, show, link @@ -18,9 +18,9 @@ # BUGS: --- # NOTES: --- # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com -# VERSION: 0.0.1 +# VERSION: 0.0.2 # CREATED: 2017-04-02 14:09:02 -# REVISION: 2017-04-02 14:51:19 +# REVISION: 2022-06-02 23:18:13 # #=============================================================================== @@ -28,12 +28,32 @@ use 5.010; use strict; use warnings; use utf8; +use experimental 'smartmatch'; +use Carp; +use Getopt::Long; +use Pod::Usage; + +use Config::General; +use File::Slurper qw{ read_text read_lines }; +use Try::Tiny; + +use HTML::TreeBuilder 5 -weak; +use HTML::Entities; +use List::Util qw{ min max }; +use List::MoreUtils qw{ any }; +use LWP::Simple; + +use DBI; +use SQL::Abstract; +use SQL::Abstract::Plugin::InsertMulti; + +use Data::Dumper; # # Version number (manually incremented) # -our $VERSION = '0.0.1'; +our $VERSION = '0.0.2'; # # Script and directory names @@ -48,8 +68,12 @@ $DIR = '.' unless $DIR; # # Constants and other declarations # -my $basedir = "$ENV{HOME}"; +my $basedir = "$ENV{HOME}/HPR/Link_Checker"; +my $configfile = "$basedir/.$PROG.cfg"; +my $db1configfile = "$basedir/.hpr_db.cfg"; +my $database2 = "$basedir/ia.db"; +my ( $dbh1, $dbh2, $sql1, $sth1, $h1 ); # # Enable Unicode mode @@ -57,8 +81,538 @@ my $basedir = "$ENV{HOME}"; binmode STDOUT, ":encoding(UTF-8)"; binmode STDERR, ":encoding(UTF-8)"; +#------------------------------------------------------------------------------- +# Options and arguments +#------------------------------------------------------------------------------- +# +# Option defaults +# +my $DEFDEBUG = 0; +my $DEFFROM = 1; +my $DEFCOUNT = 10; + +# +# Process options +# +my %options; +Options( \%options ); + +# +# Default help is minimal +# +pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1 ) + if ( $options{'help'} ); + +# +# The -documentation or -man option shows the full POD documentation through +# a pager for convenience +# +pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1, -verbose => 2 ) + if ( $options{'documentation'} ); + +# +# Collect options +# +my $DEBUG = ( defined( $options{debug} ) ? $options{debug} : $DEFDEBUG ); + +my $db1cfgfile + = ( defined( $options{dbconfig} ) ? $options{dbconfig} : $db1configfile ); + +my $cfgfile + = ( defined( $options{config} ) ? $options{config} : $configfile ); + +my $dry_run = ( defined( $options{'dry-run'} ) ? $options{'dry-run'} : 0 ); +my $verbose = ( defined( $options{verbose} ) ? $options{verbose} : 0 ); +my $from = ( defined( $options{from} ) ? $options{from} : $DEFFROM ); +my $count = ( defined( $options{count} ) ? $options{count} : $DEFCOUNT ); + +$from = $DEFFROM if $from < 1; +my @episodes = ( ( $from .. $from + $count ) ); + +# +# Report on the options in debug mode +# +if ($DEBUG > 1) { + _debug(1,'$DEBUG = ' . $DEBUG); + _debug(1,'$dry-run = ' . $dry_run); + _debug(1,'$verbose = ' . $verbose); + _debug(1,'$db1cfgfile = ' . $db1cfgfile); + _debug(1,'$cfgfile = ' . $cfgfile); + _debug(1,'$from = ' . $from); + _debug(1,'$count = ' . $count); + _debug(1,'$#episodes = ' . $#episodes); +} + +# +# Sanity checks +# +die "Unable to find $cfgfile\n" unless ( -e $cfgfile ); +die "Unable to find $db1cfgfile\n" unless ( -e $db1cfgfile ); + +#------------------------------------------------------------------------------- +# Configuration file - load data +#------------------------------------------------------------------------------- +my $conf = Config::General->new( + -ConfigFile => $cfgfile, + -InterPolateVars => 1, + -ExtendedAccess => 1, +); +my %config = $conf->getall(); + +#------------------------------------------------------------------------------- +# Connect to the database +#------------------------------------------------------------------------------- +my $db1conf = Config::General->new( + -ConfigFile => $db1cfgfile, + -InterPolateVars => 1, + -ExtendedAccess => 1, +); +my %db1cfg = $db1conf->getall(); + +#------------------------------------------------------------------------------- +# Database configuration file - load data +#------------------------------------------------------------------------------- +my $dbhost = $db1cfg{database}->{host} // '127.0.0.1'; +my $dbport = $db1cfg{database}->{port} // 3306; +my $dbname = $db1cfg{database}->{name}; +my $dbuser = $db1cfg{database}->{user}; +my $dbpwd = $db1cfg{database}->{password}; +$dbh1 = DBI->connect( "dbi:mysql:host=$dbhost;port=$dbport;database=$dbname", + $dbuser, $dbpwd, { AutoCommit => 1, RaiseError => 1 } ) + or die $DBI::errstr; + +# +# Enable client-side UTF8 +# +$dbh1->{mysql_enable_utf8} = 1; + +# +# Set the local timezone to UTC for this connection +# +$dbh1->do("set time_zone = '+00:00'") or carp $dbh1->errstr; + +emit( $verbose >= 2, "Opened MySQL database\n" ); + +#------------------------------------------------------------------------------- +# Connect to the SQLite database +#------------------------------------------------------------------------------- +$dbh2 = DBI->connect( "dbi:SQLite:dbname=$database2", "", "" ) + or die $DBI::errstr; + +$dbh2->do("PRAGMA foreign_keys = ON") or die $DBI::errstr; + +emit( $verbose >= 2, "Opened SQLite database\n" ); + +# +# The main MySQL query +# +$sql1 = q{ + SELECT * FROM eps WHERE id BETWEEN ? AND ? +}; + +$sth1 = $dbh1->prepare($sql1); + +$sth1->execute( $from, $from + $count ); +if ( $dbh1->err ) { + die $dbh1->errstr; +} + +while ( $h1 = $sth1->fetchrow_hashref ) { + printf "%04d %s %s\n", $h1->{id},$h1->{date},$h1->{title}; +} exit; +#emit( $verbose >= 2, "Configuration file: ", $cfgfile, "\n" ); +#emit( $verbose >= 2, "DB configuration file: ", $db1cfgfile, "\n" ); + +#=== FUNCTION ================================================================ +# NAME: find_external_links +# PURPOSE: Parses the HTML in a string for links so that attached assets +# on the HPR site can also be parsed and to collect external +# links for testing. +# PARAMETERS: $episode episode number we're dealing with +# $html string containing HTML +# $rlinks hashref to receive the links found +# RETURNS: Number of links found +# DESCRIPTION: Given HTML from the main notes or a subsidiary file the +# function parses this looking for links in 'a' or 'img' tags. +# Links are standardised, making them absolute if relative and +# removing any 'fragment'. The links need to be to HTML files on +# the HPR website to be of use in recursing to subsidiary +# levels. Otherwise they need to be external links that we will +# test. +# Having found a local link the filename part is extracted. If +# it follows the format 'hpr9999' then it's checked to see if +# it's for the current show. If not it's ignored. If the +# filename ends with a '/' then it's assumed it's shorthand for +# 'index.html' so this name is appended. If the local filename +# ends with '.html' then we need to parse it in turn, so we get +# the contents of the link and recurse to parse it. +# Then, if external, the link and filename are stashed in the +# hash referenced by $rlinks. We return the number of external +# links found in the pass through the HTML. +# THROWS: No exceptions +# COMMENTS: Based on 'find_links' in 'upload_manager'. +# SEE ALSO: N/A +#=============================================================================== +sub find_external_links { + my ( $episode, $html, $rlinks ) = @_; + + my ($tree, $epstr, $linkre, $re2, $filepath, + $uri, $slink, $linkcount, $content + ); + + _debug( $DEBUG >= 3, "find_external_links enter" ); + + # + # Create a tree object + # + $tree = HTML::TreeBuilder->new; + $tree->ignore_unknown(0); + $tree->no_expand_entities(1); + $tree->p_strict(1); + $tree->store_comments(1); + $tree->warn(1); + + $tree->parse_content($html) + or die "HTML::TreeBuilder failed to parse notes: $!\n"; + + my $baseURL = "http://hackerpublicradio.org"; + + $epstr = sprintf( "hpr%04d", $episode ); + $linkre = qr{ + ^https?:// + (?:www.)? + (?:hacker|hobby)publicradio.org/ + (.+)$ + }x; + + # + # Counting new links found and stashed + # + $linkcount = 0; + + # + # Scan for links + # + for ( @{ $tree->extract_links( 'a', 'img' ) } ) { + my ( $link, $element, $attr, $tag ) = @$_; + + # + # Standardise the link (expands relative URLs, removes any fragment). + # Set $URI::ABS_REMOTE_LEADING_DOTS to ensure leading dots in relative + # URIs are removed. + # + local $URI::ABS_REMOTE_LEADING_DOTS = 1; + $uri = URI->new_abs( $link, $baseURL ); + $slink = sprintf( "%s:%s", $uri->scheme, $uri->opaque ); + + # + # Is it an HPR link? + # + if ( $slink =~ $linkre ) { + # + # The URL we found might be a link into an HTML file with an + # '#anchor' component ("fragment"). Save the last bracketed match, + # without any 'fragment' if there is one to get a clean filename + # or path. + # + ( $filepath = "$+" ) =~ s/#.*$//; + + _debug( $DEBUG >= 3, "Link: $slink\n" ); + _debug( $DEBUG >= 3, "File path: $filepath\n" ); + + # + # Does this file path begin with an 'hpr' prefix? If so is it the + # show id? If not we don't want to process it. + # + if ( $filepath =~ /^(hpr[0-9]{1,4})/ ) { + if ( $1 ne $epstr ) { + _debug( $DEBUG >= 3, "Ignored $slink\n" ); + next; + } + } + + # + # The path and URL might end with a slash which means the URL is + # relying on the Web server to fill in the filename as + # 'index.html'. We have to make this explicit. + # + if ( $slink =~ /\/$/ ) { + $slink .= 'index.html'; + $filepath .= 'index.html'; + } + + # + # Initialise this hash element if needed + # + unless ( exists( $rlinks->{$episode} ) ) { + $rlinks->{$episode} = []; + } + + # + # Stash this filename if it's not already stashed, and if it's + # HTML get the link and recurse + # + unless ( + any { $_->{filename} eq $filepath } + @{ $rlinks->{$episode} } + ) + { + _debug( $DEBUG >= 3, "Stashed $slink and $filepath\n" ); + + push( + @{ $rlinks->{$episode} }, + { filename => $filepath, URL => $slink } + ); + $linkcount++; + + # + # An HTML file has to be investigated + # + if ( $filepath =~ /\.html$/ ) { + $content = get($slink); + unless ( defined($content) ) { + carp "Link $slink returned nothing\n"; + } + else { + $linkcount + += find_links( $episode, $content, $rlinks ); + } + } + + } + + } + # + # It's not an HPR URL + # + else { + # + # Stash this filename if it's not already stashed + # + unless ( + any { $_->{filename} eq $filepath } + @{ $rlinks->{$episode} } + ) + { + _debug( $DEBUG >= 3, "Stashed $slink and $filepath\n" ); + + push( + @{ $rlinks->{$episode} }, + { filename => $filepath, URL => $slink } + ); + $linkcount++; + + } + } + } + + _debug( $DEBUG >= 3, "find_links exiting with $linkcount links\n" ); + + # + # Return the link count + # + return $linkcount; + + +} + +#=== FUNCTION ================================================================ +# NAME: emit +# PURPOSE: Print text on STDERR unless silent mode has been selected +# PARAMETERS: - Boolean indicating whether to print or not +# - list of arguments to 'print' +# RETURNS: Nothing +# DESCRIPTION: This is a wrapper around 'print' to determine whether to send +# a message to STDERR depending on a boolean. We need this to be +# able to make the script silent when the -verbose option is +# not selected +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub emit { + if (shift) { + print STDERR @_; + } +} + +#=== FUNCTION ================================================================ +# NAME: _debug +# PURPOSE: Prints debug reports +# PARAMETERS: $active Boolean: 1 for print, 0 for no print +# $message Message to print +# RETURNS: Nothing +# DESCRIPTION: Outputs a message if $active is true. It removes any trailing +# newline and then adds one in the 'print' to the caller doesn't +# have to bother. Prepends the message with 'D> ' to show it's +# a debug message. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub _debug { + my ( $active, $message ) = @_; + + chomp($message); + print STDERR "D> $message\n" if $active; +} + +#=== FUNCTION ================================================================ +# NAME: Options +# PURPOSE: Processes command-line options +# PARAMETERS: $optref Hash reference to hold the options +# RETURNS: Undef +# DESCRIPTION: +# THROWS: no exceptions +# COMMENTS: none +# SEE ALSO: n/a +#=============================================================================== +sub Options { + my ($optref) = @_; + + my @options = ( + "help", "documentation|man", "debug=i", "dry-run!", + "verbose+", "dbconfig=s", "from=s", "count=i", + ); + + if ( !GetOptions( $optref, @options ) ) { + pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1, + -verbose => 0 ); + } + + return; +} + +__END__ + +#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +# Application Documentation +#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +#{{{ + +=head1 NAME + + - + +=head1 VERSION + +The initial template usually just has: + +This documentation refers to version 0.0.2 + + +=head1 USAGE + + # Brief working invocation example(s) here showing the most common usage(s) + + # This section will be as far as many users ever read + # so make it as educational and exemplary as possible. + + +=head1 REQUIRED ARGUMENTS + +A complete list of every argument that must appear on the command line. +when the application is invoked, explaining what each of them does, any +restrictions on where each one may appear (i.e. flags that must appear +before or after filenames), and how the various arguments and options +may interact (e.g. mutual exclusions, required combinations, etc.) + +If all of the application's arguments are optional this section +may be omitted entirely. + + +=head1 OPTIONS + +A complete list of every available option with which the application +can be invoked, explaining what each does, and listing any restrictions, +or interactions. + +If the application has no options this section may be omitted entirely. + + +=head1 DESCRIPTION + +A full description of the application and its features. +May include numerous subsections (i.e. =head2, =head3, etc.) + + +=head1 DIAGNOSTICS + +A list of every error and warning message that the application can generate +(even the ones that will "never happen"), with a full explanation of each +problem, one or more likely causes, and any suggested remedies. If the +application generates exit status codes (e.g. under Unix) then list the exit +status associated with each error. + + +=head1 CONFIGURATION AND ENVIRONMENT + +A full explanation of any configuration system(s) used by the application, +including the names and locations of any configuration files, and the +meaning of any environment variables or properties that can be set. These +descriptions must also include details of any configuration language used + + +=head1 DEPENDENCIES + +A list of all the other modules that this module relies upon, including any +restrictions on versions, and an indication whether these required modules are +part of the standard Perl distribution, part of the module's distribution, +or must be installed separately. + + +=head1 INCOMPATIBILITIES + +A list of any modules that this module cannot be used in conjunction with. +This may be due to name conflicts in the interface, or competition for +system or program resources, or due to internal limitations of Perl +(for example, many modules that use source code filters are mutually +incompatible). + + +=head1 BUGS AND LIMITATIONS + +A list of known problems with the module, together with some indication +whether they are likely to be fixed in an upcoming release. + +Also a list of restrictions on the features the module does provide: +data types that cannot be handled, performance issues and the circumstances +in which they may arise, practical limitations on the size of data sets, +special cases that are not (yet) handled, etc. + +The initial template usually just has: + +There are no known bugs in this module. +Please report problems to () +Patches are welcome. + +=head1 AUTHOR + + () + + +=head1 LICENCE AND COPYRIGHT + +Copyright (c) (). All rights reserved. + +Followed by whatever licence you wish to release it under. +For Perl code that is often just: + +This module is free software; you can redistribute it and/or +modify it under the same terms as Perl itself. See perldoc perlartistic. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + +=cut + +#}}} + +# [zo to open fold, zc to close or za to toggle] + # vim: syntax=perl:ts=8:sw=4:et:ai:tw=78:fo=tcrqn21:fdm=marker diff --git a/Miscellaneous/fix_tags b/Miscellaneous/fix_tags new file mode 100755 index 0000000..3422e83 --- /dev/null +++ b/Miscellaneous/fix_tags @@ -0,0 +1,1051 @@ +#!/usr/bin/perl +#=============================================================================== +# +# FILE: fix_tags +# +# USAGE: ./fix_tags [-help] [-version] [-album=ALBUMSTRING] +# [-artist=ARTISTSTRING] [-comment=COMMENTSTRING] +# [-genre=GENRESTRING] [-title=TITLESTRING] [-track=TRACKNUMBER] +# [-year=YEAR] [-filter TAGNAME=FILTERNAME] [-[no]format] +# [-width=N] [-edit=TAGNAME] [-[no]silent] audio_file ... +# +# DESCRIPTION: A tool for reporting and altering tags in audio files of +# a variety of formats +# +# OPTIONS: --- +# REQUIREMENTS: --- +# BUGS: TryCatch is failing in late May 2020. Changed to Try::Tiny +# NOTES: --- +# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com +# LICENCE: Copyright (c) year 2011, 2012, 2013, 2014, 2015, 2016, 2017, +# 2018, 2019, 2020, 2021, 2022, 2023, 2024 - Dave Morriss +# VERSION: 1.3.10 +# CREATED: 2011-12-12 22:00:34 +# REVISION: 2024-06-14 15:15:08 +# +#=============================================================================== + +use Modern::Perl '2024'; +use utf8; +use feature qw{ postderef say signatures state try }; +no warnings + qw{ experimental::postderef experimental::signatures experimental::try }; + +use Getopt::Long; +use Pod::Usage; + +use Data::Dumper; +use Data::HexDump; + +use File::stat; +use File::Temp; +use File::Slurp; +use Date::Manip::Delta; +use Date::Manip::TZ; +use Audio::TagLib; +use Encode qw(encode decode); + +#use TryCatch; # Broke in late May 2020 due to a problem with Devel::Declare +#use Try::Tiny; + +use HTML::Restrict; + +# +# Version number (manually incremented) +# +our $VERSION = '1.3.10'; + +# +# Script name +# +( my $PROG = $0 ) =~ s|.*/||mx; + +# +# Declarations +# +my ( $sb, $fyear, $ref, $tag, $aprop, $changed ); +my ( %tags, @errors ); + +# +# Display of tags. Controlled by left margin width ($lmwidth) for the tag +# name, followed by a colon and a space. The right margin starts in column +# $lmwidth+2 ($rmpos). +# +my $lmwidth = 10; +my $rmpos = $lmwidth + 2; +my $fmt1 = "%-${lmwidth}s:"; +my $fmt2 = "%-${lmwidth}s: %s\n"; +my $threshold; + +# +# The Audio::TagLib methods to call for each tag manipulated by the script. +# The number after the method name is 1 if the value being set is a string, +# and zero otherwise. +# +my %tagmethods = ( + album => [ 'setAlbum', 1 ], + artist => [ 'setArtist', 1 ], + comment => [ 'setComment', 1 ], + genre => [ 'setGenre', 1 ], + title => [ 'setTitle', 1 ], + track => [ 'setTrack', 0 ], + year => [ 'setYear', 0 ], +); + +# +# Internal routines to invoke to perform filtering tasks +# +my %filtermethods = ( + clean => \&clean_string, + underscore => \&replace_underscores, + HTML => \&remove_HTML, +); + +# +# Settings to show whether a tag can be edited with an editor +# +my %editable = ( + album => 0, + artist => 0, + comment => 1, + genre => 0, + title => 1, + track => 0, + year => 0, +); + +# +# Ensure STDOUT and STDERR are in UTF8 mode +# +binmode STDOUT, ":encoding(UTF-8)"; +binmode STDERR, ":encoding(UTF-8)"; + +#------------------------------------------------------------------------------- +# Options and arguments +#------------------------------------------------------------------------------- +my $DEF_DEBUG = 0; +my $DEFWIDTH = 80; +my $MINWIDTH = 60; + +my ( %options, %filter ); +Options( \%options, \%filter ); + +# +# Default help +# +pod2usage( + -msg => "Version $VERSION\n", + -verbose => 2, + -exitval => 1 +) if ( $options{'help'} ); + +# +# Report version and leave if requested +# +if ( $options{'version'} ) { + say "This is $PROG version $VERSION"; + exit 1; +} + +# +# Collect options. Make internal form strings if they might be UTF-8 +# +my $DEBUG = ( defined( $options{debug} ) ? $options{debug} : $DEF_DEBUG ); +my $album = $options{album}; +my $artist = $options{artist}; +my $comment = $options{comment}; +my $genre = $options{genre}; +my $title = decode('utf8',$options{title}); +my $track = $options{track}; +my $year = $options{year}; + +my $format = ( defined( $options{format} ) ? $options{format} : 0 ); +my $width = ( defined( $options{width} ) ? $options{width} : $DEFWIDTH ); + +my $edit = $options{edit}; + +my $silent = ( defined( $options{silent} ) ? $options{silent} : 0 ); + +# +# Limit the minimum width +# +$width = $MINWIDTH if ( $width < $MINWIDTH ); + +# +# Check the filter options +# +unless ( check_filters( \%filter, \@errors ) ) { + print STDERR join( "\n", @errors ), "\n"; + exit(1); +} + +# +# Check the tag is known and can be edited +# +if ( defined($edit) ) { + $edit = lc($edit); + die "Unknown tag $edit\n" unless (exists($editable{$edit})); + unless ($editable{$edit}) { + warn "Tag $edit cannot be edited\n"; + undef($edit); + } +} + +my @files = @ARGV; + +pod2usage( + -msg => "Missing arguments\n\nVersion $VERSION\n", + -exitval => 1 +) unless @files; + +#------------------------------------------------------------------------------- +# Main Loop. Processes one or more files on the command line. +#------------------------------------------------------------------------------- +foreach my $file (@files) { + unless ( -e $file ) { + warn "$file does not exist\n"; + next; + } + + # + # Report the file name + # + print "$file\n" unless $silent; + + # + # If the file is empty report it and skip it + # + if ( -z $file ) { + warn "File $file is empty\n"; + next; + } + + $sb = stat($file); + $fyear = ( localtime( $sb->mtime ) )[5] + 1900; + + # + # Extract all the tags. Catch errors if someone tries to use this tool on + # a file that Audio::TagLib doesn't know about + # + try { + $ref = Audio::TagLib::FileRef->new($file); + $tag = $ref->tag(); + $aprop = $ref->audioProperties(); + + # + # Save the tags from the file in the %tags hash + # + %tags = ( + title => $tag->title()->toCString(), + artist => $tag->artist()->toCString(), + album => $tag->album()->toCString(), + comment => $tag->comment()->toCString(), + genre => $tag->genre()->toCString(), + year => $tag->year(), + track => $tag->track(), + length => sprintf( "%s (%d sec)", + interval( $aprop->length() ), + $aprop->length() ), + ); + } + + # + # We choked on something nasty + # + catch ($e) { + warn "File $file apparently does not contain tags\n"; + next; + } + + # + # Dump the fields that could contain Unicode + # +# _debug($DEBUG eq 4, 'Before encoding...', +# "comment:\n" . HexDump($tags{comment}) . "\n", +# "title:\n" . HexDump($tags{title}) . "\n" +# ); + + # + # Try and fix any UTF-8 weirdness. + # TODO: In theory this will crash if given seriously broken contents, but + # it's not been tested yet. + # +# for my $key (qw(comment title)) { +# $tags{$key} = encode( 'utf8', $tags{$key}, Encode::FB_CROAK ); +# } + + +# # +# # TODO: test making internal form from any UTF-8 data +# # +# try { +# $tags{title} = decode( 'utf8', $tags{title}, Encode::FB_CROAK ); +# } +# catch ($e) { +# warn "Problem decoding \$tags{title}\n" +# } + + # + # Dump tags at level 3, hexdump comment and title at level 4 ł + # +# _debug($DEBUG eq 3, '%tags: ' . Dumper(\%tags)); +# _debug($DEBUG eq 4, 'After encoding...', +# "comment:\n" . HexDump($tags{comment}) . "\n", +# "title:\n" . HexDump($tags{title}) . "\n" +# ); + + # + # Report current tags (unless asked to be silent). Use formatting if + # requested and it's warranted + # + $threshold = $width - $rmpos; + unless ($silent) { + for my $key ( sort( keys(%tags) ) ) { + if ( $format && length( $tags{$key} ) > $threshold ) { + print textFormat( $tags{$key}, sprintf( $fmt1, $key ), + 'R', $rmpos, $width ), + "\n"; + } + else { +# if ( $key =~ /^(comment|title)$/ ) { + if ( $key =~ /^title$/ ) { + printf $fmt2, $key, encode('utf8',coalesce( $tags{$key}, '' )); + } + else { + printf $fmt2, $key, coalesce( $tags{$key}, '' ); + } + } + } + } + + $changed = 0; + + # + # Change album, artist name, comment, genre, track number or year if + # requested + # + $changed += changeTag( $tag, 'album', \%tags, $album, 'setAlbum', 1 ); + $changed += changeTag( $tag, 'artist', \%tags, $artist, 'setArtist', 1 ); + $changed + += changeTag( $tag, 'comment', \%tags, $comment, 'setComment', 1 ); + $changed += changeTag( $tag, 'genre', \%tags, $genre, 'setGenre', 1 ); + $changed += changeTag( $tag, 'title', \%tags, $title, 'setTitle', 1 ); + $changed += changeTag( $tag, 'track', \%tags, $track, 'setTrack', 0 ); + $changed += changeTag( $tag, 'year', \%tags, $year, 'setYear', 0 ); + + # + # Do some filtering + # + $changed += apply_filters( $tag, \%tags, \%tagmethods, \%filter, + \%filtermethods ); + + # + # Perform an edit if requested (one per invocation) + # + if ( defined($edit) ) { + $changed += edit_tag( $tag, $edit, \%tags, \%tagmethods ); + } + + # + # Update if there are changes + # + if ($changed) { + $ref->save(); + } + +} +continue { + print "\n" unless $silent; +} + +exit; + +#=== FUNCTION ================================================================ +# NAME: changeTag +# PURPOSE: Changes a tag to a new value if appropriate +# PARAMETERS: $tag Tag object +# $tagname Name of tag +# $tags Hashref containing the converted tags from the +# current file +# $newValue New value of tag or undefined +# $setFunc String containing the name of the 'set' +# function +# $isString True if the value being set is a string +# RETURNS: 1 if a change has been made, 0 otherwise +# DESCRIPTION: Checks that $newValue is defined (it can be an empty string) +# and that the new value differs from the old one, returning if +# not. The $isString value defaults to zero. Ensures that a null +# $newValue is replaced by a zero if the tag is numeric. Reports +# what change has been requested then makes the change. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: +#=============================================================================== +sub changeTag { + my ( $tag, $tagname, $tags, $newValue, $setFunc, $isString ) = @_; + + my $oldValue = $tags->{$tagname}; + + return 0 unless defined($newValue); + return 0 if $oldValue eq $newValue; + + $isString = 0 unless defined($isString); + + $newValue = 0 if ( $newValue eq '' && !$isString ); + + my $nV_utf8 = decode('utf8',$newValue); + print "Changing $tagname to '$nV_utf8'\n"; + $tag->$setFunc( + ( $isString + ? Audio::TagLib::String->new($newValue) + : $newValue + ) + ); + + $tags->{$tagname} = $newValue; + + return 1; +} + +#=== FUNCTION ================================================================ +# NAME: check_filters +# PURPOSE: Check that the filter hash contains valid settings +# PARAMETERS: $filter Hashref containing the filter options +# $errors Arrayref to contain error messages +# RETURNS: 1 (true) if all is well, otherwise 0 (false) +# DESCRIPTION: The check returns true if there are no filters. It returns +# false if there are any unknown tag names or any unknown filter +# names. +# THROWS: No exceptions +# COMMENTS: The knowledge of what is a vlaid tag name or filter name is +# within this function, which is not ideal for maintenance. +# SEE ALSO: N/A +#=============================================================================== +sub check_filters { + my ( $filters, $errors ) = @_; + + my @filterable_tags = (qw{ album artist comment genre title }); + my @valid_filters = (qw{clean underscore html}); + my @unknown; + + # + # Nothing is wrong if there are no filters + # + return 1 unless defined($filters); + + # + # Are any tag names unknown? + # + my %filterable = map { $_ => 1 } @filterable_tags; + @unknown = grep { !defined( $filterable{$_} ) } + map { lc($_) } keys( %{$filters} ); + if (@unknown) { + push( @{$errors}, "Error: Invalid filter(s)" ) + unless defined($errors); + push( @{$errors}, "Unknown tag names: " . join( ", ", @unknown ) ); + } + + # + # Are any filter names unknown? + # + my %valid = map { $_ => 1 } @valid_filters; + my @names = map { @{ $filters->{$_} } } keys( %{$filters} ); + @unknown = grep { !defined( $valid{$_} ) } map { lc($_) } @names; + if (@unknown) { + push( @{$errors}, "Error: Invalid filter(s)" ) + unless defined($errors); + push( @{$errors}, "Unknown filter names: " . join( ", ", @unknown ) ); + } + + # + # All tests passed if no errors + # + return scalar( @{$errors} ) == 0; +} + +#=== FUNCTION ================================================================ +# NAME: apply_filters +# PURPOSE: Looks for requested filters and the tags they are to be +# applied to and performs the necessary filtering +# PARAMETERS: $tag Tag object +# $tags Hashref containing the converted tags from the +# current file +# $tagmethods Hashref containing the Audio::TagLib method +# names per tag +# $filter Hashref containing filter options +# $filtermethods Hashref containing filter names and the +# routines that handle them +# RETURNS: Number of changes made +# DESCRIPTION: The $filter hash contains tag names as keys (lower- or +# upper-case). The value is an array of filter names (lower- or +# upper-case). We loop through the tag names looking for filter +# names and applying the filters we find. +# THROWS: No exceptions +# COMMENTS: The sorting should be case-insensitive. +# SEE ALSO: N/A +#=============================================================================== +sub apply_filters { + my ( $tag, $tags, $tagmethods, $filter, $filtermethods ) = @_; + + my $lc_t; + my $newtag; + my $changes = 0; + + # + # Loop through the tags we are to filter in sorted order + # + for my $t ( sort( keys( %{$filter} ) ) ) { + + # + # We need a lowercase key to access the tag + # + $lc_t = lc($t); + + # + # Loop through all available methods and apply them if requested + # + for my $f ( sort( keys( %{$filtermethods} ) ) ) { + if ( grep( /^$f$/i, @{ $filter->{$t} } ) ) { + $newtag = &{ $filtermethods->{$f} }( $tags->{$lc_t} ); + $changes += changeTag( $tag, $lc_t, $tags, + $newtag, @{ $tagmethods->{$lc_t} } ); + } + + } + } + + return $changes; +} + +#=== FUNCTION ================================================================ +# NAME: edit_tag +# PURPOSE: Edit a tag in an editor +# PARAMETERS: $tag Tag object +# $tagname Name of tag +# $tags Hashref containing the converted tags from the +# current file +# $tagmethods Hashref containing the Audio::TagLib method +# names per tag +# RETURNS: 1 if there has been a change, otherwise 0 +# DESCRIPTION: For editing very log tags like 'comment'. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub edit_tag { + my ( $tag, $tagname, $tags, $tagmethods ) = @_; + + my $oldValue = $tags->{$tagname}; + my $newValue; + my $changes = 0; + + my $tfh = File::Temp->new; + my $tfn = $tfh->filename; + print $tfh $oldValue; + $tfh->close; + die "Edit failed\n" + unless ( system( ( 'vim', $tfn ) ) == 0 ); + + $newValue = read_file( $tfn, { binmode => ':utf8' } ); + chomp($newValue); + + $changes += changeTag( $tag, $tagname, $tags, + $newValue, @{ $tagmethods->{$tagname} } ); + + return $changes; +} + +#=== FUNCTION ================================================================ +# NAME: textFormat +# PURPOSE: Formats a block of text in an indented, wrapped style with +# a label in the left column +# PARAMETERS: $text The text to be formatted, as a scalar string +# $tag The label to be added to the left of the top +# line +# $align Tag alignment, 'L' for left, otherwise right +# $lmargin Width of the left margin (assumed to be big +# enough for the tag) +# $textwidth The width of all of the text plus left margin +# (i.e. the right margin) +# RETURNS: The formatted result as a string +# DESCRIPTION: Chops the incoming text into words (thereby removing any +# formatting). Removes any leading spaces. Loops through the +# wordlist building them into lines of the right length to fit +# between the left and right margins. Saves the lines in an +# array. Adds the tag to the first line with the alignment +# requested then returns the array joined into a string. +# THROWS: No exceptions +# COMMENTS: Inspired by Text::Format but *much* simpler. In fact T::F is +# a nasty thing to have to use; I couldn't get it to do what +# this routine does. +# TODO Make the routine more resilient to silly input values. +# SEE ALSO: +#=============================================================================== +sub textFormat { + my ( $text, $tag, $align, $lmargin, $textwidth ) = @_; + + my ( $width, $word ); + my ( @words, @buff, @wrap ); + + # + # Build the tag early. If there's no text we'll just return the tag. + # + $tag = sprintf( "%*s", + ( $align =~ /L/i ? ( $lmargin - 1 ) * -1 : $lmargin - 1 ), $tag ); + + return $tag unless $text; + + $text =~ s/(^\s+|\s+$)//g; + return $tag unless $text; + + # + # Chop up the incoming text removing leading spaces + # + @words = split( /\s+/, $text ); + shift(@words) if ( @words && $words[0] eq '' ); + + # + # Compute the width of the active text + # + $width = $textwidth - $lmargin; + + # + # Format the words into lines with a blank left margin + # + while ( defined( $word = shift(@words) ) ) { + if ( length( join( ' ', @buff, $word ) ) < $width ) { + push( @buff, $word ); + } + else { + push( @wrap, ' ' x $lmargin . join( ' ', @buff ) ); + @buff = ($word); + } + } + + # + # Append any remainder + # + push( @wrap, ' ' x $lmargin . join( ' ', @buff ) ) if @buff; + + # + # Insert the tag into the first line + # + substr( $wrap[0], 0, $lmargin - 1 ) = $tag; + + # + # Return the formatted array as a string + # + return join( "\n", @wrap ); + +} + +#=== FUNCTION ================================================================ +# NAME: interval +# PURPOSE: Convert a time in seconds to a valid 'HH:MM:SS' interval +# PARAMETERS: $time the time to convert in seconds +# RETURNS: The interval string in the format 'HH:MM:SS' or undef +# DESCRIPTION: TODO +# THROWS: No exceptions +# COMMENTS: Adapted from a routine for generating valid PostgreSQL +# interval times. Probably could be simplified +# SEE ALSO: +#=============================================================================== +sub interval { + my ($time) = @_; + + return undef unless $time; ## no critic + + my $date = Date::Manip::Delta->new; + unless ( $date->parse($time) ) { + return $date->printf("%02hv:%02mv:%02sv"); + } + else { + warn "Invalid time $time\n"; + return undef; ## no critic + } + +} + +#=== FUNCTION ================================================================ +# NAME: clean_string +# PURPOSE: Clean a string of non-printables, newlines, multiple spaces +# PARAMETERS: $str The string to process +# RETURNS: The processed string +# DESCRIPTION: Removes leading and trailing spaces. Removes all non-printable +# characters. Removes all CR/LF sequences. Replaces multiple +# white space characters with a single space. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: +#=============================================================================== +#sub clean_string { +# my ($str) = @_; +# +# $str =~ s/(^\s+|\s+$)//g; +# #$str =~ tr/[[:graph:]]//c; # Documented as not working +# $str =~ s/[^[:graph:] ]/ /g; +# $str =~ tr/\x0A\x0D/ /s; +# $str =~ tr/ \t/ /s; +# +# return $str; +#} + +#=== FUNCTION ================================================================ +# NAME: clean_string +# PURPOSE: Clean a string of non-printables, newlines, multiple spaces +# PARAMETERS: $str The string to process +# RETURNS: The processed string +# DESCRIPTION: Removes leading and trailing spaces. Removes all non-printable +# characters. Removes all CR/LF sequences. Replaces multiple +# spaces with a single space. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: +#=============================================================================== +sub clean_string { + my ($str) = @_; + + # + # Leading and trailing spaces + # + $str =~ s/(^\s+|\s+$)//g; + + # + # Remove CR/LF + # + #$str =~ tr/\x0A\x0D/ /; + $str =~ s/\x0A\x0D/ /g; + + # + # Fix   – — ‘ ’ “ ” + # + #$str =~ tr/\xa0\N{U+2013}\N{U+2014}\N{U+2018}\N{U+2019}\N{U+201C}\N{U+201D}/ --''""/; + $str =~ s/\x{00a0}/ /g; + $str =~ s/\N{U+00A0}/ /g; # Test + $str =~ s/\N{U+2013}/-/g; + $str =~ s/\N{U+2014}/-/g; + $str =~ s/\N{U+2018}/'/g; + $str =~ s/\N{U+2019}/'/g; # ’ ’ ’ + $str =~ s/\N{U+2026}/.../g; + $str =~ s/\N{U+201C}/"/g; + $str =~ s/\N{U+201D}/"/g; + + # + # FIXME: Experimental. Should we convert entities instead? + # + $str =~ s/’/'/g; + $str =~ s/…/.../g; + $str =~ s/“/"/g; + $str =~ s/”/"/g; + + # + # All remaining non-graph characters to spaces + # + #$str =~ tr/[[:graph:]]//c; # Documented as not working + $str =~ s/[^[:graph:] ]/ /g; + + # + # Crunch spaces + # + $str =~ tr/ / /s; + + return $str; +} + +#=== FUNCTION ================================================================ +# NAME: replace_underscores +# PURPOSE: Replaces underscores in a string by spaces +# PARAMETERS: $str The string to process +# RETURNS: The processed string +# DESCRIPTION: +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub replace_underscores { + my ($str) = @_; + + $str =~ s/_/ /g; + + return $str; +} + +#=== FUNCTION ================================================================ +# NAME: remove_HTML +# PURPOSE: Clean a string of HTML tags +# PARAMETERS: $str The string to process +# RETURNS: The processed string +# DESCRIPTION: +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub remove_HTML { + my ($str) = @_; + + my $hr = HTML::Restrict->new(); + my $processed = $hr->process($str); + + return $processed; +} + +#=== FUNCTION ================================================================ +# NAME: coalesce +# PURPOSE: To find the first defined argument and return it +# PARAMETERS: Arbitrary number of arguments +# RETURNS: The first defined argument or undef if there are none +# DESCRIPTION: Whichever of the arbitrary number of arguments is found to be +# defined on examining them sequentially is returned. If none +# are found then the routine returns 'undef'. Modelled after the +# SQL function of the same name. +# THROWS: No exceptions +# COMMENTS: None +# SEE ALSO: N/A +#=============================================================================== +sub coalesce { + foreach (@_) { + return $_ if defined($_); + } + return undef; ## no critic +} + +#=== FUNCTION ================================================================ +# NAME: _debug +# PURPOSE: Prints debug reports +# PARAMETERS: $active Boolean: 1 for print, 0 for no print +# $messages... Arbitrary list of messages to print +# RETURNS: Nothing +# DESCRIPTION: Outputs messages if $active is true. It removes any trailing +# newline from each one and then adds one in the 'print' to the +# caller doesn't have to bother. Prepends each message with 'D>' +# to show it's a debug message. +# THROWS: No exceptions +# COMMENTS: Differs from other functions of the same name +# SEE ALSO: N/A +#=============================================================================== +sub _debug { + my $active = shift; + + my $message; + return unless $active; + + while ($message = shift) { + chomp($message); + print STDERR "D> $message\n"; + } +} + +#=== FUNCTION ================================================================ +# NAME: Options +# PURPOSE: Processes command-line options +# PARAMETERS: $optref Hash reference to hold the options +# $filter Hash reference to hold filter options +# RETURNS: Undef +# DESCRIPTION: +# THROWS: no exceptions +# COMMENTS: none +# SEE ALSO: n/a +#=============================================================================== +sub Options { + my ( $optref, $filter ) = @_; + + my @options = ( + "help", "version", "debug=i", "album:s", + "artist:s", "comment:s", "genre:s", "title:s", + "track:s", "year:s", "format!", "width=i", + "edit=s", "silent!", + ); + + # + # Implement '-filter=TAGNAME=FILTERNAME' (from the Getopt::Long manpage) + # + my %opthash + = ( "filter=s%" => sub { push( @{ $filter->{ $_[1] } }, $_[2] ) }, ); + + if ( !GetOptions( $optref, @options, %opthash ) ) { + pod2usage( -msg => "Version $VERSION\n", -exitval => 1 ); + } + + return; +} + +__END__ + +#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +# Application Documentation +#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +#{{{ + +=head1 NAME + +fix_tags - manipulate ID3 and equivalent tags + +=head1 VERSION + +This documentation refers to I version 1.3.10 + +=head1 USAGE + + fix_tags [-help] [-version] [-album=ALBUMSTRING] [-artist=ARTISTSTRING] + [-comment=COMMENTSTRING] [-genre=GENRESTRING] [-title=TITLESTRING] + [-track=TRACKNUMBER] [-year=YEAR] [-filter TAGNAME=FILTERNAME] + [-[no]format] [-width=N] [-edit=TAGNAME] [-[no]silent] audio_file ... + +=head1 OPTIONS + +=over 8 + +=item B<-help> + +Prints a brief help message describing the usage of the program, and then exits. + +=item B<-version> + +Prints the script name and version, and then exits. + +=item B<-album=ALBUMSTRING> + +Sets the album tag to the string defined by the option. Use B<-album=> to +clear the tag. + +=item B<-artist=ARTISTSTRING> + +Sets the artist tag to the string defined by the option. Use B<-artist=> to +clear the tag. + +=item B<-comment=COMMENTSTRING> + +Sets the comment tag to the string defined by the option. Use B<-comment=> to +clear the tag. + +=item B<-genre=GENRESTRING> + +Sets the genre tag to the string defined by the option. Use B<-genre=> to +clear the tag. + +=item B<-title=TITLESTRING> + +Sets the title tag to the string defined by the option. Use B<-title=> to +clear the tag. + +=item B<-track=TRACKNUMBER> + +Sets the track tag to the number defined by the option. Use B<-track=> to +set the tag to zero. + +=item B<-year=YEAR> + +Sets the year tag to the number defined by the option. Use B<-year=> to +set the tag to zero. + +=item B<-[no]format> + +If this option is enabled long tag values are wrapped in the report. The width +of the resulting lines is controlled by the B<-width=N> option. The default is +not to wrap. + +=item B<-width=N> + +Ths option controls the formatting of tag values when the B<-format> option is +chosen. If a tag value would exceed this width after displaying the tag name, +then formatting is applied. The width specified cannot be less than 40. The +default width is 80. + +=item B<-edit=TAGNAME> + +This option invokes an editor (Vim) to edit the nominated tag. The B +must be spelled out in full; abbreviations are not allowed. Only a limited +subset of tags can be edited since only the longer tags need to be processed +in this way. + +=item B<-filter TAGNAME=FILTERNAME> or B<-filter=TAGNAME=FILTERNAME> + +This option provides an interface to the filtering capability of the script. +Here B denotes the full name of one of the text tags (album, artist, +comment, genre or title - not track or year), and B is one of the +built-in filters: + +=over 4 + +=item B + +The tag string has leading and trailing spaces removed. Multiple internal +spaces and tabs are replaced by a single space. CR/LF sequences are removed, +as are all non-graphic characters. + +=item B + +All underscores in the tag string are replaced by spaces. No space compression +takes place. + +=item B + +The tag string is fed through the I module and all HTML tags +are removed. + +=back + +Neither the B nor the B parts of the option may be +abbreviated, but neither is case-sensitive. + +Multiple filters may be specified by repeating the complete option sequence. +For example: + + fix_tags -filter comment=clean -fil comment=underscore FILE + +The script processes the tags specified in the B<-filter> option in alphabetic +order, and for a given tag it also processes the filters in alphabetic order. + +=item B<-[no]silent> + +This option controls whether the filename and the tags are reported. It does +not affect whether any changes are reported. By default B<-nosilent> is +selected, and the tags are reported. + +=back + +=head1 DESCRIPTION + +This script manipulates ID3 tags (or their equivalents) in FLAC, MP3, OGG, SPX +and WAV files. + +=head1 DEPENDENCIES + + Audio::TagLib + Data::Dumper + Date::Manip::Delta + Date::Manip::TZ + File::stat + Getopt::Long + HTML::Restrict + Pod::Usage + TryCatch + + +=head1 BUGS AND LIMITATIONS + +The script can crash when processing very bizarre tag contents. Up to now this +has been resolved by using another tag tool to rewrite the tag in error. There +is as yet no real data about what has been going on. + +Please report problems to Dave Morriss (Dave.Morriss@gmail.com) +Patches are welcome. + +=head1 AUTHOR + +Dave Morriss (Dave.Morriss@gmail.com) 2011, 2012, 2013, 2014, 2015, 2016 + +=head1 LICENCE AND COPYRIGHT + +Copyright (c) Dave Morriss (Dave.Morriss@gmail.com). All rights reserved. + +This program is free software. You can redistribute it and/or modify it under +the same terms as Perl itself. + +=cut + +#}}} + +# [zo to open fold, zc to close] + +# vim: syntax=perl:ts=8:sw=4:et:ai:tw=78:fo=tcrqn21:fdm=marker diff --git a/Miscellaneous/fix_tags.bin b/Miscellaneous/fix_tags.bin new file mode 100755 index 0000000..e0339ea Binary files /dev/null and b/Miscellaneous/fix_tags.bin differ diff --git a/PostgreSQL_Database/add_hosts_to_show b/PostgreSQL_Database/add_hosts_to_show index e9ea912..6a40143 100755 --- a/PostgreSQL_Database/add_hosts_to_show +++ b/PostgreSQL_Database/add_hosts_to_show @@ -535,7 +535,7 @@ sub find_hosts { # DESCRIPTION: Places all the hosts associated with the show into an array # then compares it with the @$hosts array such that what is left # is all the hosts that do not match hosts linked to the show. -# Of cxourse, if any of the hosts given as options are not exact +# Of course, if any of the hosts given as options are not exact # matches with the names from the database (they are regexes # perhaps) they'll be regarded as different. We have to do # further processing of what's returned from querying the diff --git a/PostgreSQL_Database/hpr_schema_2.pgsql b/PostgreSQL_Database/hpr_schema_2.pgsql index 2c89c98..b2aa980 100644 --- a/PostgreSQL_Database/hpr_schema_2.pgsql +++ b/PostgreSQL_Database/hpr_schema_2.pgsql @@ -22,6 +22,19 @@ * 'episodes_id' seems stupid. * * ------------------------------------------------------------------------------ + * Tables: + * + * comments + * episodes + * episodes_hosts_xref + * episodes_series_xref + * episodes_tags_xref + * hosts + * licenses + * series + * tags + * assets + * */ /* ------------------------------------------------------------------------------ @@ -63,13 +76,15 @@ DROP FUNCTION IF EXISTS comment_changed(); -- }}} --- \/\/ licenses \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ +-- +----------+ +-- | licenses | +-- +----------+ -- {{{ /* ------------------------------------------------------------------------------ * Table 'licenses' - licenses relating to episodes (needed because 'episodes' - * and hosts' reference it) + * and 'hosts' reference it) * * license_id primary key, the licence number * short_name brief name of CC licence @@ -123,7 +138,9 @@ ALTER FUNCTION id_in_licenses(sname varchar) -- }}} --- \/\/ episodes \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ +-- +----------+ +-- | episodes | +-- +----------+ -- {{{ @@ -203,6 +220,65 @@ ALTER FUNCTION id_in_episodes(ekey varchar) -- }}} +-- +-----------+ +-- | episodes2 | +-- +-----------+ + +-- {{{ + +/* ------------------------------------------------------------------------------ + * Table 'episodes2' - TWAT and HPR shows + * + * show_id zero for a TwaT show, 1 for an HPR show + * episode_id episode number in the range of show_id values + * release_date date the episode was released + * title title of the episode + * summary summary of the episode content + * notes the show notes (as an HTML fragment) + * explicit a Boolean; true for explicit, false for otherwise + * license the licence which the show is under (US spelling) + * duration the duration (time) of the audio + * downloads number of downloads + * archived a Boolean; true if the episode has been uploaded to the IA + * archive_date date the episode was archived + * IA_URL URL to the episode on archive.org + * journal a journal of actions performed on this episode + * + * ------------------------------------------------------------------------------ */ +-- CREATE TYPE episode_status AS ENUM ('reserved', 'processing', 'posted'); + +DROP TABLE IF EXISTS episodes2 CASCADE; +CREATE TABLE episodes2 ( + show_id smallint NOT NULL DEFAULT 1 + CHECK (show_id = 0 OR show_id = 1), + episode_id smallint NOT NULL, + release_date date NOT NULL, + title varchar(128) NOT NULL, + summary varchar(128), + notes text NOT NULL, + explicit boolean NOT NULL DEFAULT TRUE, + license integer NOT NULL DEFAULT id_in_licenses('CC-BY-SA') + REFERENCES licenses (license_id), + duration interval NOT NULL DEFAULT '00:00:00', + downloads integer NOT NULL DEFAULT 0, + archived boolean NOT NULL DEFAULT FALSE, + archive_date date, + IA_URL text, + status episode_status, + journal text, + PRIMARY KEY (show_id, episode_id) +); + +ALTER TABLE episodes2 + OWNER TO hpradmin; + +CREATE INDEX episode2_release_date_idx + ON episodes2 + USING btree + (release_date); + +-- }}} + -- \/\/ hosts /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ -- {{{ diff --git a/PostgreSQL_Database/nuke_n_pave.sh b/PostgreSQL_Database/nuke_n_pave.sh index 48d3ea2..33f3148 100755 --- a/PostgreSQL_Database/nuke_n_pave.sh +++ b/PostgreSQL_Database/nuke_n_pave.sh @@ -6,6 +6,16 @@ # hosts by adding the missing hosts. # +# +# Check that the ensuing destruction is really meant! +# +echo "** Warning** This script will DELETE the PostgreSQL database 'HPR2'" +read -r -p "Are you sure? [y/N] " ans +if [[ -z $ans || ${ans^^} = 'N' ]]; then + echo "Aborting" + exit +fi + # # Directories #