HPR/hpr-tools
5
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Updates from previous repo

Browse Source 
FAQ/FAQ.mkd, FAQ/Makefile: this version of the FAQ is now out of date
    and probably should be deleted.

InternetArchive/repair_item: script to upload missing shows after tie
    out errors during the normal upload; still under development.

InternetArchive/update_state: script to update show state in the
    'reservations' table in the database. Uses the CMS interface.

Link_Checker/scan_links: under development. Not currently usable.

Miscellaneous/fix_tags: audio metadata manipulation script. Recently
    added to this repo for convenience. Updates for 'experimental::try',
    the official Perl try/catch.

PostgreSQL_Database/add_hosts_to_show, PostgreSQL_Database/hpr_schema_2.pgsql,
    PostgreSQL_Database/nuke_n_pave.sh: an old experimental Pg database
    to take over from the previous MySQL version (from before 2023).
    Kept for reference; never implemented.
This commit is contained in:
Dave Morriss 2024-06-14 16:00:04 +01:00
parent 38abbcdd39
commit 50edeccc88
12 changed files with 1874 additions and 146 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

109
FAQ/FAQ.html
View File

@ -1,66 +1,13 @@
<!DOCTYPE html> <h1 class="unnumbered" data-number="" id="hacker-public-radio-faq-2021-04-01">Hacker Public Radio FAQ 2021-04-01</h1>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<meta name="author" content="HPR Contributors">
<title>Hacker Public Radio FAQ (draft)</title>
<style type="text/css">code{white-space: pre;}</style>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<link rel="stylesheet" href="http://hackerpublicradio.org/css/hpr.css">
</head>
<body id="home">
<div id="container" class="shadow">
<header>
<h1 class="title">Hacker Public Radio FAQ (draft)</h1>
<h2 class="subtitle">A collection of questions and answers with links to the HPR site</h2>
<h2 class="author">HPR Contributors</h2>
<hr/>
</header>
<main id="maincontent">
<article>
<header>
<h1>Table of Contents</h1>
<nav id="TOC">
<ul>
<li><a href="#hacker-public-radio-faq">Hacker Public Radio FAQ</a>
<ul>
<li><a href="#what-is-hacker-public-radio"><span class="toc-section-number">0.1</span> What is Hacker Public Radio?</a></li>
<li><a href="#what-is-a-show"><span class="toc-section-number">0.2</span> What is a show?</a></li>
<li><a href="#can-i-submit-a-show-made-for-another-podcast"><span class="toc-section-number">0.3</span> Can I submit a show made for another podcast?</a></li>
<li><a href="#what-defines-a-show-as-being-produced-for-hacker-public-radio"><span class="toc-section-number">0.4</span> What defines a show as being "<em>Produced for Hacker Public Radio</em>"?</a></li>
<li><a href="#what-are-the-mandatory-elements-of-an-hpr-show"><span class="toc-section-number">0.5</span> What are the mandatory elements of an HPR show?</a></li>
<li><a href="#is-it-ok-if-i-dont-include-notes-with-my-show"><span class="toc-section-number">0.6</span> Is it OK if I don't include notes with my show?</a></li>
<li><a href="#how-do-i-include-photographs-or-other-files-with-my-show"><span class="toc-section-number">0.7</span> How do I include photographs or other files with my show?</a></li>
<li><a href="#what-is-and-is-not-edited-in-a-show-after-submission"><span class="toc-section-number">0.8</span> What is and is not edited in a show after submission?</a></li>
<li><a href="#what-are-some-common-errors-in-show-notes"><span class="toc-section-number">0.9</span> What are some common errors in show notes?</a>
<ul>
<li><a href="#html-notes"><span class="toc-section-number">0.9.1</span> HTML notes</a></li>
<li><a href="#markdown-notes"><span class="toc-section-number">0.9.2</span> Markdown notes</a></li>
<li><a href="#general-errors"><span class="toc-section-number">0.9.3</span> General errors</a></li>
</ul></li>
<li><a href="#if-i-notice-an-error-in-my-shows-details-how-can-it-be-fixed"><span class="toc-section-number">0.10</span> If I notice an error in my show's details how can it be fixed?</a></li>
<li><a href="#does-hobbypublicradio.com-have-anything-to-do-with-hpr"><span class="toc-section-number">0.11</span> Does <code>hobbypublicradio.com</code> have anything to do with HPR?</a></li>
<li><a href="#how-can-i-talk-to-other-hpr-hosts-and-listeners"><span class="toc-section-number">0.12</span> How can I talk to other HPR hosts and listeners?</a></li>
</ul></li>
</ul>
</nav>
</header>
<h1 class="unnumbered" data-number="" id="hacker-public-radio-faq">Hacker Public Radio FAQ</h1>
<h2 data-number="0.1" id="what-is-hacker-public-radio"><span class="header-section-number">0.1</span> What is Hacker Public Radio?</h2> <h2 data-number="0.1" id="what-is-hacker-public-radio"><span class="header-section-number">0.1</span> What is Hacker Public Radio?</h2>
<ul> <ul>
<li><p>Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases shows every weekday Monday through Friday.</p></li> <li><p>Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases shows (episodes) every weekday Monday through Friday.</p></li>
<li><p>What differentiates HPR from other podcasts is that the shows are crowd sourced from the community — fellow listeners like you. There is no restriction on how long shows can be, nor on the topic you can cover as long as they are not spam and "are of interest to Hackers".</p> <li><p>What differentiates HPR from other podcasts is that the shows are crowd sourced from the community — fellow listeners like you. There is no restriction on how long shows can be, nor on the topic you can cover, as long as they are not spam and "are of interest to Hackers".</p>
<p>If you want to see what topics have been covered so far just have a look at our <a href="http://hackerpublicradio.org/index_full.php">Archive</a>. We also allow for a <a href="http://hackerpublicradio.org/series.php">series</a> of shows so that hosts can go into more detail on a topic.</p></li> <p>If you want to see what topics have been covered so far just have a look at our <a href="http://hackerpublicradio.org/index_full.php">Archive</a>. We also allow for shows to be grouped into <a href="http://hackerpublicradio.org/series.php">series</a> so that hosts can go into more detail on a topic.</p></li>
<li><p>Look at the <a href="http://hackerpublicradio.org/about.php"><em>About HPR</em></a> page for a full description of <em>The HPR Community</em>, <em>Free Culture</em>, <em>Governance</em> and our <em>Patrons</em>.</p></li> <li><p>Look at the <a href="http://hackerpublicradio.org/about.php"><em>About HPR</em></a> page for a full description of <em>The HPR Community</em>, <em>Free Culture</em>, <em>Governance</em> and our <em>Patrons</em>.</p></li>
<li><p>A recent question on the mailing list asks if HPR is a Content Publication Network like YouTube, or is it a Podcast that random people contribute to? Or to put it another way, is the host addressing the Internet in general or the HPR Audience?</p> <li><p>A question on the mailing list in 2020 asked if HPR is a Content Publication Network like YouTube, or is it a Podcast that random people contribute to? Or to put it another way, is the host addressing the Internet in general or the HPR Audience?</p>
<ul> <ul>
<li>The consensus of opinion is that HPR is a Podcast contributed to by the community.</li> <li>The consensus of opinion is that HPR is a Podcast contributed to by the community, not a Content Publication Network.</li>
</ul></li> </ul></li>
</ul> </ul>
<h2 data-number="0.2" id="what-is-a-show"><span class="header-section-number">0.2</span> What is a show?</h2> <h2 data-number="0.2" id="what-is-a-show"><span class="header-section-number">0.2</span> What is a show?</h2>
@ -79,7 +26,7 @@
<li><p>Each show is expected to have certain mandatory attributes listed <a href="#what-are-the-mandatory-elements-of-an-hpr-show">below</a>.</p></li> <li><p>Each show is expected to have certain mandatory attributes listed <a href="#what-are-the-mandatory-elements-of-an-hpr-show">below</a>.</p></li>
<li><p>HPR shows are usually expected to be informative or educational, and as such some sort of accompanying written material is highly desirable. This might consist of brief notes, links to relevant web sites, or possibly longer notes and examples. Photographs, diagrams and example files are also welcome if the host feels it helps to get the message across.</p></li> <li><p>HPR shows are usually expected to be informative or educational, and as such some sort of accompanying written material is highly desirable. This might consist of brief notes, links to relevant web sites, or possibly longer notes and examples. Photographs, diagrams and example files are also welcome if the host feels it helps to get the message across.</p></li>
<li><p>Shows are released under a <a href="">Creative Commons Attribution-ShareAlike 3.0 Unported</a> license. See <a href="http://hackerpublicradio.org/stuff_you_need_to_know.php"><em>Stuff you need to know</em></a> for much more detail about this and related issues.</p></li> <li><p>Shows are released under a <a href="">Creative Commons Attribution-ShareAlike 3.0 Unported</a> license. See <a href="http://hackerpublicradio.org/stuff_you_need_to_know.php"><em>Stuff you need to know</em></a> for much more detail about this and related issues.</p></li>
<li><p>As a contributor you can choose when your show will be released. We only release one show a day, during week days, but if a slot is free you can claim it. Go to the <a href="http://hackerpublicradio.org/calendar.php"><em>calendar</em></a> page to do this. You need to have your show ready to upload before you do this though.</p></li> <li><p>As a contributor you can choose when your show will be released. We only release one show a day, during week days, but if a slot is free you can claim it. Go to the <a href="http://hackerpublicradio.org/calendar.php"><em>calendar</em></a> page to do this. You need to have your show ready to upload before you reserve a slot though.</p></li>
</ul> </ul>
<h2 data-number="0.3" id="can-i-submit-a-show-made-for-another-podcast"><span class="header-section-number">0.3</span> Can I submit a show made for another podcast?</h2> <h2 data-number="0.3" id="can-i-submit-a-show-made-for-another-podcast"><span class="header-section-number">0.3</span> Can I submit a show made for another podcast?</h2>
<ul> <ul>
@ -105,30 +52,37 @@
<li>Audio</li> <li>Audio</li>
</ul></li> </ul></li>
</ul> </ul>
<h2 data-number="0.6" id="is-it-ok-if-i-dont-include-notes-with-my-show"><span class="header-section-number">0.6</span> Is it OK if I don't include notes with my show?</h2> <h2 data-number="0.6" id="what-are-tags"><span class="header-section-number">0.6</span> What are tags?</h2>
<ul> <ul>
<li>No, please don't do that. It is important to provide good show notes for reasons described on the <a href="http://hackerpublicradio.org/request_a_slot.php"><em>Picking a slot for your show</em></a> page: <li><p>We are trying to maintain a collection of tags to allow people to find shows on the same (or similar) topic.</p></li>
<li><p>Tags are simple words or short phrases describing aspects of a show such as what topics are covered in it. To get hints about what tags have already been used you can view all the current tags in alphabetical order on this page: <a href="http://hackerpublicradio.org/tags.php">Tag summary</a></p></li>
<li><p>Some examples are: <b>android</b>, <b>creative commons</b> and <b>gaming</b>.</p></li>
<li><p>Are there any tags that are best avoided?</p></li>
</ul>
<h2 data-number="0.7" id="is-it-ok-if-i-dont-include-notes-with-my-show"><span class="header-section-number">0.7</span> Is it OK if I don't include notes with my show?</h2>
<ul> <ul>
<li>They provide the listeners with more information on the topic</li> <li><strong>Please don't do that</strong>. It is important to provide good show notes for reasons described on the <a href="http://hackerpublicradio.org/request_a_slot.php"><em>Picking a slot for your show</em></a> page:
<li>They ensure your show gets posted on time</li> <ul>
<li>They provide the listeners with more information on the topic your show is covering</li>
<li>They ensure your show gets posted on time (the volunteers don't have to make notes for you)</li>
<li>They encourage people to download your show</li> <li>They encourage people to download your show</li>
<li>They allow your show to be indexed by search engines</li> <li>They allow your show to be indexed by search engines</li>
<li>They make your show accessible to the deaf and hard of hearing</li> <li>They make your show accessible to the deaf and hard of hearing</li>
</ul></li> </ul></li>
</ul> </ul>
<h2 data-number="0.7" id="how-do-i-include-photographs-or-other-files-with-my-show"><span class="header-section-number">0.7</span> How do I include photographs or other files with my show?</h2> <h2 data-number="0.8" id="how-do-i-include-photographs-or-other-files-with-my-show"><span class="header-section-number">0.8</span> How do I include photographs or other files with my show?</h2>
<ul> <ul>
<li>TBA</li> <li>TBA</li>
</ul> </ul>
<h2 data-number="0.8" id="what-is-and-is-not-edited-in-a-show-after-submission"><span class="header-section-number">0.8</span> What is and is not edited in a show after submission?</h2> <h2 data-number="0.9" id="what-is-and-is-not-edited-in-a-show-after-submission"><span class="header-section-number">0.9</span> What is and is not edited in a show after submission?</h2>
<ul> <ul>
<li><p>We don't listen to the audio before posting a show to the website, so we don't change the content.</p></li> <li><p>We don't listen to the audio before posting a show to the website, so we don't change the content.</p></li>
<li><p>Unless you have done so and told us, we add an introduction to the audio announcing what it is. Similarly we add an <em>outro</em>. This is not <u>editing</u> since we don't change the content.</p></li> <li><p>Unless you have done so and told us you have, we add an introduction to the audio announcing what it is. Similarly we add an <em>outro</em>. This is not <u>editing</u> since we don't change the content.</p></li>
<li><p>We may make changes to the show title, summary or tags. Usually these are minor changes such as spelling corrections. The intention is not any sort of censorship, merely to make these items more understandable or easier to index.</p></li> <li><p>We may make changes to the show title, summary or tags. Usually these are minor changes such as spelling corrections. The intention is not any sort of censorship, merely to make these items more understandable or easier to index.</p></li>
<li><p>Depending on the form the show notes take we may modify these.</p> <li><p>Depending on the form the show notes take we may modify these.</p>
<ul> <ul>
<li>If what is submitted is plain text we convert it to Markdown in order to generate HTML.</li> <li>If what is submitted is plain text we convert it to Markdown in order to generate HTML.</li>
<li>If the notes are one of the permitted markup formats (mostly Markdown) we may adjust this to ensure that valid HTML is generated.</li> <li>If the notes are one of the permitted markup formats (mostly Markdown flavours) we may adjust this to ensure that valid HTML is generated.</li>
<li>If the notes are HTML we run them through an HTML checker and correct any mistakes it highlights.</li> <li>If the notes are HTML we run them through an HTML checker and correct any mistakes it highlights.</li>
<li>We may correct spelling, grammar and punctuation if this clarifies anything which might otherwise be unclear.</li> <li>We may correct spelling, grammar and punctuation if this clarifies anything which might otherwise be unclear.</li>
<li>If despite the request for show notes of some kind, none are provided, we may add notes which help to clarify the content of the show. This may be done after the show is released (when we have had a chance to listen to it).</li> <li>If despite the request for show notes of some kind, none are provided, we may add notes which help to clarify the content of the show. This may be done after the show is released (when we have had a chance to listen to it).</li>
@ -137,33 +91,33 @@
<li><p>We may make changes to a show's metadata if the host requests it (see the item on <a href="#if-i-notice-an-error-in-my-shows-details-how-can-it-be-fixed">fixing errors after upload</a>).</p></li> <li><p>We may make changes to a show's metadata if the host requests it (see the item on <a href="#if-i-notice-an-error-in-my-shows-details-how-can-it-be-fixed">fixing errors after upload</a>).</p></li>
<li><p>If a show contains links to resources which disappear after a period of time, we may at some future time change the notes to refer to any <em>Wayback Machine</em> copy that can be found.</p></li> <li><p>If a show contains links to resources which disappear after a period of time, we may at some future time change the notes to refer to any <em>Wayback Machine</em> copy that can be found.</p></li>
</ul> </ul>
<h2 data-number="0.9" id="what-are-some-common-errors-in-show-notes"><span class="header-section-number">0.9</span> What are some common errors in show notes?</h2> <h2 data-number="0.10" id="what-are-some-common-errors-in-show-notes"><span class="header-section-number">0.10</span> What are some common errors in show notes?</h2>
<p>It depends on the note format to some extent:</p> <p>It depends on the note format to some extent:</p>
<h3 data-number="0.9.1" id="html-notes"><span class="header-section-number">0.9.1</span> HTML notes</h3> <h3 data-number="0.10.1" id="html-notes"><span class="header-section-number">0.10.1</span> HTML notes</h3>
<ul> <ul>
<li><p>It's very easy to hand-write bad HTML. That's why we use an HTML checker on it, and fix the errors it reports.</p></li> <li><p>It's very easy to hand-write bad HTML. That's why we use an HTML checker on it, and fix the errors it reports.</p></li>
<li><p>A common fault is to write <code>&lt;code&gt;&lt;pre&gt;</code> to define a pre-formatted block of text using a fixed-width font. However it's illegal because the <code>&lt;pre&gt;</code> tag can't be enclosed in other tags like <code>&lt;code&gt;</code>. Use <code>&lt;pre&gt;&lt;code&gt;</code> instead.</p></li> <li><p>A common fault is to write <code>&lt;code&gt;&lt;pre&gt;</code> to define a pre-formatted block of text using a fixed-width font. However it's illegal because the <code>&lt;pre&gt;</code> tag can't be enclosed in other tags like <code>&lt;code&gt;</code>. Use <code>&lt;pre&gt;&lt;code&gt;</code> instead.</p></li>
</ul> </ul>
<h3 data-number="0.9.2" id="markdown-notes"><span class="header-section-number">0.9.2</span> Markdown notes</h3> <h3 data-number="0.10.2" id="markdown-notes"><span class="header-section-number">0.10.2</span> Markdown notes</h3>
<ul> <ul>
<li>This is quite a forgiving format. Some people embed HTML in the Markdown, and the HTML used suffers from some of the issues discussed in this document. These are <em>much</em> harder to spot. The HTML checker can't really help unless it's run on the output from Pandoc.</li> <li>This is quite a forgiving format. Some people embed HTML in the Markdown, and the HTML used suffers from some of the issues discussed in this document. These are <em>much</em> harder to spot. The HTML checker can't really help unless it's run on the output from Pandoc.</li>
</ul> </ul>
<h3 data-number="0.9.3" id="general-errors"><span class="header-section-number">0.9.3</span> General errors</h3> <h3 data-number="0.10.3" id="general-errors"><span class="header-section-number">0.10.3</span> General errors</h3>
<ul> <ul>
<li><p>Spelling is often an issue in notes. As the notes are processed they are edited in the Vim editor with the spell check option turned on. In most cases spelling problems flagged by Vim are corrected.</p></li> <li><p>Spelling is often an issue in notes. As the notes are processed they are edited in the Vim editor with the spell check option turned on. In most cases spelling problems flagged by Vim are corrected.</p></li>
<li><p>Grammar and punctuation can also be issues. We do not have very effective grammar checking tools in the note preparation workflow. Modifications may be made to split up extra-long (often <em>comma-spliced</em>) sentences, rationalise the use of capital letters and similar.</p></li> <li><p>Grammar and punctuation can also be issues. We do not have very effective grammar checking tools in the note preparation workflow. Modifications may be made to split up extra-long (often <em>comma-spliced</em>) sentences, rationalise the use of capital letters and similar.</p></li>
</ul> </ul>
<h2 data-number="0.10" id="if-i-notice-an-error-in-my-shows-details-how-can-it-be-fixed"><span class="header-section-number">0.10</span> If I notice an error in my show's details how can it be fixed?</h2> <h2 data-number="0.11" id="if-i-notice-an-error-in-my-shows-details-how-can-it-be-fixed"><span class="header-section-number">0.11</span> If I notice an error in my show's details how can it be fixed?</h2>
<ul> <ul>
<li><p>The HPR administrators can make changes to show titles, summaries, notes and so forth. Ideally send an email to <code>admin</code> at <code>hackerpublicradio.org</code> explaining what the problem is and we'll fix it for you. We'll also ensure that the changes are propagated to the relevant page on archive.org.</p></li> <li><p>The HPR administrators can make changes to show titles, summaries, notes and so forth. Ideally send an email to <code>admin</code> at <code>hackerpublicradio.org</code> explaining what the problem is and we'll fix it for you. We'll also ensure that the changes are propagated to the relevant page on archive.org.</p></li>
<li><p>Don't be tempted to send in your corrections as a comment. Comments are not propagated to archive.org, so people referring to that copy will not see the changes.</p></li> <li><p>Don't be tempted to send in your corrections as a comment. Comments are not propagated to archive.org, so people referring to that copy will not see the changes.</p></li>
</ul> </ul>
<h2 data-number="0.11" id="does-hobbypublicradio.com-have-anything-to-do-with-hpr"><span class="header-section-number">0.11</span> Does <code>hobbypublicradio.com</code> have anything to do with HPR?</h2> <h2 data-number="0.12" id="does-hobbypublicradio.com-have-anything-to-do-with-hpr"><span class="header-section-number">0.12</span> Does <code>hobbypublicradio.com</code> have anything to do with HPR?</h2>
<ul> <ul>
<li><p>Some environments prevent access to sites which use the word <code>'hacker'</code> or <code>'hackers'</code> in their names (or content in extreme cases). The site names <code>hobbypublicradio.{com,net,org}</code> have been created, which link to the main HPR site, in order to get around this restriction.</p></li> <li><p>Some environments prevent access to sites which use the word <code>'hacker'</code> or <code>'hackers'</code> in their names (or content in extreme cases). The site names <code>hobbypublicradio.{com,net,org}</code> have been created, which link to the main HPR site, in order to get around this restriction.</p></li>
<li><p>See the mailing list thread here: <a href="http://hackerpublicradio.org/pipermail/hpr_hackerpublicradio.org/2015-February/013267.html" class="uri">http://hackerpublicradio.org/pipermail/hpr_hackerpublicradio.org/2015-February/013267.html</a></p></li> <li><p>See the mailing list thread here: <a href="http://hackerpublicradio.org/pipermail/hpr_hackerpublicradio.org/2015-February/013267.html" class="uri">http://hackerpublicradio.org/pipermail/hpr_hackerpublicradio.org/2015-February/013267.html</a></p></li>
</ul> </ul>
<h2 data-number="0.12" id="how-can-i-talk-to-other-hpr-hosts-and-listeners"><span class="header-section-number">0.12</span> How can I talk to other HPR hosts and listeners?</h2> <h2 data-number="0.13" id="how-can-i-talk-to-other-hpr-hosts-and-listeners"><span class="header-section-number">0.13</span> How can I talk to other HPR hosts and listeners?</h2>
<ul> <ul>
<li>You have a number of choices: <li>You have a number of choices:
<ul> <ul>
@ -174,8 +128,3 @@
<li>An HPR tradition exists where we record a 26-hour show at the turn of the year celebrating the arrival of the New Year around the world. We use Mumble to record the show and anyone can connect to the Mumble server and speak to other HPR listeners and hosts. The recording is used to make multiple shows which are released when they are ready.</li> <li>An HPR tradition exists where we record a 26-hour show at the turn of the year celebrating the arrival of the New Year around the world. We use Mumble to record the show and anyone can connect to the Mumble server and speak to other HPR listeners and hosts. The recording is used to make multiple shows which are released when they are ready.</li>
</ul></li> </ul></li>
</ul> </ul>
</article>
</main>
</div>
</body>
</html>

55
FAQ/FAQ.mkd
View File

@ -1,9 +1,11 @@
[%# FAQ.mkd 2020-09-26 14:26:49 version: 0.0.3 -%] [%# FAQ.mkd <2021-04-01 15:17:14> version: 0.0.5 -%]
[% USE date -%]
--- ---
title: Hacker Public Radio FAQ (draft) title: Hacker Public Radio FAQ (draft) [% date.format(date.now,"%F") %]
subtitle: A collection of questions and answers with links to the HPR site subtitle: A collection of questions and answers with links to the HPR site
author: HPR Contributors author: HPR Contributors
... ...
[%# {{{ Links %]
[% about = "http://hackerpublicradio.org/about.php" -%] [% about = "http://hackerpublicradio.org/about.php" -%]
[% reqslot = "http://hackerpublicradio.org/request_a_slot.php" -%] [% reqslot = "http://hackerpublicradio.org/request_a_slot.php" -%]
[% needtoknow = "http://hackerpublicradio.org/stuff_you_need_to_know.php" -%] [% needtoknow = "http://hackerpublicradio.org/stuff_you_need_to_know.php" -%]
@ -15,32 +17,34 @@ author: HPR Contributors
[% maillist = "http://hackerpublicradio.org/maillist" -%] [% maillist = "http://hackerpublicradio.org/maillist" -%]
[% freenode = "http://webchat.freenode.net/?channels=oggcastplanet" -%] [% freenode = "http://webchat.freenode.net/?channels=oggcastplanet" -%]
[% podrec = "http://hackerpublicradio.org/series.php?id=75" -%] [% podrec = "http://hackerpublicradio.org/series.php?id=75" -%]
[% tags = "http://hackerpublicradio.org/tags.php" -%]
[%# }}} %]
# Hacker Public Radio FAQ {.unnumbered} # Hacker Public Radio FAQ [% date.format(date.now,"%F") %] {.unnumbered}
## What is Hacker Public Radio? ## What is Hacker Public Radio?
- Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases - Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases
shows every weekday Monday through Friday. shows (episodes) every weekday Monday through Friday.
- What differentiates HPR from other podcasts is that the shows are crowd - What differentiates HPR from other podcasts is that the shows are crowd
sourced from the community &mdash; fellow listeners like you. There is no sourced from the community &mdash; fellow listeners like you. There is no
restriction on how long shows can be, nor on the topic you can cover as long restriction on how long shows can be, nor on the topic you can cover, as long
as they are not spam and "are of interest to Hackers". as they are not spam and "are of interest to Hackers".
If you want to see what topics have been covered so far just have a look at If you want to see what topics have been covered so far just have a look at
our [Archive]([% index %]). We also allow for a [series]([% series %]) of our [Archive]([% index %]). We also allow for shows to be grouped into
shows so that hosts can go into more detail on a topic. [series]([% series %]) so that hosts can go into more detail on a topic.
- Look at the [*About HPR*]([% about %]) page for a full description of *The HPR - Look at the [*About HPR*]([% about %]) page for a full description of *The HPR
Community*, *Free Culture*, *Governance* and our *Patrons*. Community*, *Free Culture*, *Governance* and our *Patrons*.
- A recent question on the mailing list asks if HPR is a Content Publication - A question on the mailing list in 2020 asked if HPR is a Content Publication
Network like YouTube, or is it a Podcast that random people contribute to? Or Network like YouTube, or is it a Podcast that random people contribute to? Or
to put it another way, is the host addressing the Internet in general or the to put it another way, is the host addressing the Internet in general or the
HPR Audience? HPR Audience?
- The consensus of opinion is that HPR is a Podcast contributed to by the - The consensus of opinion is that HPR is a Podcast contributed to by the
community. community, not a Content Publication Network.
## What is a show? ## What is a show?
@ -81,7 +85,7 @@ author: HPR Contributors
- As a contributor you can choose when your show will be released. We only - As a contributor you can choose when your show will be released. We only
release one show a day, during week days, but if a slot is free you can claim it. release one show a day, during week days, but if a slot is free you can claim it.
Go to the [*calendar*]([% calendar %]) page to do this. You need to have Go to the [*calendar*]([% calendar %]) page to do this. You need to have
your show ready to upload before you do this though. your show ready to upload before you reserve a slot though.
## Can I submit a show made for another podcast? ## Can I submit a show made for another podcast?
@ -115,13 +119,30 @@ author: HPR Contributors
- Tags - Tags
- Audio - Audio
## What are tags?
- We are trying to maintain a collection of tags to allow people to find shows
on the same (or similar) topic.
- Tags are simple words or short phrases describing aspects of a show such as
what topics are covered in it. To get hints about what tags have already
been used you can view all the current tags in alphabetical order on this
page: [Tag summary]([% tags %])
- Some examples are: <b>android</b>, <b>creative commons</b> and
<b>gaming</b>.
- Are there any tags that are best avoided?
## Is it OK if I don't include notes with my show? ## Is it OK if I don't include notes with my show?
- No, please don't do that. It is important to provide good show notes for - **Please don't do that**. It is important to provide good show notes for
reasons described on the [*Picking a slot for your show*]([% reqslot %]) reasons described on the [*Picking a slot for your show*]([% reqslot %])
page: page:
- They provide the listeners with more information on the topic - They provide the listeners with more information on the topic your show
- They ensure your show gets posted on time is covering
- They ensure your show gets posted on time (the volunteers don't have to
make notes for you)
- They encourage people to download your show - They encourage people to download your show
- They allow your show to be indexed by search engines - They allow your show to be indexed by search engines
- They make your show accessible to the deaf and hard of hearing - They make your show accessible to the deaf and hard of hearing
@ -135,8 +156,8 @@ author: HPR Contributors
- We don't listen to the audio before posting a show to the website, so we - We don't listen to the audio before posting a show to the website, so we
don't change the content. don't change the content.
- Unless you have done so and told us, we add an introduction to the audio - Unless you have done so and told us you have, we add an introduction to the
announcing what it is. Similarly we add an *outro*. This is not audio announcing what it is. Similarly we add an *outro*. This is not
<u>editing</u> since we don't change the content. <u>editing</u> since we don't change the content.
- We may make changes to the show title, summary or tags. Usually these are - We may make changes to the show title, summary or tags. Usually these are
@ -147,8 +168,8 @@ author: HPR Contributors
- Depending on the form the show notes take we may modify these. - Depending on the form the show notes take we may modify these.
- If what is submitted is plain text we convert it to Markdown in order to - If what is submitted is plain text we convert it to Markdown in order to
generate HTML. generate HTML.
- If the notes are one of the permitted markup formats (mostly Markdown) - If the notes are one of the permitted markup formats (mostly Markdown
we may adjust this to ensure that valid HTML is generated. flavours) we may adjust this to ensure that valid HTML is generated.
- If the notes are HTML we run them through an HTML checker and correct - If the notes are HTML we run them through an HTML checker and correct
any mistakes it highlights. any mistakes it highlights.
- We may correct spelling, grammar and punctuation if this clarifies - We may correct spelling, grammar and punctuation if this clarifies

51
FAQ/Makefile
View File

@ -1,31 +1,52 @@
# ============================================================================== # ==============================================================================
# FAQ Makefile 2021-01-30 11:00:47 # FAQ Makefile 2021-04-01 09:52:16
# ============================================================================== # ==============================================================================
# #
# Simple Makefile to rebuild the components in this file # Simple Makefile to rebuild the components in this file
# #
all : markdown
MDFILE = FAQ.mkd
HTMLFILE = FAQ.html
DEVFILE = FAQ_dev.html
TPLFILE = FAQ.tpl
PHPFILE = FAQ.php
all : markdown php
# #
# Find all *.mkd files in the current directory and turn them into a list of # Rules for converting the Markdown into two files, one for stitching into
# *.html files as a rule target # the PHP framework via FAQ.tpl and the other just a stand-alone HTML file for
# development.
# #
markdown: $(addsuffix .html,$(basename $(wildcard *.mkd))) markdown: $(HTMLFILE) $(DEVFILE)
$(HTMLFILE): $(MDFILE)
tpage $< | pandoc -f markdown-smart -t html5 \
--number-sections --table-of-contents --toc-depth=4 -o $@
$(DEVFILE): $(MDFILE)
tpage $< | pandoc -f markdown-smart -t html5 --standalone \
--template=hpr.html5 -c http://hackerpublicradio.org/css/hpr.css \
--number-sections --table-of-contents --toc-depth=4 -o $@
# #
# Rule to get from a Markdown (*.mkd) file to an HTML equivalent allowing for # When we generate the file FAQ.html (HTML fragment) we use tpage to make
# Template Toolkit stuff within the file # a complete standalone PHP page called FAQ.php
# #
%.html: %.mkd; tpage $< | pandoc -f markdown-smart -t html5 --standalone \ .PHONY: php upload
--template=hpr.html5 -c http://hackerpublicradio.org/css/hpr.css \
--number-sections --table-of-contents --toc-depth=4 -o $@ php: $(PHPFILE)
$(PHPFILE): $(HTMLFILE)
tpage $(TPLFILE) > $(PHPFILE)
# #
# Upload the FAQ HTML to the server for development # Upload the FAQ HTML+PHP to the server for development
# #
upload: upload: $(PHPFILE)
scp -P 22074 FAQ.html hpr@hackerpublicradio.org:www/ scp -P 22074 $(PHPFILE) hpr@hackerpublicradio.org:www/
# ==============================================================================
# #
# Use 'make sync' to copy updates to the visible project area # Use 'make sync' to copy updates to the visible project area
# #
@ -35,7 +56,7 @@ upload:
sync: put sync: put
fromdir = $(HOME)/HPR/FAQ/ fromdir = $(HOME)/HPR/FAQ/
todir = $(HOME)/HPR/Projects/hpr-admin/FAQ/ todir = $(HOME)/HPR/Projects/hpr-tools/FAQ/
filter = $(fromdir).rsync_export filter = $(fromdir).rsync_export
syncmsg = Updating git directory with updates from development version syncmsg = Updating git directory with updates from development version
@ -44,5 +65,5 @@ put:
@echo "** $(syncmsg)" @echo "** $(syncmsg)"
rsync -vaP --filter=". $(filter)" $(fromdir) $(todir) rsync -vaP --filter=". $(filter)" $(fromdir) $(todir)
.PHONY: upload sync put .PHONY: sync put

BIN
InternetArchive/ia.db
View File

Binary file not shown.

27
InternetArchive/repair_item
View File

@ -23,15 +23,15 @@
# BUGS: --- # BUGS: ---
# NOTES: --- # NOTES: ---
# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
# VERSION: 0.0.6 # VERSION: 0.0.7
# CREATED: 2020-01-05 22:42:46 # CREATED: 2020-01-05 22:42:46
# REVISION: 2024-05-10 12:39:52 # REVISION: 2024-05-17 20:49:47
# #
#=============================================================================== #===============================================================================
#set -o nounset # Treat unset variables as an error #set -o nounset # Treat unset variables as an error
VERSION="0.0.6" VERSION="0.0.7"
SCRIPT=${0##*/} SCRIPT=${0##*/}
# DIR=${0%/*} # DIR=${0%/*}
@ -190,8 +190,10 @@ Options:
mode and the actions will be carried out. mode and the actions will be carried out.
-D Run in debug mode where a lot more information is -D Run in debug mode where a lot more information is
reported reported
-l N Control the number of shows that can be uploaded at -l N Control the number of files that can be uploaded
once. The range is 1 to $DEFLIMIT. during one run of the script. The range is 1 to
$DEFLIMIT. This can be helpful when there are upload
problems.
Arguments: Arguments:
item The item in the form 'hpr1234' item The item in the form 'hpr1234'
@ -274,6 +276,16 @@ else
fi fi
_DEBUG "Parsed item: $item" _DEBUG "Parsed item: $item"
#
# It's possible that the show upload failed before anything was uploaded, even
# the metadata. It's never been seen, but it seems wise to cater for it.
#
if ! ia metadata "$item" --exists > /dev/null 2>&1; then
coloured 'red' "This item is not apparently on the IA; can't continue"
coloured 'yellow' "Try running the entire upload again from the start"
exit 1
fi
# #
# Declarations # Declarations
# #
@ -392,7 +404,7 @@ else
else else
retries=0 retries=0
printf 'Uploading %s\n' "$file" coloured 'blue' "Uploading $file"
# #
# Run 'cmd'. If it succeeds then write to the log and loop for the # Run 'cmd'. If it succeeds then write to the log and loop for the
@ -414,9 +426,12 @@ else
[ "$retries" -eq "$retry_threshold" ] && { [ "$retries" -eq "$retry_threshold" ] && {
((failures++)) ((failures++))
[[ $VERBOSE -eq 1 ]] && \
coloured 'blue' "Retry limit reached; abandoning this file"
continue 2 continue 2
} }
[[ $VERBOSE -eq 1 ]] && coloured 'blue' "Pausing for $sleeptime and retrying"
sleep $sleeptime sleep $sleeptime
done # until eval ... done # until eval ...

69
InternetArchive/update_state
View File

@ -25,9 +25,9 @@
# BUGS: --- # BUGS: ---
# NOTES: --- # NOTES: ---
# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
# VERSION: 0.0.9 # VERSION: 0.0.10
# CREATED: 2022-04-19 12:50:52 # CREATED: 2022-04-19 12:50:52
# REVISION: 2023-10-21 22:51:42 # REVISION: 2024-06-01 14:19:20
# #
#=============================================================================== #===============================================================================
@ -37,7 +37,7 @@ SCRIPT=${0##*/}
# DIR=${0%/*} # DIR=${0%/*}
# shellcheck disable=SC2034 # shellcheck disable=SC2034
VERSION="0.0.9" VERSION="0.0.10"
STDOUT="/dev/fd/2" STDOUT="/dev/fd/2"
@ -54,6 +54,8 @@ source "$LIB"
# #
define_colours define_colours
# {{{ ---- Functions: ---- _usage _DEBUG
#=== FUNCTION ================================================================ #=== FUNCTION ================================================================
# NAME: _usage # NAME: _usage
# DESCRIPTION: Report usage # DESCRIPTION: Report usage
@ -73,13 +75,17 @@ processed.
Options: Options:
-h Print this help -h Print this help
-D Enable DEBUG mode where a lot of information about the workins -D Enable DEBUG mode where a lot of information about the working
of the script is displayed of the script is displayed
-d Dry-run mode. Reports what it will do but doesn't do it -d Dry-run mode. Reports what it would do but doesn't do it
-F Force the update(s) without checking the state of the show on -F Force the update(s) without checking the state of the show on
the IA the IA
-l N Limit the number of shows processed to N -l N Limit the number of shows processed to N
-m Monochrome mode - no colours -m Monochrome mode - no colours
-R Normally, if a show is not in the IA, the script retries
waiting for it to be uploaded (assuming it's being worked on
by the IA servers). Including -R limits the retries to one
which is useful when uploading multiple shows one at a time.
Examples Examples
./${SCRIPT} -h ./${SCRIPT} -h
@ -87,7 +93,10 @@ Examples
./${SCRIPT} -d ./${SCRIPT} -d
./${SCRIPT} -dm ./${SCRIPT} -dm
./${SCRIPT} -Dd ./${SCRIPT} -Dd
./${SCRIPT} -F
./${SCRIPT} -l1 ./${SCRIPT} -l1
./${SCRIPT} -m
./${SCRIPT} -R
./${SCRIPT} ./${SCRIPT}
endusage endusage
@ -107,6 +116,8 @@ _DEBUG () {
done done
} }
# }}}
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# #
@ -114,10 +125,10 @@ _DEBUG () {
# #
case $HOSTNAME in case $HOSTNAME in
hprvps|marvin|borg) hprvps|marvin|borg)
# UPLOADS="/data/IA/uploads" # UPLOADS="/data/IA/uploads"
BASEDIR="$HOME/IA" ;; BASEDIR="$HOME/IA" ;;
i7-desktop) i7-desktop)
# UPLOADS="$HOME/HPR/IA/uploads" # UPLOADS="$HOME/HPR/IA/uploads"
BASEDIR="$HOME/HPR/IA" ;; BASEDIR="$HOME/HPR/IA" ;;
*) *)
echo "Wrong host!"; exit 1 ;; echo "Wrong host!"; exit 1 ;;
@ -149,16 +160,17 @@ RETRIES=3
# #
# Option defaults # Option defaults
# #
COLOUR=1 # use colours by default COLOUR=1 # use colours by default
DRYRUN=0 # live mode by default DRYRUN=0 # live mode by default
DEBUG=0 DEBUG=0
FORCE=0 FORCE=0
RETRYING=1 # retry if a show's not on the IA
DEFLIMIT=20 DEFLIMIT=20
# #
# Process options # Process options
# #
while getopts :hdDFl:m opt while getopts :hdDFl:mR opt
do do
case "${opt}" in case "${opt}" in
h) _usage;; h) _usage;;
@ -167,6 +179,7 @@ do
F) FORCE=1;; F) FORCE=1;;
l) LIMIT=$OPTARG;; l) LIMIT=$OPTARG;;
m) COLOUR=0;; m) COLOUR=0;;
R) RETRYING=0;;
?) echo "$SCRIPT: Invalid option; aborting"; exit 1;; ?) echo "$SCRIPT: Invalid option; aborting"; exit 1;;
esac esac
done done
@ -189,6 +202,10 @@ if [[ $FORCE -eq 1 ]]; then
coloured 'yellow' "Forcing updates without checking the IA state" coloured 'yellow' "Forcing updates without checking the IA state"
fi fi
if [[ $RETRYING -eq 0 ]]; then
coloured 'yellow' "Not retrying updates if the show is missing"
fi
# #
# Check the argument count after any options # Check the argument count after any options
# #
@ -220,7 +237,7 @@ _DEBUG "reservations = $reservations"
# "queue" from the variable 'reservations' which contains lines returned from # "queue" from the variable 'reservations' which contains lines returned from
# querying the CMS status interface. # querying the CMS status interface.
# #
count=0 showcount=0
while read -r line; do while read -r line; do
if [[ $line =~ ^([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),.*$ ]]; then if [[ $line =~ ^([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),.*$ ]]; then
state="${BASH_REMATCH[5]}" state="${BASH_REMATCH[5]}"
@ -232,7 +249,16 @@ while read -r line; do
if [[ $state = 'MEDIA_TRANSCODED' ]]; then if [[ $state = 'MEDIA_TRANSCODED' ]]; then
_DEBUG "show = $show, state = $state" _DEBUG "show = $show, state = $state"
retry_count=$RETRIES #
# If we're retrying (waiting for a show to be uploaded) then loop
# $RETRIES times, otherwise don't retry at all
#
if [[ $RETRYING -eq 1 ]]; then
retry_count=$RETRIES
else
retry_count=1
fi
while [ $retry_count -gt 0 ]; do while [ $retry_count -gt 0 ]; do
# #
# Look for the show on the IA. If not found we sleep 30 # Look for the show on the IA. If not found we sleep 30
@ -240,7 +266,8 @@ while read -r line; do
# times, controlled by $RETRIES, then we give up this show. If # times, controlled by $RETRIES, then we give up this show. If
# there are more shows then we keep going. # there are more shows then we keep going.
# #
if [ $FORCE -eq 1 ] || ia list "hpr$show" > /dev/null 2>&1; then if [ $FORCE -eq 1 ] || ia metadata "hpr$show" --exists > /dev/null 2>&1; then
# if [ $FORCE -eq 1 ] || ia list "hpr$show" > /dev/null 2>&1; then
command="${QUERY1}?ep_num=${show}&status=UPLOADED_TO_IA" command="${QUERY1}?ep_num=${show}&status=UPLOADED_TO_IA"
command_bak="${QUERY1_BAK}?ep_num=${show}&status=UPLOADED_TO_IA" command_bak="${QUERY1_BAK}?ep_num=${show}&status=UPLOADED_TO_IA"
@ -282,19 +309,23 @@ while read -r line; do
((retry_count--)) ((retry_count--))
done done
if [[ $retry_count -eq 0 ]]; then #
# Are all retries done, and are we retrying anyway?
#
if [[ $retry_count -eq 0 && $RETRYING -eq 1 ]]; then
coloured 'red' "Failed to update show $show; retry count reached" coloured 'red' "Failed to update show $show; retry count reached"
coloured 'yellow' "The command 'ia list hpr$show' repeatedly returned \"failure\"" coloured 'yellow' "The command 'ia list hpr$show' repeatedly returned \"failure\""
coloured 'yellow' "Database updates not done" coloured 'yellow' "Database updates not done"
coloured 'yellow' "Try again later with './update_state'" coloured 'yellow' "Try again later with './${SCRIPT}'"
fi fi
# #
# Stop the loop if we have reached the limiting number # Stop the loop if we have reached the limiting number
# #
((count++)) ((showcount++))
[[ $count -eq $LIMIT ]] && { [[ $showcount -eq $LIMIT ]] && {
echo "Upload limit ($LIMIT) reached" echo "Upload limit ($LIMIT) reached"
((--showcount))
break break
} }
@ -303,9 +334,9 @@ while read -r line; do
done <<< "$reservations" done <<< "$reservations"
if [[ $DRYRUN -eq 0 ]]; then if [[ $DRYRUN -eq 0 ]]; then
echo "Number of shows processed successfully: $count" echo "Number of shows processed successfully: $showcount"
fi fi
exit exit
# vim: syntax=sh:ts=8:sw=4:ai:et:tw=78:fo=tcrqn21
# vim: syntax=sh:ts=8:sw=4:ai:et:tw=78:fo=tcrqn21:fdm=marker

564
Link_Checker/scan_links
View File

@ -3,7 +3,7 @@
# #
# FILE: scan_links # FILE: scan_links
# #
# USAGE: ./scan_links # USAGE: ./scan_links [-help] [-[no]verbose] [-config=FILE]
# #
# DESCRIPTION: Scan the notes in the database for links. Test each link to # DESCRIPTION: Scan the notes in the database for links. Test each link to
# see if it's available. Keep a record of the date, show, link # see if it's available. Keep a record of the date, show, link
@ -18,9 +18,9 @@
# BUGS: --- # BUGS: ---
# NOTES: --- # NOTES: ---
# AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com # AUTHOR: Dave Morriss (djm), Dave.Morriss@gmail.com
# VERSION: 0.0.1 # VERSION: 0.0.2
# CREATED: 2017-04-02 14:09:02 # CREATED: 2017-04-02 14:09:02
# REVISION: 2017-04-02 14:51:19 # REVISION: 2022-06-02 23:18:13
# #
#=============================================================================== #===============================================================================
@ -28,12 +28,32 @@ use 5.010;
use strict; use strict;
use warnings; use warnings;
use utf8; use utf8;
use experimental 'smartmatch';
use Carp;
use Getopt::Long;
use Pod::Usage;
use Config::General;
use File::Slurper qw{ read_text read_lines };
use Try::Tiny;
use HTML::TreeBuilder 5 -weak;
use HTML::Entities;
use List::Util qw{ min max };
use List::MoreUtils qw{ any };
use LWP::Simple;
use DBI;
use SQL::Abstract;
use SQL::Abstract::Plugin::InsertMulti;
use Data::Dumper;
# #
# Version number (manually incremented) # Version number (manually incremented)
# #
our $VERSION = '0.0.1'; our $VERSION = '0.0.2';
# #
# Script and directory names # Script and directory names
@ -48,8 +68,12 @@ $DIR = '.' unless $DIR;
# #
# Constants and other declarations # Constants and other declarations
# #
my $basedir = "$ENV{HOME}"; my $basedir = "$ENV{HOME}/HPR/Link_Checker";
my $configfile = "$basedir/.$PROG.cfg";
my $db1configfile = "$basedir/.hpr_db.cfg";
my $database2 = "$basedir/ia.db";
my ( $dbh1, $dbh2, $sql1, $sth1, $h1 );
# #
# Enable Unicode mode # Enable Unicode mode
@ -57,8 +81,538 @@ my $basedir = "$ENV{HOME}";
binmode STDOUT, ":encoding(UTF-8)"; binmode STDOUT, ":encoding(UTF-8)";
binmode STDERR, ":encoding(UTF-8)"; binmode STDERR, ":encoding(UTF-8)";
#-------------------------------------------------------------------------------
# Options and arguments
#-------------------------------------------------------------------------------
#
# Option defaults
#
my $DEFDEBUG = 0;
my $DEFFROM = 1;
my $DEFCOUNT = 10;
#
# Process options
#
my %options;
Options( \%options );
#
# Default help is minimal
#
pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1 )
if ( $options{'help'} );
#
# The -documentation or -man option shows the full POD documentation through
# a pager for convenience
#
pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1, -verbose => 2 )
if ( $options{'documentation'} );
#
# Collect options
#
my $DEBUG = ( defined( $options{debug} ) ? $options{debug} : $DEFDEBUG );
my $db1cfgfile
= ( defined( $options{dbconfig} ) ? $options{dbconfig} : $db1configfile );
my $cfgfile
= ( defined( $options{config} ) ? $options{config} : $configfile );
my $dry_run = ( defined( $options{'dry-run'} ) ? $options{'dry-run'} : 0 );
my $verbose = ( defined( $options{verbose} ) ? $options{verbose} : 0 );
my $from = ( defined( $options{from} ) ? $options{from} : $DEFFROM );
my $count = ( defined( $options{count} ) ? $options{count} : $DEFCOUNT );
$from = $DEFFROM if $from < 1;
my @episodes = ( ( $from .. $from + $count ) );
#
# Report on the options in debug mode
#
if ($DEBUG > 1) {
_debug(1,'$DEBUG = ' . $DEBUG);
_debug(1,'$dry-run = ' . $dry_run);
_debug(1,'$verbose = ' . $verbose);
_debug(1,'$db1cfgfile = ' . $db1cfgfile);
_debug(1,'$cfgfile = ' . $cfgfile);
_debug(1,'$from = ' . $from);
_debug(1,'$count = ' . $count);
_debug(1,'$#episodes = ' . $#episodes);
}
#
# Sanity checks
#
die "Unable to find $cfgfile\n" unless ( -e $cfgfile );
die "Unable to find $db1cfgfile\n" unless ( -e $db1cfgfile );
#-------------------------------------------------------------------------------
# Configuration file - load data
#-------------------------------------------------------------------------------
my $conf = Config::General->new(
-ConfigFile => $cfgfile,
-InterPolateVars => 1,
-ExtendedAccess => 1,
);
my %config = $conf->getall();
#-------------------------------------------------------------------------------
# Connect to the database
#-------------------------------------------------------------------------------
my $db1conf = Config::General->new(
-ConfigFile => $db1cfgfile,
-InterPolateVars => 1,
-ExtendedAccess => 1,
);
my %db1cfg = $db1conf->getall();
#-------------------------------------------------------------------------------
# Database configuration file - load data
#-------------------------------------------------------------------------------
my $dbhost = $db1cfg{database}->{host} // '127.0.0.1';
my $dbport = $db1cfg{database}->{port} // 3306;
my $dbname = $db1cfg{database}->{name};
my $dbuser = $db1cfg{database}->{user};
my $dbpwd = $db1cfg{database}->{password};
$dbh1 = DBI->connect( "dbi:mysql:host=$dbhost;port=$dbport;database=$dbname",
$dbuser, $dbpwd, { AutoCommit => 1, RaiseError => 1 } )
or die $DBI::errstr;
#
# Enable client-side UTF8
#
$dbh1->{mysql_enable_utf8} = 1;
#
# Set the local timezone to UTC for this connection
#
$dbh1->do("set time_zone = '+00:00'") or carp $dbh1->errstr;
emit( $verbose >= 2, "Opened MySQL database\n" );
#-------------------------------------------------------------------------------
# Connect to the SQLite database
#-------------------------------------------------------------------------------
$dbh2 = DBI->connect( "dbi:SQLite:dbname=$database2", "", "" )
or die $DBI::errstr;
$dbh2->do("PRAGMA foreign_keys = ON") or die $DBI::errstr;
emit( $verbose >= 2, "Opened SQLite database\n" );
#
# The main MySQL query
#
$sql1 = q{
SELECT * FROM eps WHERE id BETWEEN ? AND ?
};
$sth1 = $dbh1->prepare($sql1);
$sth1->execute( $from, $from + $count );
if ( $dbh1->err ) {
die $dbh1->errstr;
}
while ( $h1 = $sth1->fetchrow_hashref ) {
printf "%04d %s %s\n", $h1->{id},$h1->{date},$h1->{title};
}
exit; exit;
#emit( $verbose >= 2, "Configuration file: ", $cfgfile, "\n" );
#emit( $verbose >= 2, "DB configuration file: ", $db1cfgfile, "\n" );
#=== FUNCTION ================================================================
# NAME: find_external_links
# PURPOSE: Parses the HTML in a string for links so that attached assets
# on the HPR site can also be parsed and to collect external
# links for testing.
# PARAMETERS: $episode episode number we're dealing with
# $html string containing HTML
# $rlinks hashref to receive the links found
# RETURNS: Number of links found
# DESCRIPTION: Given HTML from the main notes or a subsidiary file the
# function parses this looking for links in 'a' or 'img' tags.
# Links are standardised, making them absolute if relative and
# removing any 'fragment'. The links need to be to HTML files on
# the HPR website to be of use in recursing to subsidiary
# levels. Otherwise they need to be external links that we will
# test.
# Having found a local link the filename part is extracted. If
# it follows the format 'hpr9999' then it's checked to see if
# it's for the current show. If not it's ignored. If the
# filename ends with a '/' then it's assumed it's shorthand for
# 'index.html' so this name is appended. If the local filename
# ends with '.html' then we need to parse it in turn, so we get
# the contents of the link and recurse to parse it.
# Then, if external, the link and filename are stashed in the
# hash referenced by $rlinks. We return the number of external
# links found in the pass through the HTML.
# THROWS: No exceptions
# COMMENTS: Based on 'find_links' in 'upload_manager'.
# SEE ALSO: N/A
#===============================================================================
sub find_external_links {
my ( $episode, $html, $rlinks ) = @_;
my ($tree, $epstr, $linkre, $re2, $filepath,
$uri, $slink, $linkcount, $content
);
_debug( $DEBUG >= 3, "find_external_links enter" );
#
# Create a tree object
#
$tree = HTML::TreeBuilder->new;
$tree->ignore_unknown(0);
$tree->no_expand_entities(1);
$tree->p_strict(1);
$tree->store_comments(1);
$tree->warn(1);
$tree->parse_content($html)
or die "HTML::TreeBuilder failed to parse notes: $!\n";
my $baseURL = "http://hackerpublicradio.org";
$epstr = sprintf( "hpr%04d", $episode );
$linkre = qr{
^https?://
(?:www.)?
(?:hacker|hobby)publicradio.org/
(.+)$
}x;
#
# Counting new links found and stashed
#
$linkcount = 0;
#
# Scan for links
#
for ( @{ $tree->extract_links( 'a', 'img' ) } ) {
my ( $link, $element, $attr, $tag ) = @$_;
#
# Standardise the link (expands relative URLs, removes any fragment).
# Set $URI::ABS_REMOTE_LEADING_DOTS to ensure leading dots in relative
# URIs are removed.
#
local $URI::ABS_REMOTE_LEADING_DOTS = 1;
$uri = URI->new_abs( $link, $baseURL );
$slink = sprintf( "%s:%s", $uri->scheme, $uri->opaque );
#
# Is it an HPR link?
#
if ( $slink =~ $linkre ) {
#
# The URL we found might be a link into an HTML file with an
# '#anchor' component ("fragment"). Save the last bracketed match,
# without any 'fragment' if there is one to get a clean filename
# or path.
#
( $filepath = "$+" ) =~ s/#.*$//;
_debug( $DEBUG >= 3, "Link: $slink\n" );
_debug( $DEBUG >= 3, "File path: $filepath\n" );
#
# Does this file path begin with an 'hpr' prefix? If so is it the
# show id? If not we don't want to process it.
#
if ( $filepath =~ /^(hpr[0-9]{1,4})/ ) {
if ( $1 ne $epstr ) {
_debug( $DEBUG >= 3, "Ignored $slink\n" );
next;
}
}
#
# The path and URL might end with a slash which means the URL is
# relying on the Web server to fill in the filename as
# 'index.html'. We have to make this explicit.
#
if ( $slink =~ /\/$/ ) {
$slink .= 'index.html';
$filepath .= 'index.html';
}
#
# Initialise this hash element if needed
#
unless ( exists( $rlinks->{$episode} ) ) {
$rlinks->{$episode} = [];
}
#
# Stash this filename if it's not already stashed, and if it's
# HTML get the link and recurse
#
unless (
any { $_->{filename} eq $filepath }
@{ $rlinks->{$episode} }
)
{
_debug( $DEBUG >= 3, "Stashed $slink and $filepath\n" );
push(
@{ $rlinks->{$episode} },
{ filename => $filepath, URL => $slink }
);
$linkcount++;
#
# An HTML file has to be investigated
#
if ( $filepath =~ /\.html$/ ) {
$content = get($slink);
unless ( defined($content) ) {
carp "Link $slink returned nothing\n";
}
else {
$linkcount
+= find_links( $episode, $content, $rlinks );
}
}
}
}
#
# It's not an HPR URL
#
else {
#
# Stash this filename if it's not already stashed
#
unless (
any { $_->{filename} eq $filepath }
@{ $rlinks->{$episode} }
)
{
_debug( $DEBUG >= 3, "Stashed $slink and $filepath\n" );
push(
@{ $rlinks->{$episode} },
{ filename => $filepath, URL => $slink }
);
$linkcount++;
}
}
}
_debug( $DEBUG >= 3, "find_links exiting with $linkcount links\n" );
#
# Return the link count
#
return $linkcount;
}
#=== FUNCTION ================================================================
# NAME: emit
# PURPOSE: Print text on STDERR unless silent mode has been selected
# PARAMETERS: - Boolean indicating whether to print or not
# - list of arguments to 'print'
# RETURNS: Nothing
# DESCRIPTION: This is a wrapper around 'print' to determine whether to send
# a message to STDERR depending on a boolean. We need this to be
# able to make the script silent when the -verbose option is
# not selected
# THROWS: No exceptions
# COMMENTS: None
# SEE ALSO: N/A
#===============================================================================
sub emit {
if (shift) {
print STDERR @_;
}
}
#=== FUNCTION ================================================================
# NAME: _debug
# PURPOSE: Prints debug reports
# PARAMETERS: $active Boolean: 1 for print, 0 for no print
# $message Message to print
# RETURNS: Nothing
# DESCRIPTION: Outputs a message if $active is true. It removes any trailing
# newline and then adds one in the 'print' to the caller doesn't
# have to bother. Prepends the message with 'D> ' to show it's
# a debug message.
# THROWS: No exceptions
# COMMENTS: None
# SEE ALSO: N/A
#===============================================================================
sub _debug {
my ( $active, $message ) = @_;
chomp($message);
print STDERR "D> $message\n" if $active;
}
#=== FUNCTION ================================================================
# NAME: Options
# PURPOSE: Processes command-line options
# PARAMETERS: $optref Hash reference to hold the options
# RETURNS: Undef
# DESCRIPTION:
# THROWS: no exceptions
# COMMENTS: none
# SEE ALSO: n/a
#===============================================================================
sub Options {
my ($optref) = @_;
my @options = (
"help", "documentation|man", "debug=i", "dry-run!",
"verbose+", "dbconfig=s", "from=s", "count=i",
);
if ( !GetOptions( $optref, @options ) ) {
pod2usage( -msg => "$PROG version $VERSION\n", -exitval => 1,
-verbose => 0 );
}
return;
}
__END__
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
# Application Documentation
#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#{{{
=head1 NAME
<application name> - <One line description of application's purpose>
=head1 VERSION
The initial template usually just has:
This documentation refers to <application name> version 0.0.2
=head1 USAGE
# Brief working invocation example(s) here showing the most common usage(s)
# This section will be as far as many users ever read
# so make it as educational and exemplary as possible.
=head1 REQUIRED ARGUMENTS
A complete list of every argument that must appear on the command line.
when the application is invoked, explaining what each of them does, any
restrictions on where each one may appear (i.e. flags that must appear
before or after filenames), and how the various arguments and options
may interact (e.g. mutual exclusions, required combinations, etc.)
If all of the application's arguments are optional this section
may be omitted entirely.
=head1 OPTIONS
A complete list of every available option with which the application
can be invoked, explaining what each does, and listing any restrictions,
or interactions.
If the application has no options this section may be omitted entirely.
=head1 DESCRIPTION
A full description of the application and its features.
May include numerous subsections (i.e. =head2, =head3, etc.)
=head1 DIAGNOSTICS
A list of every error and warning message that the application can generate
(even the ones that will "never happen"), with a full explanation of each
problem, one or more likely causes, and any suggested remedies. If the
application generates exit status codes (e.g. under Unix) then list the exit
status associated with each error.
=head1 CONFIGURATION AND ENVIRONMENT
A full explanation of any configuration system(s) used by the application,
including the names and locations of any configuration files, and the
meaning of any environment variables or properties that can be set. These
descriptions must also include details of any configuration language used
=head1 DEPENDENCIES
A list of all the other modules that this module relies upon, including any
restrictions on versions, and an indication whether these required modules are
part of the standard Perl distribution, part of the module's distribution,
or must be installed separately.
=head1 INCOMPATIBILITIES
A list of any modules that this module cannot be used in conjunction with.
This may be due to name conflicts in the interface, or competition for
system or program resources, or due to internal limitations of Perl
(for example, many modules that use source code filters are mutually
incompatible).
=head1 BUGS AND LIMITATIONS
A list of known problems with the module, together with some indication
whether they are likely to be fixed in an upcoming release.
Also a list of restrictions on the features the module does provide:
data types that cannot be handled, performance issues and the circumstances
in which they may arise, practical limitations on the size of data sets,
special cases that are not (yet) handled, etc.
The initial template usually just has:
There are no known bugs in this module.
Please report problems to <Maintainer name(s)> (<contact address>)
Patches are welcome.
=head1 AUTHOR
<Author name(s)> (<contact address>)
=head1 LICENCE AND COPYRIGHT
Copyright (c) <year> <copyright holder> (<contact address>). All rights reserved.
Followed by whatever licence you wish to release it under.
For Perl code that is often just:
This module is free software; you can redistribute it and/or
modify it under the same terms as Perl itself. See perldoc perlartistic.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
=cut
#}}}
# [zo to open fold, zc to close or za to toggle]
# vim: syntax=perl:ts=8:sw=4:et:ai:tw=78:fo=tcrqn21:fdm=marker # vim: syntax=perl:ts=8:sw=4:et:ai:tw=78:fo=tcrqn21:fdm=marker

1051
Miscellaneous/fix_tags Executable file
View File

File diff suppressed because it is too large Load Diff

BIN
Miscellaneous/fix_tags.bin Executable file
View File

Binary file not shown.

2
PostgreSQL_Database/add_hosts_to_show
View File

@ -535,7 +535,7 @@ sub find_hosts {
# DESCRIPTION: Places all the hosts associated with the show into an array # DESCRIPTION: Places all the hosts associated with the show into an array
# then compares it with the @$hosts array such that what is left # then compares it with the @$hosts array such that what is left
# is all the hosts that do not match hosts linked to the show. # is all the hosts that do not match hosts linked to the show.
# Of cxourse, if any of the hosts given as options are not exact # Of course, if any of the hosts given as options are not exact
# matches with the names from the database (they are regexes # matches with the names from the database (they are regexes
# perhaps) they'll be regarded as different. We have to do # perhaps) they'll be regarded as different. We have to do
# further processing of what's returned from querying the # further processing of what's returned from querying the

82
PostgreSQL_Database/hpr_schema_2.pgsql
View File

@ -22,6 +22,19 @@
* 'episodes_id' seems stupid. * 'episodes_id' seems stupid.
* *
* ------------------------------------------------------------------------------ * ------------------------------------------------------------------------------
* Tables:
*
* comments
* episodes
* episodes_hosts_xref
* episodes_series_xref
* episodes_tags_xref
* hosts
* licenses
* series
* tags
* assets
*
*/ */
/* ------------------------------------------------------------------------------ /* ------------------------------------------------------------------------------
@ -63,13 +76,15 @@ DROP FUNCTION IF EXISTS comment_changed();
-- }}} -- }}}
-- \/\/ licenses \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ -- +----------+
-- | licenses |
-- +----------+
-- {{{ -- {{{
/* ------------------------------------------------------------------------------ /* ------------------------------------------------------------------------------
* Table 'licenses' - licenses relating to episodes (needed because 'episodes' * Table 'licenses' - licenses relating to episodes (needed because 'episodes'
* and hosts' reference it) * and 'hosts' reference it)
* *
* license_id primary key, the licence number * license_id primary key, the licence number
* short_name brief name of CC licence * short_name brief name of CC licence
@ -123,7 +138,9 @@ ALTER FUNCTION id_in_licenses(sname varchar)
-- }}} -- }}}
-- \/\/ episodes \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ -- +----------+
-- | episodes |
-- +----------+
-- {{{ -- {{{
@ -203,6 +220,65 @@ ALTER FUNCTION id_in_episodes(ekey varchar)
-- }}} -- }}}
-- +-----------+
-- | episodes2 |
-- +-----------+
-- {{{
/* ------------------------------------------------------------------------------
* Table 'episodes2' - TWAT and HPR shows
*
* show_id zero for a TwaT show, 1 for an HPR show
* episode_id episode number in the range of show_id values
* release_date date the episode was released
* title title of the episode
* summary summary of the episode content
* notes the show notes (as an HTML fragment)
* explicit a Boolean; true for explicit, false for otherwise
* license the licence which the show is under (US spelling)
* duration the duration (time) of the audio
* downloads number of downloads
* archived a Boolean; true if the episode has been uploaded to the IA
* archive_date date the episode was archived
* IA_URL URL to the episode on archive.org
* journal a journal of actions performed on this episode
*
* ------------------------------------------------------------------------------ */
-- CREATE TYPE episode_status AS ENUM ('reserved', 'processing', 'posted');
DROP TABLE IF EXISTS episodes2 CASCADE;
CREATE TABLE episodes2 (
show_id smallint NOT NULL DEFAULT 1
CHECK (show_id = 0 OR show_id = 1),
episode_id smallint NOT NULL,
release_date date NOT NULL,
title varchar(128) NOT NULL,
summary varchar(128),
notes text NOT NULL,
explicit boolean NOT NULL DEFAULT TRUE,
license integer NOT NULL DEFAULT id_in_licenses('CC-BY-SA')
REFERENCES licenses (license_id),
duration interval NOT NULL DEFAULT '00:00:00',
downloads integer NOT NULL DEFAULT 0,
archived boolean NOT NULL DEFAULT FALSE,
archive_date date,
IA_URL text,
status episode_status,
journal text,
PRIMARY KEY (show_id, episode_id)
);
ALTER TABLE episodes2
OWNER TO hpradmin;
CREATE INDEX episode2_release_date_idx
ON episodes2
USING btree
(release_date);
-- }}}
-- \/\/ hosts /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ -- \/\/ hosts /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
-- {{{ -- {{{

10
PostgreSQL_Database/nuke_n_pave.sh
View File

@ -6,6 +6,16 @@
# hosts by adding the missing hosts. # hosts by adding the missing hosts.
# #
#
# Check that the ensuing destruction is really meant!
#
echo "** Warning** This script will DELETE the PostgreSQL database 'HPR2'"
read -r -p "Are you sure? [y/N] " ans
if [[ -z $ans || ${ans^^} = 'N' ]]; then
echo "Aborting"
exit
fi
# #
# Directories # Directories
# #