Move under www to ease rsync
This commit is contained in:
159
www/eps/hpr3551/hpr3551_full_shownotes.html
Executable file
159
www/eps/hpr3551/hpr3551_full_shownotes.html
Executable file
@@ -0,0 +1,159 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="generator" content="pandoc">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
|
||||
<meta name="author" content="Dave Morriss">
|
||||
<title>Bash snippet - some possibly helpful hints (HPR Show 3551)</title>
|
||||
<style type="text/css">code{white-space: pre;}</style>
|
||||
<!--[if lt IE 9]>
|
||||
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
|
||||
<![endif]-->
|
||||
<link rel="stylesheet" href="https://hackerpublicradio.org/css/hpr.css">
|
||||
</head>
|
||||
|
||||
<body id="home">
|
||||
<div id="container" class="shadow">
|
||||
<header>
|
||||
<h1 class="title">Bash snippet - some possibly helpful hints (HPR Show 3551)</h1>
|
||||
<h2 class="subtitle">Using ‘eval’, ‘mapfile’ and environment variables</h2>
|
||||
<h2 class="author">Dave Morriss</h2>
|
||||
<hr/>
|
||||
</header>
|
||||
|
||||
<main id="maincontent">
|
||||
<article>
|
||||
<header>
|
||||
<h1>Table of Contents</h1>
|
||||
<nav id="TOC">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#tasks">Tasks</a>
|
||||
<ul>
|
||||
<li><a href="#generating-bash-variables">Generating Bash variables</a></li>
|
||||
<li><a href="#filling-a-bash-array">Filling a Bash array</a></li>
|
||||
<li><a href="#turning-debugging-on">Turning debugging on</a></li>
|
||||
</ul></li>
|
||||
<li><a href="#conclusion">Conclusion</a></li>
|
||||
<li><a href="#links">Links</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
</header>
|
||||
<h2 id="overview">Overview</h2>
|
||||
<p>I write a moderate number of Bash scripts these days. Bash is not a programming language as such, but it’s quite powerful in what it can do by itself, and with other tools it’s capable of many things.</p>
|
||||
<p>I have enjoyed writing such scripts for many years on a variety of hardware and operating systems, and Bash is my favourite - partly because Linux itself is so flexible.</p>
|
||||
<p>This is just a short show describing three things I tend to do in Bash scripts to assist with some tasks I find I need to undertake.</p>
|
||||
<ol type="1">
|
||||
<li>Generate Bash variables from a text file - usually output from a program</li>
|
||||
<li>Fill Bash arrays with data from a file or other source</li>
|
||||
<li>Use environment variables to control the Bash script’s execution</li>
|
||||
</ol>
|
||||
<h2 id="tasks">Tasks</h2>
|
||||
<h3 id="generating-bash-variables">Generating Bash variables</h3>
|
||||
<p>There’s a Bash command <code>'eval'</code> that can be used to evaluate a string as a command (or series of commands). The evaluation takes place in the current shell, so anything returned - Bash variables in this case - is available to the current script.</p>
|
||||
<p>This is different from setting variables in a sub-shell (child process). This is because such variables are local to the subshell, and disappear when it finishes.</p>
|
||||
<p>The <code>eval</code> command takes a list of arguments which are concatenated into a string and the resulting string is evaluated.</p>
|
||||
<p>The <code>eval</code> command is seen as potentially dangerous in that it will execute any command it is given. Thus scripts should take precautions that the command (or commands) are predictable. Do not write a Bash script that executes whatever is given to it!</p>
|
||||
<p>One particular case I use <code>eval</code> for is to set variables from a text file. The file is generated from the HPR show upload process and I want to grab the title, summary and host name so I can generate an index file for any supplementary files uploaded with a show.</p>
|
||||
<p>The file contains text like:</p>
|
||||
<pre><code>Host_Name: Dave Morriss
|
||||
Title: Battling with English - part 4
|
||||
Summary: Some confusion with English plurals; strange language changes</code></pre>
|
||||
<p>In my script the file name is in a variable <code>RAWFILE</code>, and I run the following command:</p>
|
||||
<pre><code>eval "$(sed -n "/^\(Title\|Summary\|Host_Name\):/{s/^\([^:]\+\):\t/\1='/;s/$/'/;p}" "$RAWFILE")"</code></pre>
|
||||
<p>Taking the <code>sed</code> command by itself, we get:</p>
|
||||
<pre><code>$ sed -n "/^\(Title\|Summary\|Host_Name\):/{s/^\([^:]\+\):\t/\1='/;s/$/'/;p}" "$RAWFILE"
|
||||
Host_Name='Dave Morriss'
|
||||
Title='Battling with English - part 4'
|
||||
Summary='Some confusion with English plurals; strange language changes'</code></pre>
|
||||
<p>The <code>sed</code> commands find any line beginning with one of the three keywords and generate output consisting of that keyword, an equals sign and rest of the line. Thus the matched lines are turned into <code>VAR='string'</code> sequences.</p>
|
||||
<p>So, <code>eval</code> executes these and sets the relevant variables, which the script can access.</p>
|
||||
<p>This method is not foolproof. If a string contains a single quote the <code>eval</code> will fail. For the moment I haven’t guarded against this.</p>
|
||||
<h3 id="filling-a-bash-array">Filling a Bash array</h3>
|
||||
<p>I have a need to fill a Bash indexed array with sorted filenames in a script that deals with pictures sent in with HPR shows.</p>
|
||||
<p>I want to use the <code>find</code> command to find the pictures, searching for files which end in <code>jpg</code>, <code>JPG</code>, <code>png</code> or <code>PNG</code>. I don’t want to visit sub-directories. The command I want is:</p>
|
||||
<pre><code>find "$SHOWDIR" -maxdepth 1 -regextype egrep -regex '.*\.(jpg|JPG|png|PNG)'</code></pre>
|
||||
<p>The variable <code>SHOWDIR</code> contains the directory holding the files uploaded for the show.</p>
|
||||
<p>Given that I want to sort these files alphabetically, I can populate an array by this method:</p>
|
||||
<pre><code>declare -a pix
|
||||
pix=( $(find "$SHOWDIR" -maxdepth 1 -regextype egrep -regex '.*\.(jpg|JPG|png|PNG)' | sort) )</code></pre>
|
||||
<p>The output from the <code>find</code> and <code>sort</code> commands in the command substitution expression will consist of a number of newline separated lines, but Bash will replace the newlines by spaces, so the array-defining parenthesised list will consist of space-delimited filenames, which will be placed in the array.</p>
|
||||
<p>However, what if a file name contains a space? It’s bad practice, but it’s permitted, so it might happen.</p>
|
||||
<p>This is where the <code>mapfile</code> command might help. This was introduced in episode <a href="https://hackerpublicradio.org/eps.php?id=2739" title="Bash Tips - 19; Arrays in Bash (part 4)">2739</a> where its options were described. Typing <code>help mapfile</code> in a terminal will show a similar description.</p>
|
||||
<pre><code>declare -a pix
|
||||
mapfile -t pix < \
|
||||
<(find "$SHOWDIR" -maxdepth 1 -regextype egrep -regex '.*\.(jpg|JPG|png|PNG)' | sort)</code></pre>
|
||||
<p>We use a process substitution here which preserves newlines. One of the features of <code>mapfile</code> that is useful in this context is the <code>-t</code> option which removes the default delimiter, a newline. The delimiter can be changed with the <code>-d DELIM</code> option. The text between delimiters is what is written to the array elements, so as long as there are no filenames with newlines in them this will be better than the previous method.</p>
|
||||
<p>To be 100% safe the <code>find</code> command should use the <code>-print0</code> option which uses a <em>null</em> character instead of the default newline, and <code>mapfile</code> should be changed to this delimiter. We also need to tell <code>sort</code> to use <em>null</em> as a line delimiter which is done by adding the <code>-z</code> option<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>.</p>
|
||||
<pre><code>declare -a pix
|
||||
mapfile -d '' -t pix < \
|
||||
<(find "$SHOWDIR" -maxdepth 1 -regextype egrep -regex '.*\.(jpg|JPG|png|PNG)' -print0 | sort -z)</code></pre>
|
||||
<p>What it doesn’t tell you in <code>'help mapfile'</code> is that an empty string as a delimiter (<code>-d ''</code>) causes <code>mapfile</code> to use a <em>null</em> delimiter. There doesn’t seem to be any other way to do this - or nothing I could find anyway. You can read this information in the <code>Bash</code> manpage.</p>
|
||||
<p>Having discovered this information while preparing this show I shall certainly update my script to use it!</p>
|
||||
<h3 id="turning-debugging-on">Turning debugging on</h3>
|
||||
<p>I find I need to add debugging statements to the more complicated scripts I write, and to help with this I usually define a fairly simple function to do it. Here’s what I often use:</p>
|
||||
<pre><code>_DEBUG () {
|
||||
[ "$DEBUG" == 0 ] && return
|
||||
for msg in "$@"; do
|
||||
printf 'D> %s\n' "$msg"
|
||||
done
|
||||
}</code></pre>
|
||||
<p>This uses a global variable <code>'DEBUG'</code> and exits without doing anything if the variable contains zero. If it is non-zero the function prints its arguments preceded by <code>'D> '</code> to show it is debug output.</p>
|
||||
<p>I add calls to this function throughout my script if I want to check that values are what I expect them to be.</p>
|
||||
<p>The issue is how to turn debugging mode on and off. There are several ways, from the simplest (least elegant) to the most complicated in terms of coding.</p>
|
||||
<ol type="1">
|
||||
<li>Edit the script to set the <code>DEBUG</code> variable to 1 or zero</li>
|
||||
<li>Set it through an external variable visible to the script</li>
|
||||
<li>Add option processing to the script and use an option to enable or disable debug mode</li>
|
||||
</ol>
|
||||
<p>I use tend to choice 3 when I’m already dealing with options, but if not then I use choice 2. This is the one I’ll explain now.</p>
|
||||
<p>I use Vim as my editor, and in Vim I use a plugin (<code>'BashSupport'</code>) with which can define boilerplate text to be added to scripts. I have configured this to generate various definitions and declarations whenever I create a new Bash script. One of the lines I add to all of my scripts is:</p>
|
||||
<pre><code>SCRIPT=${0##*/}</code></pre>
|
||||
<p>This takes the default variable <code>$0</code> and strips off everything up to the last <code>'/'</code> character, thus leaving just the name of the script. I talked about these capabilities in show <a href="https://hackerpublicradio.org/eps.php?id=1648" title="Bash parameter manipulation">1648</a>.</p>
|
||||
<p>I have recently started adding the following lines:</p>
|
||||
<pre><code>DEBUGVAR="${SCRIPT}_DEBUG"
|
||||
DEBUG="${!DEBUGVAR:-0}"</code></pre>
|
||||
<p>This defines a variable <code>'DEBUGVAR'</code> which contains the name of the script concatenated with <code>'_DEBUG'</code>. Then, assuming the script name is <code>testscript</code>, the <code>'DEBUG'</code> variable is defined to contain the contents of a variable called <code>'testscript_DEBUG'</code>. The exclamation mark (<code>'!'</code>) in front of the variable name causes Bash to use its <u>contents</u>, which is a form of <em>indirection</em>. If the indirected variable is not found a default of zero is set.</p>
|
||||
<p>This means that debugging can be turned on by calling the script thus:</p>
|
||||
<pre><code>testscript_DEBUG=1 ./testscript</code></pre>
|
||||
<p>Variables set from the command line are visible to scripts. One set <u>on</u> the command line only lasts while the script (or command) is executing.</p>
|
||||
<p>You could set it as an exported (environment) variable:</p>
|
||||
<pre><code>export testscript_DEBUG=1
|
||||
./testscript</code></pre>
|
||||
<p>and then it would continue after the script had run. I prefer not to do this.</p>
|
||||
<p>I name my debug variables as I do so that there’s less chance of them affecting scripts other than the one I’m currently debugging!</p>
|
||||
<h2 id="conclusion">Conclusion</h2>
|
||||
<p>These are just three things I have found myself using in recent Bash scripts, which I hope might prove to be useful to you.</p>
|
||||
<p>If you have hints like this which you could share, please make an HPR show about them. We are always in need of shows, and at the time of writing (2022-02-26) we are particularly in need!</p>
|
||||
<h2 id="links">Links</h2>
|
||||
<ul>
|
||||
<li>Various links:
|
||||
<ul>
|
||||
<li><a href="https://unix.stackexchange.com/questions/23111/what-is-the-eval-command-in-bash">What is the “eval” command in bash?</a></li>
|
||||
<li><a href="https://www.shell-tips.com/bash/arrays/">A Complete Guide on How To Use Bash Arrays</a></li>
|
||||
<li><a href="https://www.computerhope.com/unix/bash/mapfile.htm">Bash mapfile builtin command</a></li>
|
||||
<li><a href="https://www.gnu.org/software/bash/manual/html_node/Simple-Command-Expansion.html">Bash manual 3.7.1 Simple Command Expansion</a></li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
<ul>
|
||||
<li>Previous episodes in this series about arrays:
|
||||
<ul>
|
||||
<li><a href="https://hackerpublicradio.org/eps.php?id=2709">Bash Tips - 16</a></li>
|
||||
<li><a href="https://hackerpublicradio.org/eps.php?id=2719">Bash Tips - 17</a></li>
|
||||
<li><a href="https://hackerpublicradio.org/eps.php?id=2729">Bash Tips - 18</a></li>
|
||||
<li><a href="https://hackerpublicradio.org/eps.php?id=2739">Bash Tips - 19</a></li>
|
||||
<li><a href="https://hackerpublicradio.org/eps.php?id=2756">Bash Tips - 20</a></li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
<section class="footnotes" role="doc-endnotes">
|
||||
<hr />
|
||||
<ol>
|
||||
<li id="fn1" role="doc-endnote"><p>I don’t think I mentioned the need for <code>sort -z</code> in the audio. However later testing showed that this option is needed to sort the output properly.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
|
||||
</ol>
|
||||
</section>
|
||||
</article>
|
||||
</main>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user