Files
hpr_website/www/eps/hpr1951/hpr1951_full_shownotes.html

356 lines
23 KiB
HTML
Raw Permalink Normal View History

2025-10-28 18:39:57 +01:00
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<meta name="author" content="Dave Morriss">
<title>Some additional Bash tips (HPR Show 1951)</title>
<style type="text/css">code{white-space: pre;}</style>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<link rel="stylesheet" href="http://hackerpublicradio.org/css/hpr.css">
</head>
<body id="home">
<div id="container" class="shadow">
<header>
<h1 class="title">Some additional Bash tips (HPR Show 1951)</h1>
<h2 class="author">Dave Morriss</h2>
<hr/>
</header>
<main id="maincontent">
<article>
<header>
<h1>Table of Contents</h1>
<nav id="TOC">
<ul>
<li><a href="#expansion">Expansion</a><ul>
<li><a href="#note">Note</a></li>
<li><a href="#arithmetic-expansion">Arithmetic expansion</a><ul>
<li><a href="#examples-of-arithmetic-evaluation">Examples of Arithmetic Evaluation</a></li>
</ul></li>
</ul></li>
<li><a href="#links">Links</a></li>
<li><a href="#manual-page-extracts">Manual Page Extracts</a><ul>
<li><a href="#expansion-1">EXPANSION</a><ul>
<li><a href="#brace-expansion">Brace Expansion</a></li>
<li><a href="#tilde-expansion">Tilde Expansion</a></li>
<li><a href="#parameter-expansion">Parameter Expansion</a></li>
<li><a href="#command-substitution">Command Substitution</a></li>
<li><a href="#arithmetic-expansion-1">Arithmetic Expansion</a></li>
</ul></li>
<li><a href="#arithmetic-evaluation">ARITHMETIC EVALUATION</a></li>
</ul></li>
</ul>
</nav>
</header>
<h2 id="expansion">Expansion</h2>
<p>As we saw in the last episode <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a> there are seven types of expansion applied to the command line in the following order:</p>
<ul>
<li>Brace expansion (we looked at this subject in episode <a href="http://hackerpublicradio.org/eps/hpr1884" title="Some more Bash tips">1884</a>)</li>
<li>Tilde expansion (seen in episode <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>)</li>
<li>Parameter and variable expansion (this was covered in episode <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1648</a>)</li>
<li>Command substitution (seen in episode <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>)</li>
<li>Arithmetic expansion</li>
<li>Word splitting</li>
<li>Pathname expansion</li>
</ul>
<p>There is also another, <em>process substitution</em>, which occurs after <em>arithmetic expansion</em> on systems that can implement it.</p>
<p>We will look at one more of these expansion types in this episode but since there is a lot to cover, we'll continue this subject in a later episode.</p>
<h4 id="note">Note</h4>
<p>For this episode I have changed the convention I am using for indicating commands and their output to the following:</p>
<pre><code>$ echo &quot;Message&quot;
Message</code></pre>
<p>The line beginning with a <strong>$</strong> is the command that is typed and the rest is what is returned by the command.</p>
<p>It was pointed out to me that there was ambiguity in the examples in previous episodes, for which I apologise.</p>
<h3 id="arithmetic-expansion">Arithmetic expansion</h3>
<p>This form of expansion evaluates an arithmetic expression and returns the result. The format is:</p>
<pre><code>$((expression))</code></pre>
<p>So an example might be:</p>
<pre><code>$ echo $((42/5))
8</code></pre>
<p>This is integer arithmetic; the fractional part is simply thrown away.</p>
<p>To digress: if you want the full fractional answer then using the <code>bc</code> command would probably be wiser. This was covered in Dann Washko's &quot;<em>Linux in the Shell</em>&quot; series in HPR show number <a href="http://hackerpublicradio.org/eps/hpr1202" title="LiTS 025: bc">1202</a>.</p>
<p>For example, using <code>bc</code> in command substitution as in:</p>
<pre><code>$ echo $(echo &quot;scale=2; 42/5&quot; | bc)
8.40</code></pre>
<p>The &quot;<code>scale=2</code>&quot; is required to make <code>bc</code> output the result with two decimal places. By default it does not do this.</p>
<p>Note that using <code>echo</code> to report the result of this command sequence is not normally useful. It is used here just to demonstrate the point. Writing something like the following makes more sense in a script:</p>
<pre><code>$ res=$(echo &quot;scale=2; 42/5&quot; | bc)
$ echo $res
8.40</code></pre>
<p>The expressions allowed by Bash in arithmetic expansion include the use of variables. Normally these variables are written as just the plain name without the leading '<strong>$</strong>', though adding this is permitted. For example:</p>
<pre><code>$ x=42
$ echo $((x/5))
8
$ echo $(($x/5))
8</code></pre>
<p>There are potential pitfalls with using the '<strong>$</strong>' however, as we will see. The expression is subject to variable expansion, so in the second example above <strong>$x</strong> becomes 42 and the expression resolves to <code>42/5</code>.</p>
<p>If a variable is null or unset (and is used without the leading '<strong>$</strong>') then it evaluates to zero. This is another reason not to use the parameter substitution method.</p>
<p>The value of a variable is always interpreted as an integer. If it is not an integer (for example, if it's a text string) then it is treated as zero.</p>
<pre><code>$ str=&quot;A&quot;
$ echo $((str*2))
0
$ str=&quot;0xA&quot;
$ echo $((str*2))
20</code></pre>
<p>Bash also interprets non-decimal numerical constants (as in the second example). For a start, any number beginning with a zero is taken to be octal, and hexadecimal numbers are denoted by a leading <code>0x</code> or <code>0X</code>.</p>
<p>Be aware that the way in which octal constants are written lead to unexpected outcomes:</p>
<pre><code>$ x=010
$ echo $((x))
8
$ x=018
$ echo $((x))
bash: 018: value too great for base (error token is &quot;018&quot;)
$ printf -v x &quot;%03d\n&quot; 19
$ echo $((x))
bash: 019: value too great for base (error token is &quot;019&quot;)</code></pre>
<p>There is also a complete system of defining numbers with bases between 2 and 64. Such numbers are written as:</p>
<p><em>base</em>#<em>number</em></p>
<p>If the '<em>base</em>#' is omitted then base 10 is used (or the octal and hexadecimal conventions above may be used).</p>
<p>Like in hexadecimal numbers, other characters are used to show the digits of other bases. These are in order 'a' to 'z', 'A' to 'Z' and '@' and '_'.</p>
<p>The contexts in which these number formats are understood by Bash are limited. Consider the following:</p>
<pre><code>$ x=16#F
$ echo $x
16#F
$ x=0xF
$ echo $x
0xF</code></pre>
<p>Bash has not converted these values, but has treated them like strings.</p>
<p>It is possible to declare a variable as an integer (and set its value) thus:</p>
<pre><code>$ declare -i II=16#F
$ echo $II
15</code></pre>
<p>In this case the base 16 number has been converted.</p>
<p>There is also a <strong><code>let</code></strong> command that will evaluate such numeric constants:</p>
<pre><code>$ let x=16#F
$ echo $x
15</code></pre>
<p>Alternatively, using arithmetic expansion syntax causes interpretation to take place:</p>
<pre><code>$ x=16#F
$ echo $((x))
15</code></pre>
<p>The following loop could be used to examine the decimal values 0..64 using base64 notation. I have written it here as a short Bash script which could be placed in a file:</p>
<pre><code>#!/usr/bin/env bash
for x in {0..9} {a..z} {A..Z} @ _; do
n=&quot;64#$x&quot;
echo &quot;$n=$((n))&quot;
done</code></pre>
<p>The script reports values like:</p>
<pre><code>64#0=0
64#1=1
64#2=2
.
.
64#Y=60
64#Z=61
64#@=62
64#_=63</code></pre>
<p>There is more than can be said about this, but I will leave you to explore. I could possibly talk about this subject in another episode if there is any interest.</p>
<h4 id="examples-of-arithmetic-evaluation">Examples of Arithmetic Evaluation</h4>
<p>The way in which the <em>arithmetic expression</em> in an <em>arithmetic expansion</em> is interpreted is defined in the Bash manpage under the <strong>ARITHMETIC EVALUATION</strong> heading. A copy of this is included at the end of these notes.</p>
<p>The use of arithmetic evaluation in Bash is quite powerful but has some problems. I could devote a whole episode to this subject, but I will restrict myself in this episode. I prepared a few examples of some of the operators which I hope will give some food for thought.</p>
<h5 id="pre--and-post-increment-and-decrement">Pre- and post-increment and decrement</h5>
<p>These operators increment or decrement the contents of a variable by 1. The pre- and post- effects control whether the operation is performed before or after the value is used.</p>
<p>Here are some examples of the effects:</p>
<pre><code>$ val=87
$ echo &quot;$((++val)) $val&quot;
88 88
$ echo &quot;$((--val)) $val&quot;
87 87
$ echo &quot;$((val++)) $val&quot;
87 88
$ echo &quot;$((val--)) $val&quot;
88 87</code></pre>
<p>Note that the pre- and post-increment and decrement operators need variables, not numbers. That means that the following is acceptable, as we saw:</p>
<pre><code>$ myvar=128
$ echo $((++myvar))
129</code></pre>
<p>However, placing a <strong>$</strong> in front of the variable name causes substitution to take place and its contents to be used, which is either not acceptable or leads to unwanted effects:</p>
<pre><code>$ myvar=128
$ echo $((++myvar))
129
$ echo $(($myvar--))
bash: 129--: syntax error: operand expected (error token is &quot;-&quot;)</code></pre>
<p>Also be aware that the following expression is not illegal, but is potentially confusing:</p>
<pre><code>$ myvar=128
$ echo $((--$myvar))
128</code></pre>
<p>It substitutes the value of <code>myvar</code> which then has two '<strong>-</strong>' signs in front of it: <code>--128</code>. The effect is the same as <code>-(-128)</code>, in other words, the two minus signs cancel one another. Plenty of scope for confusion!</p>
<h5 id="unary-minus">Unary minus</h5>
<pre><code>$ int=16
$ echo $((-int))
-16</code></pre>
<p>This example just turns 16 into minus 16, as you would expect. We already saw this when discussing the pre-decrement operator.</p>
<h5 id="exponentiation">Exponentiation</h5>
<pre><code>$ echo $((int**2))
256</code></pre>
<p>Here we compute 16<sup>2</sup>.</p>
<h5 id="more-complex-expressions-with-parentheses">More complex expressions with parentheses</h5>
<pre><code>$ echo $(( (int**2 + 3) % 5 ))
4</code></pre>
<p>This adds 3 to the result of 16<sup>2</sup> (259) then returns the remainder after division by 5. We need the parentheses to prevent the <strong>%</strong> (remainder) operator applying to the 3.</p>
<h5 id="bitwise-shift-bitwise-or">Bitwise shift, bitwise OR</h5>
<pre><code>$ echo $((int&gt;&gt;1))
8
$ echo $((int&lt;&lt;1))
32
$ echo $(( (int&lt;&lt;1) | 8 ))
40
$ printf &quot;%#x\n&quot; $(( (int&lt;&lt;1) | 8 ))
0x28</code></pre>
<ol type="1">
<li>Since 16 is binary 10000, shifting it to the right once returns 1000 which is the binary representation of decimal 8.</li>
<li>Shifting 16 to the left once returns 100000 which is 32 in decimal.</li>
<li>Taking 16 shifted left 1 (32) and binary OR'ing 8 to it is the same as binary 100000 OR 01000 which is 101000, which is decimal 40.</li>
<li>The same calculation printed as hexadecimal is 28, which can be visualised in binary as 0010 1000. The <code>printf</code> format <strong>%#x</strong> prints numbers in hexadecimal with a leading <strong>0x</strong>.</li>
</ol>
<h5 id="conditional-operator">Conditional operator</h5>
<p>The conditional operator is similar to the equivalent in C and many other languages but only operates with integer values.</p>
<pre><code>$ myvar=$((int&lt;&lt;3))
$ msg=(&#39;under 100&#39; &#39;between 100 and 200&#39; &#39;over 200&#39;)
$ range=$((myvar&gt;100?$((myvar&gt;200?2:1)):0))
$ echo &quot;myvar=$myvar, range=$range, message: ${msg[$range]}&quot;
myvar=128, range=1, message: between 100 and 200</code></pre>
<p>Here <code>myvar</code> is set to 16 shifted left 3 places, which is the same as multiplying it by 2 3 times, resulting in 128.</p>
<p>We declare an array <code>msg</code> which holds three text strings (index 0, 1 and 2).</p>
<p>Then <code>range</code> is set to the result of a complex expression. If <code>myvar</code> is greater than 100 then the second arithmetic expansion is used which tests to see if <code>myvar</code> is greater than 200. If it is then the result returned is 2, otherwise 1 is returned. If the value of <code>myvar</code> is less than 100 then 0 is returned.</p>
<p>So a value of 0 means &quot;<em>under 100</em>&quot;, 1 means &quot;<em>between 100 and 200</em>&quot; and 2 means &quot;<em>over 200</em>&quot;. The <code>echo</code> reports the values of <code>myvar</code> and <code>range</code> and uses <code>range</code> to index the appropriate element of the array <code>msg</code> (we looked at array indexing in episode <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1684</a>).</p>
<pre><code>$ myvar=$((int&lt;&lt;4))
$ range=$((myvar&gt;100?$((myvar&gt;200?2:1)):0))
$ echo &quot;myvar=$myvar, range=$range, message: ${msg[$range]}&quot;
myvar=256, range=2, message: over 200</code></pre>
<p>Here we set <code>myvar</code> to 16 shifted left 4 places, or in other words 16 times 2<sup>4</sup>, which is 256. We then recalculate the value of <code>range</code> and use the same <code>echo</code> as before.</p>
<p>These are not particularly robust examples of conditional expressions, but hopefully they serve to make the point.</p>
<h5 id="assignment">Assignment</h5>
<p>Variable assignments may be performed in these arithmetic expressions. The assignment operators also include combinations with arithmetic operators:</p>
<pre><code>$ echo $((x=12#20))
24
$ echo $((x*=2))
48
$ echo $((x%=5))
3</code></pre>
<p>In this example <code>x</code> is set to 20<sub>12</sub> (&quot;two zero base 12&quot;) which is decimal 24. This is then multiplied by 2 and saved back into <code>x</code>, then the remainder of division by 5 is saved in <code>x</code>.</p>
<pre><code>$ echo $((b=2#1000))
8
$ echo $((b|=2#10))
10</code></pre>
<p>Here the number 1000<sub>2</sub> (&quot;one zero zero zero base 2&quot;) is saved in <code>b</code>, which is decimal 8. This is then bitwise OR'ed with 10<sub>2</sub> (&quot;one zero base 2&quot;) which is 2. The result saved in <code>b</code> is decimal 10 (or 1010<sub>2</sub>).</p>
<p>There is no simple way of printing binary numbers in Bash. If you have difficulty in visualising them, you can use <code>bc</code> as follows:</p>
<pre><code>$ echo &quot;obase=2;$b&quot; | bc
1010</code></pre>
<p>The <code>obase</code> variable in <code>bc</code> defines the output number base.</p>
<h2 id="links">Links</h2>
<ul>
<li>Bash Reference Manual:
<ul>
<li>Arithmetic Expansion: <a href="https://www.gnu.org/software/bash/manual/html_node/Arithmetic-Expansion.html#Arithmetic-Expansion" class="uri">https://www.gnu.org/software/bash/manual/html_node/Arithmetic-Expansion.html#Arithmetic-Expansion</a></li>
<li>Shell Arithmetic: <a href="https://www.gnu.org/software/bash/manual/html_node/Shell-Arithmetic.html#Shell-Arithmetic" class="uri">https://www.gnu.org/software/bash/manual/html_node/Shell-Arithmetic.html#Shell-Arithmetic</a></li>
</ul></li>
<li>HPR episode 1202 &quot;LiTS 025: bc&quot; hosted by <a href="http://hackerpublicradio.org/correspondents.php?hostid=7">Dann Washko</a>: <a href="http://hackerpublicradio.org/eps/hpr1202" class="uri">http://hackerpublicradio.org/eps/hpr1202</a></li>
<li>HPR episode 1648 &quot;<em>Bash parameter manipulation</em>&quot;: <a href="http://hackerpublicradio.org/eps/hpr1648" class="uri">http://hackerpublicradio.org/eps/hpr1648</a></li>
<li>HPR episode 1843 &quot;<em>Some Bash tips</em>&quot;: <a href="http://hackerpublicradio.org/eps/hpr1843" class="uri">http://hackerpublicradio.org/eps/hpr1843</a></li>
<li>HPR episode 1884 &quot;<em>Some more Bash tips</em>&quot;: <a href="http://hackerpublicradio.org/eps/hpr1884" class="uri">http://hackerpublicradio.org/eps/hpr1884</a></li>
<li>HPR episode 1903 &quot;<em>Some further Bash tips</em>&quot;: <a href="http://hackerpublicradio.org/eps/hpr1903" class="uri">http://hackerpublicradio.org/eps/hpr1903</a></li>
</ul>
<hr />
<h1 id="manual-page-extracts">Manual Page Extracts</h1>
<h2 id="expansion-1">EXPANSION</h2>
<p>Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: <em>brace expansion</em>, <em>tilde expansion</em>, <em>parameter and variable expansion</em>, <em>command substitution</em>, <em>arithmetic expansion</em>, <em>word splitting</em>, and <em>pathname expansion</em>.</p>
<p>The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.</p>
<p>On systems that can support it, there is an additional expansion available: <em>process substitution</em>. This is performed at the same time as tilde, parameter, variable, and arithmetic expansion and command substitution.</p>
<p>Only brace expansion, word splitting, and pathname expansion can change the number of words of the expansion; other expansions expand a single word to a single word. The only exceptions to this are the expansions of &quot;<strong>$@</strong>&quot; and &quot;<strong>${name[@]}</strong>&quot; as explained above (see <strong>PARAMETERS</strong>).</p>
<h3 id="brace-expansion">Brace Expansion</h3>
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1884" title="Some more Bash tips">1884</a>.</p>
<h3 id="tilde-expansion">Tilde Expansion</h3>
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>.</p>
<h3 id="parameter-expansion">Parameter Expansion</h3>
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1648</a>.</p>
<h3 id="command-substitution">Command Substitution</h3>
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>.</p>
<h3 id="arithmetic-expansion-1">Arithmetic Expansion</h3>
<p>Arithmetic expansion allows the evaluation of an arithmetic expression and the substitution of the result. The format for arithmetic expansion is:</p>
<pre><code>$((expression))</code></pre>
<p>The old format <code>$[expression]</code> is deprecated and will be removed in upcoming versions of bash.</p>
<p>The expression is treated as if it were within double quotes, but a double quote inside the parentheses is not treated specially. All tokens in the expression undergo parameter and variable expansion, command substitution, and quote removal. The result is treated as the arithmetic expression to be evaluated. Arithmetic expansions may be nested.</p>
<p>The evaluation is performed according to the rules listed below under <strong>ARITHMETIC EVALUATION</strong>. If expression is invalid, bash prints a message indicating failure and no substitution occurs.</p>
<hr />
<h2 id="arithmetic-evaluation">ARITHMETIC EVALUATION</h2>
<p>The shell allows arithmetic expressions to be evaluated, under certain circumstances (see the <strong>let</strong> and <strong>declare</strong> builtin commands and <strong>Arithmetic Expansion</strong>). Evaluation is done in fixed-width integers with no check for overflow, though division by 0 is trapped and flagged as an error. The operators and their precedence, associativity, and values are the same as in the C language. The following list of operators is grouped into levels of equal-precedence operators. The levels are listed in order of decreasing precedence.</p>
<dl>
<dt><strong>id++</strong> <strong>id--</strong></dt>
<dd><pre><code>variable post-increment and post-decrement</code></pre>
</dd>
<dt><strong>++id --id</strong></dt>
<dd><pre><code>variable pre-increment and pre-decrement</code></pre>
</dd>
<dt><strong>- +</strong></dt>
<dd><pre><code>unary minus and plus</code></pre>
</dd>
<dt><strong>! ~</strong></dt>
<dd><pre><code>logical and bitwise negation</code></pre>
</dd>
<dt><strong>**</strong></dt>
<dd><pre><code>exponentiation</code></pre>
</dd>
<dt><strong>* / %</strong></dt>
<dd><pre><code>multiplication, division, remainder</code></pre>
</dd>
<dt><strong>+ -</strong></dt>
<dd><pre><code>addition, subtraction</code></pre>
</dd>
<dt><strong>&lt;&lt; &gt;&gt;</strong></dt>
<dd><pre><code>left and right bitwise shifts</code></pre>
</dd>
<dt><strong>&lt;= &gt;= &lt; &gt;</strong></dt>
<dd><pre><code>comparison</code></pre>
</dd>
<dt><strong>== !=</strong></dt>
<dd><pre><code>equality and inequality</code></pre>
</dd>
<dt><strong>&amp;</strong></dt>
<dd><pre><code>bitwise AND</code></pre>
</dd>
<dt><strong>^</strong></dt>
<dd><pre><code>bitwise exclusive OR</code></pre>
</dd>
<dt><strong>|</strong></dt>
<dd><pre><code>bitwise OR</code></pre>
</dd>
<dt><strong>&amp;&amp;</strong></dt>
<dd><pre><code>logical AND</code></pre>
</dd>
<dt><strong>||</strong></dt>
<dd><pre><code>logical OR</code></pre>
</dd>
<dt><strong>expr?expr:expr</strong></dt>
<dd><pre><code>conditional operator</code></pre>
</dd>
<dt><strong>= *= /= %= += -= &lt;&lt;= &gt;&gt;= &amp;= ^= |=</strong></dt>
<dd><pre><code>assignment</code></pre>
</dd>
<dt><strong>expr1 , expr2</strong></dt>
<dd><pre><code>comma</code></pre>
</dd>
</dl>
<!-- ******** -->
<p>Shell variables are allowed as operands; parameter expansion is performed before the expression is evaluated. Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax. A shell variable that is null or unset evaluates to 0 when referenced by name without using the parameter expansion syntax. The value of a variable is evaluated as an arithmetic expression when it is referenced, or when a variable which has been given the <em>integer</em> attribute using <strong>declare -i</strong> is assigned a value. A null value evaluates to 0. A shell variable need not have its <em>integer</em> attribute turned on to be used in an expression.</p>
<p>Constants with a leading 0 are interpreted as octal numbers. A leading 0x or 0X denotes hexadecimal. Otherwise, numbers take the form [<em>base#</em>]n, where the optional <em>base</em> is a decimal number between 2 and 64 representing the arithmetic <em>base</em>, and n is a number in that <em>base</em>. If <em>base#</em> is omitted, then <em>base</em> 10 is used. When specifying n, the digits greater than 9 are represented by the lowercase letters, the uppercase letters, @, and _, in that order. If <em>base</em> is less than or equal to 36, lowercase and uppercase letters may be used interchangeably to represent numbers between 10 and 35.</p>
<p>Operators are evaluated in order of precedence. Sub-expressions in parentheses are evaluated first and may override the precedence rules above.</p>
<!--
vim: syntax=markdown:ts=8:sw=4:ai:et:tw=78:fo=tcqn:fdm=marker
-->
</article>
</main>
</div>
</body>
</html>