371 lines
30 KiB
HTML
371 lines
30 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html>
|
||
|
|
<head>
|
||
|
|
<meta charset="utf-8">
|
||
|
|
<meta name="generator" content="pandoc">
|
||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
|
||
|
|
<meta name="author" content="Dave Morriss">
|
||
|
|
<title>Some other Bash tips (HPR Show 2045)</title>
|
||
|
|
<style type="text/css">code{white-space: pre;}</style>
|
||
|
|
<!--[if lt IE 9]>
|
||
|
|
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
|
||
|
|
<![endif]-->
|
||
|
|
<link rel="stylesheet" href="http://hackerpublicradio.org/css/hpr.css">
|
||
|
|
</head>
|
||
|
|
|
||
|
|
<body id="home">
|
||
|
|
<div id="container" class="shadow">
|
||
|
|
<header>
|
||
|
|
<h1 class="title">Some other Bash tips (HPR Show 2045)</h1>
|
||
|
|
<h2 class="author">Dave Morriss</h2>
|
||
|
|
<hr/>
|
||
|
|
</header>
|
||
|
|
|
||
|
|
<main id="maincontent">
|
||
|
|
<article>
|
||
|
|
<header>
|
||
|
|
<h1>Table of Contents</h1>
|
||
|
|
<nav id="TOC">
|
||
|
|
<ul>
|
||
|
|
<li><a href="#expansion">Expansion</a><ul>
|
||
|
|
<li><a href="#process-substitution">Process substitution</a><ul>
|
||
|
|
<li><a href="#process-substitution-examples">Process Substitution Examples</a></li>
|
||
|
|
</ul></li>
|
||
|
|
<li><a href="#word-splitting">Word splitting</a><ul>
|
||
|
|
<li><a href="#some-examples-of-word-splitting">Some examples of word splitting</a></li>
|
||
|
|
<li><a href="#the-internal-field-separator-ifs">The Internal Field Separator (IFS)</a></li>
|
||
|
|
</ul></li>
|
||
|
|
</ul></li>
|
||
|
|
<li><a href="#links">Links</a></li>
|
||
|
|
<li><a href="#manual-page-extracts">Manual Page Extracts</a><ul>
|
||
|
|
<li><a href="#expansion-1">EXPANSION</a><ul>
|
||
|
|
<li><a href="#brace-expansion">Brace Expansion</a></li>
|
||
|
|
<li><a href="#tilde-expansion">Tilde Expansion</a></li>
|
||
|
|
<li><a href="#parameter-expansion">Parameter Expansion</a></li>
|
||
|
|
<li><a href="#command-substitution">Command Substitution</a></li>
|
||
|
|
<li><a href="#arithmetic-expansion">Arithmetic Expansion</a></li>
|
||
|
|
<li><a href="#process-substitution-1">Process Substitution</a></li>
|
||
|
|
<li><a href="#word-splitting-1">Word Splitting</a></li>
|
||
|
|
</ul></li>
|
||
|
|
</ul></li>
|
||
|
|
</ul>
|
||
|
|
</nav>
|
||
|
|
</header>
|
||
|
|
<h2 id="expansion">Expansion</h2>
|
||
|
|
<p>As we saw in the last episode <a href="http://hackerpublicradio.org/eps/hpr1951" title="Some additional Bash tips">1951</a> (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:</p>
|
||
|
|
<ul>
|
||
|
|
<li>Brace expansion (we looked at this subject in episode <a href="http://hackerpublicradio.org/eps/hpr1884" title="Some more Bash tips">1884</a>)</li>
|
||
|
|
<li>Tilde expansion (seen in episode <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>)</li>
|
||
|
|
<li>Parameter and variable expansion (this was covered in episode <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1648</a>)</li>
|
||
|
|
<li>Command substitution (seen in episode <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>)</li>
|
||
|
|
<li>Arithmetic expansion (seen in episode <a href="http://hackerpublicradio.org/eps/hpr1951" title="Some additional Bash tips">1951</a>)</li>
|
||
|
|
<li>Process substitution</li>
|
||
|
|
<li>Word splitting</li>
|
||
|
|
<li>Pathname expansion</li>
|
||
|
|
</ul>
|
||
|
|
<p>We will look at process substitution and word splitting in this episode but since there is a lot to cover in these subjects, we'll save pathname expansion for the next episode.</p>
|
||
|
|
<h3 id="process-substitution">Process substitution</h3>
|
||
|
|
<p>The <em>process substitution</em> feature in Bash is a way in which data can be passed to or from a process. A <em>process</em> is most simply thought of as one or more commands running together.</p>
|
||
|
|
<p>Not all Unix systems can implement <em>process substitution</em> since it either uses what are known as <a href="https://en.wikipedia.org/wiki/Named_pipe" title="Named pipe">"<em>named pipes</em>"</a> (or <em>FIFO</em>s) or special files called <code>/dev/fd/<n></code> (where <em><n></em> represents a number). These are temporary data storage structures which separate processes are able to access for reading or writing.</p>
|
||
|
|
<p>The name of this pipe or <code>/dev/fd/<n></code> "<em>interconnecting file</em>" is passed as an argument to the initial command. This can be a difficult concept to understand, and we'll look at it in more detail soon.</p>
|
||
|
|
<p>There are two forms of <em>process substitution</em>:</p>
|
||
|
|
<pre><code>>(command list)
|
||
|
|
<(command list)</code></pre>
|
||
|
|
<p>The first form receives input which has been sent via the interconnecting file and passes it to the command list. The second form generates output from the command list and passes it on via the interconnecting file which needs to be read to receive the result. In both cases there must be no spaces between the '<code><</code>' or the '<code>></code>' and the open parenthesis.</p>
|
||
|
|
<p>Some experiments with simplified commands might help to clarify this. First consider a simple pipeline:</p>
|
||
|
|
<pre><code>$ echo Test | sed -e 's/^.*$/[&]/'
|
||
|
|
[Test]</code></pre>
|
||
|
|
<p>This is a pipeline where the <code>echo</code> command generates data on its STDOUT which is passed to the <code>sed</code> command on its STDIN via the pipe. The <code>sed</code> command modifies what it receives by placing square brackets around it and passes the result to its STDOUT and the text is displayed by the shell (Bash).</p>
|
||
|
|
<p>Contrast this with <em>process substitution</em>. If we want to generate the same text and pass it to <code>sed</code> within a process we might write:</p>
|
||
|
|
<pre><code>$ echo Test >(sed -e 's/^.*$/[&]/')
|
||
|
|
Test /dev/fd/63</code></pre>
|
||
|
|
<p>This does not work as expected. What is happening here is that an interconnecting file name (<code>/dev/fd/63</code>) has been created and passed to the <code>echo</code> command, with the expectation that it will be used to send data to the process. However, this is seen by the <code>echo</code> which has simply displayed it. The <em>process substitution</em> containing the <code>sed</code> command has received nothing.</p>
|
||
|
|
<p>This example needs to be rewritten by adding a redirection symbol (<strong>></strong>) after the <code>echo</code>:</p>
|
||
|
|
<pre><code>$ echo Test > >(sed -e 's/^.*$/[&]/')
|
||
|
|
[Test]</code></pre>
|
||
|
|
<p>Behind the scenes, Bash will have changed this (invisibly) into something like:</p>
|
||
|
|
<pre><code>$ echo Test > /dev/fd/63 >(sed -e 's/^.*$/[&]/')</code></pre>
|
||
|
|
<p>This time the interconnection between the command on the left and the <em>process substitution</em> expression holding the <code>sed</code> command has been made.</p>
|
||
|
|
<p>Note that using a pipe instead also does not work:</p>
|
||
|
|
<pre><code>$ echo Test | >(sed -e 's/^.*$/[&]/')
|
||
|
|
bash: /dev/fd/63: Permission denied</code></pre>
|
||
|
|
<p>This is because the filename is being used on the right of the pipe symbol where a command, script or program name is expected.</p>
|
||
|
|
<p>The corresponding version of this example using the other form of <em>process substitution</em> is:</p>
|
||
|
|
<pre><code>$ sed -e 's/^.*$/[&]/' <(echo Test)
|
||
|
|
[Test]</code></pre>
|
||
|
|
<p>Here the interconnecting file name is being provided to the <code>sed</code> command. To visualise this we can modify the <code>sed</code> script by using the <strong>F</strong> command, a GNU extension which reports the name of the input file (followed by a newline):</p>
|
||
|
|
<pre><code>$ sed -e 'F;s/^.*$/[&]/' <(echo Test)
|
||
|
|
/dev/fd/63
|
||
|
|
[Test]</code></pre>
|
||
|
|
<p>To wrap up, the Bash manual page states the following in the context of <em>process substitution</em> as part of the larger topic of <strong>Expansion</strong>:</p>
|
||
|
|
<p><em>When available, process substitution is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion.</em></p>
|
||
|
|
<p>We have now seen all of these expansion types in this series.</p>
|
||
|
|
<h4 id="process-substitution-examples">Process Substitution Examples</h4>
|
||
|
|
<p>An example of the first form is:</p>
|
||
|
|
<pre><code>$ echo "Hacker Public Radio" > >(sed -ne 's/\([A-Z]\)\([a-z]\+\)/\l\1\U\2/gp;')
|
||
|
|
hACKER pUBLIC rADIO</code></pre>
|
||
|
|
<p>This just uses a <code>sed</code> script to reverse the case of the words fed to it. It is equivalent to the very simple tests we did earlier to demonstrate the concepts of process substitution.</p>
|
||
|
|
<p>An example of the second form:</p>
|
||
|
|
<pre><code>$ sed -ne '1~5p' <(nl -w2 -ba -s': ' sed_demo1.txt)
|
||
|
|
1: Hacker Public Radio (HPR) is an Internet Radio show (podcast) that releases
|
||
|
|
6:
|
||
|
|
11: what topics have been covered so far just have a look at our Archive.</code></pre>
|
||
|
|
<p>This is a modification of an example used in the "<em>Introduction to sed</em>" series. Here <code>sed</code> has been requested to print line 1 of its input stream, followed by every fifth line thereafter. We used the <code>nl</code> (number lines) command to number the incoming lines to make it clear what was happening.</p>
|
||
|
|
<p>Here the <code>nl</code> command is being used in a process substitution to feed data to <code>sed</code> on its STDIN channel (via the interconnecting file of course). Note that this is just a demonstration of the topic, it would not make sense to use these two commands together in this way.</p>
|
||
|
|
<p>Another example of the second form:</p>
|
||
|
|
<pre><code>$ join <(shuf -n5 /usr/share/dict/words | sed -e "s/'.*$//" | nl) \
|
||
|
|
<(shuf -n5 /usr/share/dict/words | sed -e "s/'.*$//" | nl)
|
||
|
|
1 brine rationed
|
||
|
|
2 resister desks
|
||
|
|
3 democrats gall
|
||
|
|
4 Margie Bligh
|
||
|
|
5 segregates screwdrivers</code></pre>
|
||
|
|
<p>This example uses the <code>join</code> command to join two lists of random words. This command expects two files (or two data sources) containing lines with identical join fields. The default field is the first.</p>
|
||
|
|
<p>The two processes provide the words, both doing the same thing. The <code>shuf</code> command selects 5 words from the system dictionary. We use <code>sed</code> to strip any apostrophe and the characters that follow from these words. We then number the words using <code>nl</code>. Each process will generate the same numbers, and these make the field for <code>join</code>.</p>
|
||
|
|
<p>Thus the first pipeline will generate five words and add the numbers 1-5 to them, and so will the second, and so <code>join</code> will join together word 1 from the first stream with word 1 from the second, and so forth.</p>
|
||
|
|
<p>It's a fairly useless example, but I find it amusing.</p>
|
||
|
|
<p>A final example. This one uses a Bash <code>while</code> loop to receive lines from a database query in a multi-command <em>process substitution</em> expression (think of this as an excerpt from a longer script):</p>
|
||
|
|
<pre><code>count=0
|
||
|
|
while read id name
|
||
|
|
do
|
||
|
|
printf "%02d \"%s\"\n" $id "$name"
|
||
|
|
((count++))
|
||
|
|
done < <(echo "select id, name from series order by lower(name);" | sqlite3 hpr_talks.db)
|
||
|
|
echo "Found $count series"
|
||
|
|
|
||
|
|
51 "10 buck review"
|
||
|
|
80 "5150 Shades of Beer"
|
||
|
|
38 "A Little Bit of Python"
|
||
|
|
79 "Accessibility"
|
||
|
|
22 "All Songs Considered"
|
||
|
|
83 "April Fools Shows"
|
||
|
|
42 "Bash Scripting"
|
||
|
|
.
|
||
|
|
.
|
||
|
|
Found 82 series</code></pre>
|
||
|
|
<p>The query is requesting a list of series numbers and names from a SQLite database containing information about HPR shows. The query is made by using <code>echo</code> to pipe an SQL query expression to the <code>sqlite3</code> command.</p>
|
||
|
|
<p>The output from the query is being collected by a <code>read</code> statement into two variables <code>id</code> and <code>name</code>. These are then printed with a <code>printf</code> command, though such a loop would normally be used to perform some action using these variables, not just print them.</p>
|
||
|
|
<p>Normally such a <code>while</code> loop would read from a file by placing a redirection symbol (<strong><</strong>) and a file name after the <code>done</code> part. Here, instead of a file we are reading the output of a process which is querying a database.</p>
|
||
|
|
<p>A counter variable <code>count</code> is set up before the loop, and incremented<a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a> within it. The final total is reported after the loop.</p>
|
||
|
|
<p>A sample of the output is shown after the code snippet.</p>
|
||
|
|
<p>As an aside, it is possible to build a loop using a pipeline, avoiding <em>process substitution</em> altogether:</p>
|
||
|
|
<pre><code>count=0
|
||
|
|
echo "select id, name from series order by lower(name);" |\
|
||
|
|
sqlite3 hpr_talks.db |\
|
||
|
|
while read id name
|
||
|
|
do
|
||
|
|
printf "%02d \"%s\"\n" $id "$name"
|
||
|
|
((count++))
|
||
|
|
done
|
||
|
|
echo "Found $count series"</code></pre>
|
||
|
|
<p>The problem is that the final <code>echo</code> returns a count of zero.</p>
|
||
|
|
<p>This can be puzzling to new users of Bash - it certainly puzzled me when I first encountered it. It is because the <code>while</code> loop in this case runs in a separate process. Bash does not share variables between processes, so the <code>count</code> variable inside the loop is different from the one initialised before the loop. Thus the <code>count</code> outside the loop remains at zero.</p>
|
||
|
|
<p>If there is interest in looking further at this issue and other "<em>Bash Gotchas</em>" like it it could be included in a future episode of this (sub-)series.</p>
|
||
|
|
<h3 id="word-splitting">Word splitting</h3>
|
||
|
|
<p>The subject of <em>word splitting</em> is important to the understanding of how Bash works. Not fully appreciating this has been the downfall of many Bash script-writers, myself included.</p>
|
||
|
|
<h4 id="some-examples-of-word-splitting">Some examples of word splitting</h4>
|
||
|
|
<p>By default, and looked at simplistically, Bash separates words using spaces. The following simple function can be used to report how many arguments it has received in order to demonstrate this:</p>
|
||
|
|
<pre><code>function countargs () {
|
||
|
|
echo $#
|
||
|
|
}</code></pre>
|
||
|
|
<p>The function is called <code>countargs</code> for "count arguments". When it is called with arguments these are available to the function in the same way that arguments are available to a script. So, the special variable <code>#</code> contains the argument count.</p>
|
||
|
|
<p>Calling <code>countargs</code> with no arguments gives the answer 0:</p>
|
||
|
|
<pre><code>$ countargs
|
||
|
|
0</code></pre>
|
||
|
|
<p>However, calling it with a string argument returns 1, showing that a string is a <em>word</em> in the context of a Bash script:</p>
|
||
|
|
<pre><code>$ countargs "Mary had a little lamb"
|
||
|
|
1
|
||
|
|
$ countargs 'Mary had a little lamb'
|
||
|
|
1</code></pre>
|
||
|
|
<p>This also works if the string is empty:</p>
|
||
|
|
<pre><code>$ countargs ""
|
||
|
|
1</code></pre>
|
||
|
|
<p>When variables are used things become a little more complex:</p>
|
||
|
|
<pre><code>$ str="fish fingers and custard"
|
||
|
|
$ countargs $str
|
||
|
|
4</code></pre>
|
||
|
|
<p>Here the variable <code>str</code> has been expanded, which has resulted in four words being passed to the function. This has happened because Bash has applied <em>word splitting</em> to the result of expanding <code>str</code>.</p>
|
||
|
|
<p>If you want to pass a string like this to a function such that it can be used in the same way as in the calling environment, then it needs to be enclosed in double quotes:</p>
|
||
|
|
<pre><code>$ countargs "$str"
|
||
|
|
1</code></pre>
|
||
|
|
<p>While we are examining arguments on the command line it might be useful to create another function. This one, <code>printargs</code>, simply prints its arguments, one per line with the argument number in front of each.</p>
|
||
|
|
<pre><code>function printargs() {
|
||
|
|
i=1
|
||
|
|
for arg; do
|
||
|
|
echo "$i $arg"
|
||
|
|
((i++))
|
||
|
|
done
|
||
|
|
}</code></pre>
|
||
|
|
<p>This function uses a Bash <code>for</code> loop. Normally this is written as:</p>
|
||
|
|
<pre><code>for var in list; do
|
||
|
|
# Do things
|
||
|
|
done</code></pre>
|
||
|
|
<p>However, if the <code>in list</code> part is omitted the loop cycles through the arguments passed to a script or function. That is what is being done here, using the variable <code>arg</code> to hold the values. The variable <code>i</code> is used to count through the argument numbers.</p>
|
||
|
|
<p>We will look at Bash loops in more detail in a later episode.</p>
|
||
|
|
<p>Using this function instead of <code>countargs</code> we get:</p>
|
||
|
|
<pre><code>$ printargs $str
|
||
|
|
1 fish
|
||
|
|
2 fingers
|
||
|
|
3 and
|
||
|
|
4 custard
|
||
|
|
|
||
|
|
$ printargs "$str"
|
||
|
|
1 fish fingers and custard</code></pre>
|
||
|
|
<p>We can see that word splitting has taken place in the first example but in the second enclosing the variable expansion in double quotes has suppressed this splitting.</p>
|
||
|
|
<h4 id="the-internal-field-separator-ifs">The Internal Field Separator (IFS)</h4>
|
||
|
|
<p>As we have seen, word splitting normally takes place using spaces as the word delimiter.</p>
|
||
|
|
<p>In fact, this delimiter is controlled by a special Bash variable "<code>IFS</code>" (which stands for <a href="https://en.wikipedia.org/wiki/Internal_field_separator" title="Internal Field Separator">"<em>Internal Field Separator</em>"</a>).</p>
|
||
|
|
<p>Normally, by default, the <code>IFS</code> variable contains three characters: a <em>space</em>, a <em>tab</em> and a <em>newline</em>. If the <code>IFS</code> variable is unset it is treated as if it holds these characters. However, if its value is <em>null</em> then no splitting occurs.</p>
|
||
|
|
<p>It is important to understand the difference between <em>unset</em> and <em>null</em> if you want to manipulate this variable. If a Bash variable is <strong>unset</strong> it is not defined at all. This can be achieved with the command:</p>
|
||
|
|
<pre><code>unset IFS</code></pre>
|
||
|
|
<p>If the variable is <strong>null</strong> then it is defined but has no value, which can be achieved with the command:</p>
|
||
|
|
<pre><code>IFS=</code></pre>
|
||
|
|
<p>If you are changing the <code>IFS</code> variable and need to change it back to its default value then this can be achieved in several ways. First, it can be defined explicitly by typing a <em>space</em>, <em>tab</em> and a <em>newline</em> in a string. This is not as simple as it seems though, so an alternative is this:</p>
|
||
|
|
<pre><code>printf -v IFS " \t\n"</code></pre>
|
||
|
|
<p>This relies on the ability of <code>printf</code> to generate special characters using escape sequences, and the <code>-v</code> option which writes the result to a variable.</p>
|
||
|
|
<p>The technique normally used in a Bash script is to save the value of <code>IFS</code> before changing it, then restore it later. For example:</p>
|
||
|
|
<pre><code>oldIFS="$IFS"
|
||
|
|
IFS=":"
|
||
|
|
|
||
|
|
# Commands using changed IFS
|
||
|
|
|
||
|
|
IFS="$oldIFS"</code></pre>
|
||
|
|
<p>One method of checking what the <code>IFS</code> variable contains is by using the <code>cat</code> command with the option <code>-A</code> which is a shorthand way of using the options <code>-v</code>, <code>-T</code> and <code>-E</code> which have the following effects:</p>
|
||
|
|
<ul>
|
||
|
|
<li>Option <code>-v</code> displays non-printing characters (except <em>tab</em>)</li>
|
||
|
|
<li>Option <code>-T</code> displays <em>tab</em> characters as "<code>^I</code>"</li>
|
||
|
|
<li>Option <code>-E</code> displays a <strong>$</strong> character at the end of each line</li>
|
||
|
|
</ul>
|
||
|
|
<p>Simply echoing <code>IFS</code> into <code>cat</code> will do it:</p>
|
||
|
|
<pre><code>$ echo "$IFS" | cat -A
|
||
|
|
^I$
|
||
|
|
$</code></pre>
|
||
|
|
<p>The output shows the <em>space</em>, followed by the <code>^I</code> representation of <em>tab</em> followed by a dollar sign which marks the end of a line. This is because of the newline character. The next line also contains a dollar sign because this represents the line generated by the newline character.</p>
|
||
|
|
<p>An alternative is to use the <code>od</code> (<em>octal dump</em>) command. This is intended for dumping the contents of files to examine their binary formats. Here I have chosen the <code>-a</code> option which generates character names, and <code>-c</code> which shows characters as backslash escapes:</p>
|
||
|
|
<pre><code>$ echo -n "$IFS" | od -ac
|
||
|
|
0000000 sp ht nl
|
||
|
|
\t \n
|
||
|
|
0000003</code></pre>
|
||
|
|
<p>The leading numbers are offsets in the "<em>file</em>". Note that we used the <code>-n</code> option to <code>echo</code> to prevent it generating an extra newline.</p>
|
||
|
|
<p>So, now we know that the <code>IFS</code> variable contains three characters by default, and any of these will be used as a delimiter. So, it is possible to prepare strings as follows:</p>
|
||
|
|
<pre><code>$ str=" Wynken, Blynken, and Nod one night
|
||
|
|
> Sailed off in a wooden shoe "</code></pre>
|
||
|
|
<p>The <code>></code> character on the second line is the Bash prompt indicating that the string is incomplete because the closing quote has not yet been entered. Note the existence of leading and trailing spaces.</p>
|
||
|
|
<pre><code>$ printargs $str
|
||
|
|
1 Wynken,
|
||
|
|
2 Blynken,
|
||
|
|
3 and
|
||
|
|
4 Nod
|
||
|
|
5 one
|
||
|
|
6 night
|
||
|
|
7 Sailed
|
||
|
|
8 off
|
||
|
|
9 in
|
||
|
|
10 a
|
||
|
|
11 wooden
|
||
|
|
12 shoe</code></pre>
|
||
|
|
<p>Note that the embedded newline is treated as a word delimiter and the leading and trailing spaces are ignored.</p>
|
||
|
|
<p>If we quote the string (and add square brackets around it to show leading and trailing spaces) we get the following:</p>
|
||
|
|
<pre><code>$ printargs "[$str]"
|
||
|
|
1 [ Wynken, Blynken, and Nod one night
|
||
|
|
Sailed off in a wooden shoe ]</code></pre>
|
||
|
|
<p>This time the leading spaces and embedded newline are retained and printed as part of the string.</p>
|
||
|
|
<p>Finally we will look at how the <code>IFS</code> variable can be used to perform word splitting on other delimiters.</p>
|
||
|
|
<p>In this example we'll define a string then save the old <code>IFS</code> value and set a new one. We will use an underscore as the delimiter:</p>
|
||
|
|
<pre><code>$ str="all dressed up - and nowhere to go"
|
||
|
|
$ oldIFS="$IFS"
|
||
|
|
$ IFS="_"
|
||
|
|
$ printargs $str
|
||
|
|
1 all dressed up - and nowhere to go</code></pre>
|
||
|
|
<p>Note that the string is no longer split up since it contains none of the delimiters in the <code>IFS</code> variable.</p>
|
||
|
|
<p>Here we use one of the Bash features we met in episode <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1648 "Bash parameter manipulation"</a>, <em>Pattern substitution</em>, with which we change all spaces to underscores:</p>
|
||
|
|
<pre><code>$ printargs ${str// /_}
|
||
|
|
1 all
|
||
|
|
2 dressed
|
||
|
|
3 up
|
||
|
|
4 -
|
||
|
|
5 and
|
||
|
|
6 nowhere
|
||
|
|
7 to
|
||
|
|
8 go</code></pre>
|
||
|
|
<p>Note that we get 8 words this way since the hyphen is treated as a word.</p>
|
||
|
|
<p>If however we change the <code>IFS</code> variable again to include the hyphen as a delimiter we get a different result:</p>
|
||
|
|
<pre><code>$ IFS="_-"
|
||
|
|
$ printargs ${str// /_}
|
||
|
|
1 all
|
||
|
|
2 dressed
|
||
|
|
3 up
|
||
|
|
4
|
||
|
|
5
|
||
|
|
6 and
|
||
|
|
7 nowhere
|
||
|
|
8 to
|
||
|
|
9 go</code></pre>
|
||
|
|
<p>Here we have 9 words since the hyphen is now a delimiter. It might be useful to show the result of the substitution before word splitting:</p>
|
||
|
|
<pre><code>$ echo "${str// /_}"
|
||
|
|
all_dressed_up_-_and_nowhere_to_go</code></pre>
|
||
|
|
<p>There are three delimiters in sequence here which are interpreted as two null words (words 4 and 5 above).</p>
|
||
|
|
<p>Don't forget to restore the <code>IFS</code> variable afterwards otherwise you will probably find Bash behaves in a rather unexpected way:</p>
|
||
|
|
<pre><code>$ IFS="$oldIFS"</code></pre>
|
||
|
|
<h2 id="links">Links</h2>
|
||
|
|
<ul>
|
||
|
|
<li>HPR episode 1648 "<em>Bash parameter manipulation</em>": <a href="http://hackerpublicradio.org/eps/hpr1648" class="uri">http://hackerpublicradio.org/eps/hpr1648</a></li>
|
||
|
|
<li>HPR episode 1843 "<em>Some Bash tips</em>": <a href="http://hackerpublicradio.org/eps/hpr1843" class="uri">http://hackerpublicradio.org/eps/hpr1843</a></li>
|
||
|
|
<li>HPR episode 1884 "<em>Some more Bash tips</em>": <a href="http://hackerpublicradio.org/eps/hpr1884" class="uri">http://hackerpublicradio.org/eps/hpr1884</a></li>
|
||
|
|
<li>HPR episode 1903 "<em>Some further Bash tips</em>": <a href="http://hackerpublicradio.org/eps/hpr1903" class="uri">http://hackerpublicradio.org/eps/hpr1903</a></li>
|
||
|
|
<li>HPR episode 1951 "<em>Some additional Bash tips</em>": <a href="http://hackerpublicradio.org/eps/hpr1951" class="uri">http://hackerpublicradio.org/eps/hpr1951</a><br />
|
||
|
|
<br />
|
||
|
|
</li>
|
||
|
|
<li>"<em>Introduction to sed</em>" series on HPR:
|
||
|
|
<ul>
|
||
|
|
<li><em>Part 1</em>: <a href="http://hackerpublicradio.org/eps/hpr1976" class="uri">http://hackerpublicradio.org/eps/hpr1976</a></li>
|
||
|
|
<li><em>Part 2</em>: <a href="http://hackerpublicradio.org/eps/hpr1986" class="uri">http://hackerpublicradio.org/eps/hpr1986</a></li>
|
||
|
|
<li><em>Part 3</em>: <a href="http://hackerpublicradio.org/eps/hpr1997" class="uri">http://hackerpublicradio.org/eps/hpr1997</a></li>
|
||
|
|
<li><em>Part 4</em>: <a href="http://hackerpublicradio.org/eps/hpr2011" class="uri">http://hackerpublicradio.org/eps/hpr2011</a><br />
|
||
|
|
<br />
|
||
|
|
</li>
|
||
|
|
</ul></li>
|
||
|
|
<li>Wikipedia article on the "<em>Named pipe</em>": <a href="https://en.wikipedia.org/wiki/Named_pipe" class="uri">https://en.wikipedia.org/wiki/Named_pipe</a></li>
|
||
|
|
<li>Wikipedia article on the "<em>IFS</em>" variable: <a href="https://en.wikipedia.org/wiki/Internal_field_separator" class="uri">https://en.wikipedia.org/wiki/Internal_field_separator</a></li>
|
||
|
|
<li><em>Advanced Bash-Scripting Guide</em>: Section 9.1 <em>Bash Internal Variables</em> (including <code>IFS</code>): <a href="http://www.tldp.org/LDP/abs/html/internalvariables.html" class="uri">http://www.tldp.org/LDP/abs/html/internalvariables.html</a></li>
|
||
|
|
</ul>
|
||
|
|
<hr />
|
||
|
|
<h1 id="manual-page-extracts">Manual Page Extracts</h1>
|
||
|
|
<h2 id="expansion-1">EXPANSION</h2>
|
||
|
|
<p>Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: <em>brace expansion</em>, <em>tilde expansion</em>, <em>parameter and variable expansion</em>, <em>command substitution</em>, <em>arithmetic expansion</em>, <em>word splitting</em>, and <em>pathname expansion</em>.</p>
|
||
|
|
<p>The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.</p>
|
||
|
|
<p>On systems that can support it, there is an additional expansion available: <em>process substitution</em>. This is performed at the same time as tilde, parameter, variable, and arithmetic expansion and command substitution.</p>
|
||
|
|
<p>Only brace expansion, word splitting, and pathname expansion can change the number of words of the expansion; other expansions expand a single word to a single word. The only exceptions to this are the expansions of "<strong>$@</strong>" and "<strong>${name[@]}</strong>" as explained above (see <strong>PARAMETERS</strong>).</p>
|
||
|
|
<h3 id="brace-expansion">Brace Expansion</h3>
|
||
|
|
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1884" title="Some more Bash tips">1884</a>.</p>
|
||
|
|
<h3 id="tilde-expansion">Tilde Expansion</h3>
|
||
|
|
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>.</p>
|
||
|
|
<h3 id="parameter-expansion">Parameter Expansion</h3>
|
||
|
|
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1648" title="Bash parameter manipulation">1648</a>.</p>
|
||
|
|
<h3 id="command-substitution">Command Substitution</h3>
|
||
|
|
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1903" title="Some further Bash tips">1903</a>.</p>
|
||
|
|
<h3 id="arithmetic-expansion">Arithmetic Expansion</h3>
|
||
|
|
<p>See the notes for HPR show <a href="http://hackerpublicradio.org/eps/hpr1951" title="Some additional Bash tips">1951</a>.</p>
|
||
|
|
<h3 id="process-substitution-1">Process Substitution</h3>
|
||
|
|
<p>Process substitution is supported on systems that support named pipes (FIFOs) or the <strong>/dev/fd</strong> method of naming open files. It takes the form of <(list) or >(list). The process list is run with its input or output connected to a FIFO or some file in <strong>/dev/fd</strong>. The name of this file is passed as an argument to the current command as the result of the expansion. If the >(list) form is used, writing to the file will provide input for list. If the <(list) form is used, the file passed as an argument should be read to obtain the output of list.</p>
|
||
|
|
<p>When available, process substitution is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion.</p>
|
||
|
|
<h3 id="word-splitting-1">Word Splitting</h3>
|
||
|
|
<p>The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for <em>word splitting</em>.</p>
|
||
|
|
<p>The shell treats each character of <strong>IFS</strong> as a delimiter, and splits the results of the other expansions into words using these characters as field terminators. If <strong>IFS</strong> is unset, or its value is exactly <strong><space><tab><newline></strong>, the default, then sequences of <strong><space></strong>, <strong><tab></strong>, and <strong><newline></strong> at the beginning and end of the results of the previous expansions are ignored, and any sequence of <strong>IFS</strong> characters not at the beginning or end serves to delimit words. If <strong>IFS</strong> has a value other than the default, then sequences of the whitespace characters space and tab are ignored at the beginning and end of the word, as long as the whitespace character is in the value of <strong>IFS</strong> (an <strong>IFS</strong> whitespace character). Any character in <strong>IFS</strong> that is not <strong>IFS</strong> whitespace, along with any adjacent <strong>IFS</strong> whitespace characters, delimits a field. A sequence of <strong>IFS</strong> whitespace characters is also treated as a delimiter. If the value of <strong>IFS</strong> is null, no word splitting occurs.</p>
|
||
|
|
<p>Explicit null arguments ("" or '') are retained. Unquoted implicit null arguments, resulting from the expansion of parameters that have no values, are removed. If a parameter with no value is expanded within double quotes, a null argument results and is retained.</p>
|
||
|
|
<p>Note that if no expansion occurs, no splitting is performed.</p>
|
||
|
|
<!--
|
||
|
|
vim: syntax=markdown:ts=8:sw=4:ai:et:tw=78:fo=tcqn:fdm=marker
|
||
|
|
-->
|
||
|
|
<section class="footnotes">
|
||
|
|
<hr />
|
||
|
|
<ol>
|
||
|
|
<li id="fn1"><p>For some reason I said "post decrement" in the audio, where this is obviously a "post <strong>increment</strong>". Oops!<a href="#fnref1">↩</a></p></li>
|
||
|
|
</ol>
|
||
|
|
</section>
|
||
|
|
</article>
|
||
|
|
</main>
|
||
|
|
</div>
|
||
|
|
</body>
|
||
|
|
</html>
|