241 lines
16 KiB
HTML
241 lines
16 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html>
|
||
|
|
<head>
|
||
|
|
<meta charset="utf-8">
|
||
|
|
<meta name="generator" content="pandoc">
|
||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
|
||
|
|
<title></title>
|
||
|
|
<style type="text/css">code{white-space: pre;}</style>
|
||
|
|
<!--[if lt IE 9]>
|
||
|
|
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
|
||
|
|
<![endif]-->
|
||
|
|
</head>
|
||
|
|
<body>
|
||
|
|
<h1 id="hpr1648---full-show-notes">HPR1648 - full show notes</h1>
|
||
|
|
<ul>
|
||
|
|
<li>Title: Bash parameter manipulation</li>
|
||
|
|
<li>Host: Dave Morriss</li>
|
||
|
|
</ul>
|
||
|
|
<h2 id="bash-parameter-manipulation">Bash parameter manipulation</h2>
|
||
|
|
<p>I'm a great fan of using the Linux command line and enjoy writing shell scripts using the Bash shell.</p>
|
||
|
|
<ul>
|
||
|
|
<li><p><em>BASH</em> (or more usually <em>Bash</em> or <em>bash</em>) is the name of a <a href="http://en.wikipedia.org/wiki/Unix_shell">Unix shell</a>. The name stands for <em>Bourne Again SHell</em>, which is a play on words. Bash is an extension of the shell originally written by <a href="http://en.wikipedia.org/wiki/Stephen_Richard_Bourne">Stephen Bourne</a> in 1978, usually known as <em>SH</em>.</p></li>
|
||
|
|
<li><p><a href="http://en.wikipedia.org/wiki/Bash_%28Unix_shell%29">Bash</a> was written as part of the <a href="http://en.wikipedia.org/wiki/GNU_Project">GNU Project</a> which forms part of the Linux Operating System.</p></li>
|
||
|
|
<li><p>A shell is the part of the operating system that interprets commands, more commonly known as the command line.</p></li>
|
||
|
|
<li><p>A knowledge of Bash is very helpful if you would like to be able to use the power of the command line. It is also the way to learn how to build Bash scripts for automating the tasks you need to perform.</p></li>
|
||
|
|
</ul>
|
||
|
|
<p>In this episode we look at what parameters are in Bash, and how they can be created and manipulated. There are many features in Bash that you can use to do this, but they are not easy to find.</p>
|
||
|
|
<p>As I was learning my way around Bash it took me a while to find these. Once I had found them I wanted to make a "cheat sheet" I could stick on the wall to remind me how to do things. I am sharing the result of this process with you.</p>
|
||
|
|
<p>The version of Bash which I used for this episode is <em>4.3.30(1)-release</em></p>
|
||
|
|
<h3 id="what-is-a-parameter">What is a parameter?</h3>
|
||
|
|
<p>A Bash parameter, more commonly referred to as a variable, is a named item which holds a value. Parameters are created thus:</p>
|
||
|
|
<pre><code>username='droog'</code></pre>
|
||
|
|
<p>There should be no spaces before or after the '=' sign. The parameter called <code>username</code> now contains the string 'droog'.</p>
|
||
|
|
<p>To use the contents of <code>username</code> the name should be prefixed with a dollar ($) sign as in:</p>
|
||
|
|
<pre><code>echo "Your username is $username"
|
||
|
|
-> Your username is droog</code></pre>
|
||
|
|
<p>The line beginning '<strong>-></strong>' is what will be generated by the above statement. I will be using this method of signifying output through these notes.</p>
|
||
|
|
<p>An alternative way of referring to the contents of a parameter is to enclose it with curly brackets (braces) after the dollar sign. This makes the variable name unambiguous when there is a possibility of misinterpretation</p>
|
||
|
|
<h3 id="arrays">Arrays</h3>
|
||
|
|
<p>As well as the simple parameters seen so far, Bash also provides <em>arrays</em>. The simplest form is indexed by integer numbers, starting with zero and is defined either as a bracketed list:</p>
|
||
|
|
<pre><code>weekdays=(monday tuesday wednesday thursday friday)</code></pre>
|
||
|
|
<p>or individually as indexed elements:</p>
|
||
|
|
<pre><code>weekend[0]='saturday'
|
||
|
|
weekend[1]='sunday'</code></pre>
|
||
|
|
<p>There is a lot more to arrays in Bash than this, as well as there being <em>associative arrays</em> indexed by strings, but we will leave them for another time.</p>
|
||
|
|
<p>Referring to arrays is achieved using curly braces and indices:</p>
|
||
|
|
<pre><code>echo ${weekdays[4]}
|
||
|
|
-> friday</code></pre>
|
||
|
|
<p>The entire array can be referenced with '@' or '*' as an index:</p>
|
||
|
|
<pre><code>echo ${weekdays[@]}
|
||
|
|
-> monday tuesday wednesday thursday friday</code></pre>
|
||
|
|
<p>Knowing this much about arrays is necessary to understand the following parameter manipulation expressions.</p>
|
||
|
|
<h3 id="manipulating-parameters">Manipulating parameters</h3>
|
||
|
|
<p>If you want to change the value of a parameter there are many ways of doing it. A lot of scripts you may encounter might use <em>sed</em>, <em>awk</em> or <em>cut</em> to do this. For example, you might see this:</p>
|
||
|
|
<pre><code>date='2014-10-27'
|
||
|
|
month=$(echo $date | cut -f2 -d'-')
|
||
|
|
echo "The month number is $month"
|
||
|
|
-> The month number is 10</code></pre>
|
||
|
|
<p>Here <em>cut</em> was used to do the job of extracting the '10' from the contents of <code>date</code>. However, this is inefficient since it causes a whole new process to be run just to do this simple thing. Bash contains the facilities to do this all by itself. Here's how, using <em>substring expansion</em>:</p>
|
||
|
|
<pre><code>month=${date:5:2}</code></pre>
|
||
|
|
<p>Variable <code>month</code> is set to the characters of <code>date</code> from position 5 for 2 characters (the position is zero based by the way).</p>
|
||
|
|
<h3 id="parameter-manipulation-features">Parameter manipulation features</h3>
|
||
|
|
<p>I have demonstrated each of these briefly. Included with these notes are two other resources: the relevant text from the bash man page and some examples in diagrammatic form. Both are PDF files generated from LibreOffice.</p>
|
||
|
|
<p>The man page extract was originally made for my own benefit as a cheat sheet to help to remind me how to use these features. If it benefits you then great. If not then no problem.</p>
|
||
|
|
<p>The diagram was also meant for me to place on a pin-board over my desk, so it includes colour and seems a little more friendly. If you like it then you're welcome.</p>
|
||
|
|
<p>I have also included the examples below, with a little more explanation. I hope it helps.</p>
|
||
|
|
<h4 id="use-default-values">Use default values</h4>
|
||
|
|
<p>Returns the value of the parameter unless it's not defined or is null, in which case the default is returned:</p>
|
||
|
|
<pre><code>unset name
|
||
|
|
echo ${name:-Undefined}
|
||
|
|
-> Undefined
|
||
|
|
name="Charlie"
|
||
|
|
echo ${name:-Undefined}
|
||
|
|
-> Charlie</code></pre>
|
||
|
|
<h4 id="assign-default-values">Assign default values</h4>
|
||
|
|
<p>Set the value of the parameter if it is undefined or null:</p>
|
||
|
|
<pre><code>unset page
|
||
|
|
echo ${page:=portrait}
|
||
|
|
-> portrait
|
||
|
|
echo $page
|
||
|
|
-> portrait
|
||
|
|
page="landscape"
|
||
|
|
echo ${page:=portrait}
|
||
|
|
-> landscape
|
||
|
|
echo $page
|
||
|
|
-> landscape</code></pre>
|
||
|
|
<h4 id="display-error-if-null-or-unset">Display Error if Null or Unset</h4>
|
||
|
|
<p>Displays an error and causes the enclosing script to exit if the parameter is not set (the error message contains details of where in the script the problem occurred):</p>
|
||
|
|
<pre><code>echo ${length:?length is unset}
|
||
|
|
-> ... length: length is unset</code></pre>
|
||
|
|
<h4 id="use-alternate-value">Use Alternate Value</h4>
|
||
|
|
<p>If the parameter is null or unset then nothing is substituted.</p>
|
||
|
|
<pre><code>fish="trout"
|
||
|
|
echo ${fish:+salmon}
|
||
|
|
-> salmon
|
||
|
|
echo $fish
|
||
|
|
-> trout</code></pre>
|
||
|
|
<h4 id="substring-expansion">Substring Expansion</h4>
|
||
|
|
<p>This one is quite complex. The first value is an offset and the second a length. If the length is omitted then the rest of the string is returned.</p>
|
||
|
|
<p>A negative offset means to count backwards from the end of the string. Note that the sign must be preceded by a space to avoid being misinterpreted as the <em>default value</em> form.</p>
|
||
|
|
<p>A negative length is not really a length, but means to return the string between the offset and the position backwards from the end of the string.</p>
|
||
|
|
<p>Sections of arrays may also be indexed with this expression. The offset is an offset into the elements of the array and the length is a count of elements.</p>
|
||
|
|
<pre><code>animal="aardvark"
|
||
|
|
echo ${animal:4}
|
||
|
|
-> vark
|
||
|
|
message="No such file"
|
||
|
|
echo ${message:0:7}
|
||
|
|
-> No such
|
||
|
|
echo ${message: -4}
|
||
|
|
-> file
|
||
|
|
echo ${message:3:-4}
|
||
|
|
-> such
|
||
|
|
|
||
|
|
colours=(red orange yellow green blue indigo violet)
|
||
|
|
echo ${colours[@]:1:3}
|
||
|
|
-> orange yellow green
|
||
|
|
echo ${colours[@]:5}
|
||
|
|
-> indigo violet</code></pre>
|
||
|
|
<h4 id="names-matching-prefix">Names matching prefix</h4>
|
||
|
|
<p>This is for reporting names of variables. We do not show all the names in the examples below.</p>
|
||
|
|
<pre><code>echo ${!BASH*}
|
||
|
|
-> BASH BASHOPTS BASHPID ...
|
||
|
|
export coord_x=42
|
||
|
|
export coord_y=100
|
||
|
|
echo ${!coord*}
|
||
|
|
-> coord_x coord_y</code></pre>
|
||
|
|
<h4 id="list-of-array-keys">List of array keys</h4>
|
||
|
|
<p>Lists the array indices (more generally <em>keys</em>) in an array.</p>
|
||
|
|
<pre><code>colours=( red green blue )
|
||
|
|
echo ${!colours[@]}
|
||
|
|
-> 0 1 2</code></pre>
|
||
|
|
<h4 id="parameter-length">Parameter length</h4>
|
||
|
|
<p>Shows the length of a parameter. If used with an array it returns the number of elements.</p>
|
||
|
|
<p>Note the second example below saves the result of the <em>date</em> command in an array <code>dt</code>. There are 6 fields separated by spaces, so the array element count reflects this.</p>
|
||
|
|
<pre><code>veg="Broccoli"
|
||
|
|
echo ${#veg}
|
||
|
|
-> 8
|
||
|
|
dt=($(date))
|
||
|
|
echo ${#dt[@]}
|
||
|
|
-> 6</code></pre>
|
||
|
|
<h4 id="remove-matching-prefix-pattern">Remove matching prefix pattern</h4>
|
||
|
|
<p>This removes characters from the front of a string. The pattern used can contain an asterisk ('*') meaning an arbitrary number of characters. If two hash characters ('#') are used the longest match is removed.</p>
|
||
|
|
<p>So, in the first examples '*/' with one hash just removes the leading '/', but with two hashes everything up to and including the last '/' is removed. This is equivalent to the built-in <code>basename</code> command.</p>
|
||
|
|
<p>When applied to an array every element can be trimmed as shown in the second example. Note that here we are saving the result of trimming the leading '_' back into the array.</p>
|
||
|
|
<pre><code>dir="/home/dave/some/dir"
|
||
|
|
echo ${dir#/home/dave/}
|
||
|
|
-> some/dir
|
||
|
|
echo ${dir#*/}
|
||
|
|
-> home/dave/some/dir
|
||
|
|
echo ${dir##*/}
|
||
|
|
-> dir
|
||
|
|
|
||
|
|
colours=(_red _green _blue)
|
||
|
|
colours=(${colours[@]#_})
|
||
|
|
echo ${colours[@]}
|
||
|
|
-> red green blue</code></pre>
|
||
|
|
<h4 id="remove-matching-suffix-pattern">Remove matching suffix pattern</h4>
|
||
|
|
<p>This feature is similar to the previous one and removes characters from the end of a string. The pattern used to determine what to remove is the same as before, and the use of double '%' characters makes the deletion affect the maximum number of characters.</p>
|
||
|
|
<p>Note that using '/*' in the examples has an effect similar to the <code>dirname</code> command.</p>
|
||
|
|
<p>A common use is shown in the second example where the extension of a filename is deleted and replaced.</p>
|
||
|
|
<pre><code>dir="/home/dave/some/dir"
|
||
|
|
echo ${dir%/some/dir}
|
||
|
|
-> /home/dave
|
||
|
|
echo ${dir%/*}
|
||
|
|
-> /home/dave/some
|
||
|
|
echo ${dir%%/some/*}
|
||
|
|
-> /home/dave
|
||
|
|
|
||
|
|
filename='great_view.jpg'
|
||
|
|
filename="${filename%.jpg}.png"
|
||
|
|
echo $filename
|
||
|
|
-> great_view.png</code></pre>
|
||
|
|
<h4 id="pattern-substitution">Pattern substitution</h4>
|
||
|
|
<p>This feature permits quite sophisticated changes to be made to a string or an array. The first part after the first '/' is the pattern to match, and can contain '*' as described before. See the <a href="http://wiki.bash-hackers.org/syntax/pe">Bash Hackers</a> site for a very good explanation of this. The second string is what is to replace the target.</p>
|
||
|
|
<p>If two '/' characters follow the parameter name then all matches that are found are replaced.</p>
|
||
|
|
<p>If the pattern string begins with a '#' then it must match at the start of the parameter, however if the pattern string begins with a '%' then it must match at the end of the parameter.</p>
|
||
|
|
<pre><code>msg='An ant is an ant'
|
||
|
|
echo ${msg/ant/insect}
|
||
|
|
-> An insect is an ant
|
||
|
|
echo ${msg/%ant/insect}
|
||
|
|
-> An ant is an insect
|
||
|
|
echo ${msg//ant/insect}
|
||
|
|
-> An insect is an insect
|
||
|
|
|
||
|
|
colours=(red green blue)
|
||
|
|
echo ${colours[@]/green/yellow}
|
||
|
|
-> red yellow blue
|
||
|
|
echo ${colours[@]/#/_}
|
||
|
|
-> _red _green _blue</code></pre>
|
||
|
|
<h4 id="case-modification">Case modification</h4>
|
||
|
|
<p>Finally, this feature changes the case of letters. The '^' symbol makes matching letters into uppercase, and ',' converts to lowercase. A single '^' or ',' after the parameter name makes one change, whereas doubling this symbol changes every matching character.</p>
|
||
|
|
<p>Note the use of a pattern enclosed in square brackets matches the enclosed letters. Adding a '^' to the start of this list inverts the matching effect as seen below.</p>
|
||
|
|
<pre><code>msg='the quick brown fox'
|
||
|
|
echo ${msg^}
|
||
|
|
-> The quick brown fox
|
||
|
|
echo ${msg^^}
|
||
|
|
-> THE QUICK BROWN FOX
|
||
|
|
echo ${msg^^o}
|
||
|
|
-> the quick brOwn fOx
|
||
|
|
echo ${msg^^[tqbf]}
|
||
|
|
-> The Quick Brown Fox
|
||
|
|
echo ${msg^^[^tqbf]}
|
||
|
|
-> tHE qUICK bROWN fOX</code></pre>
|
||
|
|
<h2 id="links">Links</h2>
|
||
|
|
<ul>
|
||
|
|
<li>Definitions:
|
||
|
|
<ul>
|
||
|
|
<li>Unix shell <a href="http://en.wikipedia.org/wiki/Unix_shell">http://en.wikipedia.org/wiki/Unix_shell</a></li>
|
||
|
|
<li>Bash <a href="http://en.wikipedia.org/wiki/Bash_%28Unix_shell%29">http://en.wikipedia.org/wiki/Bash_%28Unix_shell%29</a></li>
|
||
|
|
<li>GNU Project <a href="http://en.wikipedia.org/wiki/GNU_Project">http://en.wikipedia.org/wiki/GNU_Project</a></li>
|
||
|
|
</ul></li>
|
||
|
|
<li>References
|
||
|
|
<ul>
|
||
|
|
<li>Shell Parameter Expansion <a href="http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html">http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html</a></li>
|
||
|
|
<li>Bash Hackers on parameter expansion <a href="http://wiki.bash-hackers.org/syntax/pe">http://wiki.bash-hackers.org/syntax/pe</a></li>
|
||
|
|
</ul></li>
|
||
|
|
<li>Previous HPR shows on the shell or shell scripting:
|
||
|
|
<ul>
|
||
|
|
<li>2008-03-03 HPR0045: Shell Scripting (dosman) <a href="http://hackerpublicradio.org/eps/hpr0045">http://hackerpublicradio.org/eps/hpr0045</a></li>
|
||
|
|
<li>2008-03-12 HPR0052: UCLUG: Newbie Shell Scripting (Dave Yates) <a href="http://hackerpublicradio.org/eps/hpr0052">http://hackerpublicradio.org/eps/hpr0052</a></li>
|
||
|
|
<li>2010-03-24 HPR0531: bash loops (Ken Fallon) <a href="http://hackerpublicradio.org/eps/hpr0531">http://hackerpublicradio.org/eps/hpr0531</a></li>
|
||
|
|
<li>2010-08-11 HPR0562: Introduction to bash scripting (Ken Fallon) <a href="http://hackerpublicradio.org/eps/hpr0562">http://hackerpublicradio.org/eps/hpr0562</a></li>
|
||
|
|
<li>2010-11-17 HPR0598: Bash Scripting: Episode 2 Command Line Basics (Ken Fallon) <a href="http://hackerpublicradio.org/eps/hpr0598">http://hackerpublicradio.org/eps/hpr0598</a></li>
|
||
|
|
<li>2012-05-22 HPR0992: Linux In The Shell 007 - Chmod and Unix Permissions. (Dann) <a href="http://hackerpublicradio.org/eps/hpr0992">http://hackerpublicradio.org/eps/hpr0992</a></li>
|
||
|
|
<li>2012-06-05 HPR1002: Linux In The Shell 008 - free: Understanding Linux Memory Usage (Dann) <a href="http://hackerpublicradio.org/eps/hpr1002">http://hackerpublicradio.org/eps/hpr1002</a></li>
|
||
|
|
<li>2013-03-05 HPR1197: What I do with bash scripts (Jon Kulp) <a href="http://hackerpublicradio.org/eps/hpr1197">http://hackerpublicradio.org/eps/hpr1197</a></li>
|
||
|
|
<li>2013-04-09 HPR1222: LiTS 027: mathematical commands (Dann) <a href="http://hackerpublicradio.org/eps/hpr1222">http://hackerpublicradio.org/eps/hpr1222</a></li>
|
||
|
|
<li>2013-05-14 HPR1247: Recording Terrestrial Radio with bash scipts and cron jobs (Jon Kulp) <a href="http://hackerpublicradio.org/eps/hpr1247">http://hackerpublicradio.org/eps/hpr1247</a></li>
|
||
|
|
<li>2013-05-22 HPR1253: Linux in the Shell Ep 30 - vmstat (Dann) <a href="http://hackerpublicradio.org/eps/hpr1253">http://hackerpublicradio.org/eps/hpr1253</a></li>
|
||
|
|
</ul></li>
|
||
|
|
<li>Resources
|
||
|
|
<ul>
|
||
|
|
<li>Full show notes <a href="hpr1648_full_shownotes.html">hpr1648_full_shownotes.html</a></li>
|
||
|
|
<li>Bash man page extract <a href="hpr1648_summary.pdf">hpr1648_summary.pdf</a></li>
|
||
|
|
<li>Parameter manipulation "<em>cheat sheet</em>" <a href="hpr1648_diagram.pdf">hpr1648_diagram.pdf</a></li>
|
||
|
|
<li>Various annotated Bash scripts which make up my Magnatune Downloader (described in show <a href="http://hackerpublicradio.org/eps/hpr1204">hpr1204</a>) <a href="https://gitorious.org/magnatune-downloader/magnatune-downloader/source/master:">https://gitorious.org/magnatune-downloader/magnatune-downloader/source/master:</a></li>
|
||
|
|
</ul></li>
|
||
|
|
</ul>
|
||
|
|
</body>
|
||
|
|
</html>
|