hpr-knowledge-base/hpr_transcripts/hpr1648.txt

Episode: 1648
Title: HPR1648: Bash parameter manipulation
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr1648/hpr1648.mp3
Transcribed: 2025-10-18 06:19:58

---

It's Wednesday 26 November 2014, this is HPR Episode 1648 entitled Bash Parameter Manitulation
and is part of the series Bash scripting.
It is hosted by Dave Morris' and is about 41 minutes long.
Feedback can be sent to Dave. Morris' at mail.com or by leaving a comment on this episode.
The summary is a summary and aid memoir of Bash Parameter Expansion Methods.
This episode of HPR is brought to you by An Honesthost.com.
Get 15% discount on all shared hosting with the offer code HPR15, that's HPR15.
Get your web hosting that's honest and fair at An Honesthost.com.
Hello, Hacker Public Radio, this is Dave Morris.
Today I'm bringing you a show which I'm calling Bash Parameter Manitulation.
So let me set the scene a little bit.
I'm a keen user of the command line.
I tend to live mostly in the command line or in a web browser, or possibly an editor.
That's largely the command line's largely because that's all there was back in the days
when I first started using computers back in the early 70s.
In fact, in many cases there wasn't even that.
It was punch cards or something like that.
But so I tend to think in terms of commands that I type to a computer a lot of the time.
In those days the commands were pretty primitive, nowhere near as flexible as they are now.
But they did have the concept of being able to put commands in a file which you would
then be able to execute, and that's what we now call a script.
In those days it was called a job command language file or a macro in some context.
Various other names I'm sure that I've not come across.
So the concept of a script of a bunch of commands in a file means that you can effectively
use the command line, the commands that are available to you in the command line as a type
of program.
These things use what they often call variables or parameters to hold data for intermediate
use.
So you'd store something in there and then use it and then discard it.
So what I want to talk to you about today is this business of parameters or variables.
And I want to do that in the context of the bash scripting language.
Bash is the name of one of the unique shells available to you on a Linux system.
It stands for Born Again Shell which is a play on words because it comes from the original
shell SH and it was rewritten by a guy called Born.
So Born Again Shell is a sort of rubbish joke.
Anyway, it's part of the GNU project and it's generally available and it's pretty popular
I would say, certainly the shell that I use.
I guess I'm jumping in somewhere in the middle of bash here.
You might be asking, well, why aren't you talking about bash from the beginning?
Well there are a number of shows that talk about using shells, particularly bash, I think,
in the HBR archive and I've put details of these in the show notes.
I did consider starting a whole series on this and maybe I'll start in the middle
and then begin again at the start or something like that.
I don't know.
It depends how enthusiastic I feel about the whole thing.
The reason I'm doing this particular show is because the bash shell has got some quite
sophisticated features in it currently, the current release which I suspect a lot of people
don't know about.
Obviously I would imagine most advanced bash users do know about these but I wouldn't
class my stuff as particularly advanced and it took me a while to discover these and
to fully understand what they were about.
What I tend to do and I find something like this and need to get my head around is to
write stuff down for my own consumption, maybe something I can stick up on a pinboard
to remind myself, or have a cheat sheet available in a folder that I can look at to remind
me how to use it.
Something that condenses the information down into a minimal space.
What I've done for this show is to give you the two cheat sheets that I've constructed
for this, one of which is really just a lift from the bash man page but I've reformatted
it and put some emphasis in it and so forth and put it into a table, it's probably a
little bit easier to read.
The other thing I've done is to prepare a diagram with examples of these various features.
So I won't be referring to these again but they're part of the bundle that I'm putting
up on HPR to accompany this show so I hope you find them useful.
Otherwise most of what I'm talking about is covered in the show notes themselves.
In fact you might find the show notes going a bit more depth than I speak about because
I don't want to make this show too long to tell you about stuff that you can read.
So I guess it's probably a good idea to start with just looking at what a variable is and
how you work with them in bash.
I imagine most people will know this but it probably does no harm just to quickly whizz
through this.
So a parameter or a variable is a simple thing that holds a value and you might type something
like username equals followed by some string.
The string actually can be enclosed in double quotes or single quotes or it can just be
a bare set of characters.
You need to put quotes around it if you've got spaces in it.
So username equals and then the string to set up a variable called username.
There mustn't be any spaces before or after the equals sign.
That's really important because the bash parser will immediately stop parsing this as a
variable declaration if it finds spaces here.
If you then want to use this parameter that you've called username then you can use it in
an echo command or in various other places.
My example shows it being used in an echo.
You just proceed the name of the variable with a dollar sign and it will, echo will print
out the contents.
So there's an example in the show notes of how you would do this.
There are times when the variable name and the string that you're trying to embed it
in perhaps it's hard to tell where one ends and the next begins and in those cases you
can put curly brackets or otherwise known as braces around the name of the variable after
the dollar sign.
So this would be dollar sign, open curly bracket, username, close curly bracket and that
makes it completely unambiguous.
And you do need these braces when you're doing some of the tricks with variables that I'll
be talking about.
The other type of variable that I'm sure people know about but again it's worth maybe
skipping through fairly rapidly is an array.
It's possible to define an array in bash, array being a structure which contains multiple
pieces of information.
And the simplest form of this is an array which is indexed by integer numbers starting
at zero.
You can either declare this as a bracketed list so my example shows weekdays equals open
bracket.
This is a round bracket.
Trying to be consistent with the naming for these things, I know there are differences
between the UK and the States to how these things are named a parenthesis, we call it.
Open parenthesis, Monday space, Tuesday space, Wednesday etc. throughout the weekdays
and then close parenthesis.
That will declare a variable, an array variable called weekdays which contains elements which
are indexed as zero, one, two, three, four and each of these points to a specific weekday
name.
You can if you wish to declare these things, my second example is a weekend square bracket
zero equals quote Saturday, quote.
What that does is to declare an array called weekend, the first element, the zeroth element
of which is the string Saturday.
So that's a very simple and quick definition of what arrays, how you would create an array.
There are other types of arrays, there's associative arrays which are hash tables which can be indexed
by strings in bash as well but I'm not going to talk about those.
So when you want to refer to an array, here you have to actually use the braces that I
was mentioning before.
So if you wanted to display the contents of the array weekdays index by four, then you
would type echo, space, dollar, brace, opening brace that is weekdays, square bracket four,
close square bracket, close brace.
You can see this in the show notes.
So that would get you that specific element and that would report it.
An entire array can be echoed or printed or used in various contexts and that in that
case the index that you use is an at sign or an asterisk.
They do have different meanings but I won't go into those just now.
So I'm mentioning these because the examples that I'm going to talk about will use simple
variables and arrays.
So one of the things you often see in scripts is maybe setting a variable.
So I've got an example under the heading manipulating parameters where a variable called date
is being set to 2014 hyphen 10 hyphen 27 or enclosed in single quotes.
So you might set it to a value like that just a literal string.
There are other ways that you can set variables to the result of expressions as well.
So the next statement in my example here is a month.
I want to extract the month value from a string like this.
And so that consists of a statement that says month equals.
Now here we have an example of setting a variable month to an expression.
In this particular case it's a little set, it's a it's a couple of commands, a pipeline
of commands which will produce a value and will return it.
And what you often see is a thing like this, dollar sign, open parenthesis, echo dollar
date. So that will cause the date string to be output to stand it out.
And then a pipe symbol, a vertical bar, then you pipe that into the command cut minus F2.
That means get field number two, which is the one, the 10 is 2014 hyphen 10.
And minus D, open quote, hyphen, close quote, close parenthesis.
What that says is cut second field where the fields are delimited by hyphen or dashes
have you like to call them.
So that is a way of doing this and then you'd find that with an echo which says the month
number is dollar month and you would get back 10.
But doing it that way means that you've had to invoke an echo, you've had to invoke
a cut in a pipeline to extract the relevant piece of information.
This has caused a whole bunch of processes to be fired up to do a pretty simple thing
picking a couple of characters out of a string.
Now this is, you can do this in within bash itself without the overhead of creating processes.
And there's a feature called substring expansion within bash where you would specify month equals
dollar, open brace, date, colon five, colon two, close brace, where what that's doing is
it's going to the fifth element starting from zero, indexing the string from zero and
grabbing two characters from that position.
So that sets the variable month to the characters of date from position five, four, two characters
and it achieves the same thing as the echo date cut that I mentioned before and it does
it a lot more efficiently.
Now you may be saying who cares about efficiency, I've got a Intel i7 machine on my desktop
and it's got more power than I ever used.
Why do I care about this sort of thing?
Well, I believe that it's a good practice to be thinking about these things when you're
writing scripts because the day will come when you're trying to write a script to run
on a Raspberry Pi or something even less powerful than a Raspberry Pi.
And you might find that the overhead of running something that creates processes to do
this will be such that it becomes very slow, especially if you're doing it a lot.
And so thinking about the efficiency of these things is quite an important thing to do.
It's certainly part of my history working in the Unix world for a large number of years
back to the days when they were probably a lot more feeble than a Raspberry Pi.
So let's cut to the chase then.
The various functions I'm going to talk about, they have names or at least they have various
titles in the bashman page.
You won't find these in some of the online resources, but I'm using them just as a convenient
way of referring to them.
So let's start with the one that's called use default values.
In this case, when you are substituting a parameter, you're expanding the parameter,
then you can test to see whether it's defined or is no, whether it actually contains a value.
So I've got an example here where the first line says unset name.
So we've got a variable called name, we're unsetting it, we're removing it, we're deleting
it totally, assuming it exists.
This is followed by echo, dollar, open curly bracket, name, colon, hyphen, undefined,
close curly bracket.
What that says is, if name is defined, then echo its value.
If it's not defined, then provide the word undefined.
Now, because we've just unset it, this particular expression will return undefined.
So that's a way of testing whether a value is set or not in a script.
So if we continue and set the value, you've got an example here that sets it to the name Charlie,
then we echo it, then using the same expression as before, then we see the value Charlie returned.
Next we have the one called assigned default values.
Now, this is a similar sort of idea except that it sets the variable to a value.
So I'm using here a variable called page, and I start by unsetting it, make sure that
it is, it doesn't exist.
Then echo, dollar, open brace, page, colon equals, portrait, close brace.
What that's doing is, if the value is not set or it doesn't exist, then set it to the
string portrait.
I'm assuming that this is something using the page orientation value in a script.
So in this particular case, it's not set.
So the result would be returned as portrait.
If we then echo it, so the next line says echo dollar page, then it contains portrait.
So not only was it recognized that it wasn't set, but it was set to that particular value.
So if we now set it to new value, landscape, which is the next line, then do the same
expression again, then the result is landscape, it's not portrait at all, because the test
is it set or is it null, came back as false.
So therefore we just use the value moment and echoing it there after we get landscape.
But that's quite a useful thing to do in a script where you're maybe setting something
from an argument and you want to give it a default if it's not been supplied.
Next we have display error if null or unset.
And this is another test which will produce an error and cause the script that's doing
this test to exit if a parameter is not set.
So this one, the example is an echo dollar open brace length colon question mark, then
you follow that with the error message you want to produce, which in this case I've put
length is unset, close brace.
This will produce an error message which contains somewhere, it will contain a prefix which
talks about where about in the script it's failed, I've not included that in my example.
But you will get the name of the variable length followed by a colon and the error message
that I mentioned length is unset.
This one is called use alternate value and this one checks if the parameter is null or unset.
And if it is, then nothing is substituted otherwise it is.
So the example says fish variables fish equals trout and then we echo dollar open brace
fish colon plus salmon and close brace and the result is we return the value salmon.
So in this particular case the parameter is not null and it's not unset and in this case
we return the value in the in the braces.
But then the next line in the example shows echoing the variable fish and it's still
set at trout.
I must admit I never used that just off the top of my head I can't think of many instances
that I'd want to but due to my lack of imagination more than anything else.
So that's the bundle of functions which test for values and do various things as a consequence.
The next one is called substring expansion and I've mentioned this already briefly.
It takes a variable so you do dollar open brace name of parameter variable colon then
a number then another colon and another number and close brace.
What that's doing is it's the first value is an offset into a string into the variable
and the second one is a length.
If the length part is emitted then it means just all of the string from that particular
point from that from the offset.
A negative offset means to count backwards from the end of the string but because that
would result in the sequence colon minus which we've already seen earlier on you've got
to put colon space minus if you're going to do this thing in order to avoid being misinterpreted
as the default value form.
If you give a negative length then it's not really a length but it means to return the
string between the offset and the position backwards from the end of the string which
is difficult to get your head round up and you can also index sections of arrays with
this stuff.
So here's there's some examples here which I'll go through quickly I mean hopefully you
will be able to follow them yourself.
So if you set a variable animal to equal the string R bar then if we echo dollar open
brace animal colon 4 close brace then what we're saying is display the string from position
4 to the end and remember that these are indexed by index from 0.
So the fourth character the index 4 is the V followed by the rest of the string which
returns a VARC.
If we set a variable message to the string no such file then echoing this message from
position 0 4 7 will pick up the first 7 characters which will be no such.
Taking the same variable message and making the offset minus 4 then you will get the last
4 characters of the string which is file.
And echoing message with an offset of 3 and a negative length of minus 4 will pick up
the word such because 3 is the position of the S of such and minus 4 counts backwards
from the end of the string so it will skip over the four letters of file, reduce such.
If we look at the next example we've set a variable which is an array to equal in parentheses
the names red, orange, yellow, green, blue, indigo, violet, close parentheses and if we
then wish to index elements of this we can do it with dollar open brace colors with an
add in square brackets thereafter so that means that's a way of indicating the entirety
of the array, colon 1, colon 3 that means go to element 1 remember they count from 0 and
return 3 elements so that will return orange, yellow and green.
The next one is using the same colors array dollar open brace colors, open square brackets
at close square brackets colon 5 what that means is go to element 5 which is indigo and
return all the rest of the elements which is be indigo and violet.
This is quite powerful you can do a lot of quite interesting things with this in a script.
The next one names matching prefix never really needed to use this but like I say it's
probably due to my lack of imagination more than anything else.
This is a way of interrogating bash for the various variables that it knows about.
The example I've given here echo dollar open brace exclamation bash capital B a s h
asterisk close brace.
What that's saying is return me all variables whose name begins with b a s h and it's
quite a long list I haven't listed them on it but things like bash bash hops bash pit
these are all variables which if you look at the man page you will find described.
The next example is simply example of creating two variables called coord underscore x and
coord underscore y setting them to values if I then wanted to find out what the names
of these variables were then echo dollar open brace exclamation coord cwr d asterisk close
brace returns the two names that I just mentioned coord x and coord y the next one is something
a bit more useful this is returning a list of array keys it's actually more often array indices
but keys are the more generic term since remember there are there are indexed arrays indexed
numbers and raised indexed by strings which we didn't deal with earlier on so here we're
setting an array colors to equal the three names red green blue and if we then provide
the expression echo dollar open brace exclamation mark colors bracket
excuse me bracket square bracket at closed square bracket close brace then we see the indexes
which 0 1 and 2 the next one is about the length of a parameter and it can also be used with an
array where it returns a number of elements you might have seen something similar to this when
you're looking at the arguments to a script I'm not really dealing with the numbered arguments
in this talk so what we have here in in terms of example is a variable veg vg equals set being set
to the string broccoli if we then echo dollar open brace hash mark pan sign for americans v e g
close brace we get back length of the string which is eight now the next example is a case
where we are setting an array array dt equal to the result of a command and so if I just read this
out to you dt equals open parenthesis then follow that with dollar open parenthesis again date
close parenthesis close parenthesis what that's doing is the the inner dollar
parenthesis date invokes the date command which returns a string which which consists of the
various bits of the date the day that day number and the year and so on so forth forget the actual
order but you can try it yourself and see that's the whole thing is enclosed in parenthesis which
is a way of declaring an array so what happens is each of the separate pieces the six fields
that are returned by a date are dropped into this array because of spaces that is the
delimiter that bash expects so if we then echo dollar open brace hash sign dt then open square bracket
at close square brackets close brace that's again just remember this this defines it as an array
and we're talking about the entirety of the array then the answer comes back as six so it's a useful
way of finding out how long an array is now we're coming on to some more interesting features for
manipulating the internals of strings and arrays this one's called remove matching prefix pattern
so this as you might gather removes characters from the front of a string the the general format
is the name of the variable followed by a hash sign followed by a pattern that is to be removed
now the pattern that's mentioned is similar to the sort of patterns you can use when you're
searching for files on the command line in things like ls or whatever so it can contain an
asterisk meaning an arbitrary number of characters for example now in my examples I start off with
setting a variable called dir dir equals to a path a file path in which which case is slash
home slash Dave slash some slash dir so if I then echo dollar open brace dir the variable name hash
slash home slash Dave slash close brace what that's saying is from this variable from this string
remove slash home slash Dave slash from the front not too surprisingly we get back the string
some slash dir if we did this differently and use the expression dollar open brace dir hash
star slash close brace what that's saying is the first number of any characters that are not
a slash followed by a slash to be removed from the front well there are no characters other than
a slash on the front so it just takes the first slash off and it returns home without et cetera
without the leading slash if on the other hand we use two hashes which is the other the other
capability of this particular function then we would use dollar dir hash hash star
asterisk I mean slash close brace then what that will do it will return it will remove all of the
strings followed by a slash until around a more so that will remove everything until the last
component of this path name so this is similar to the built-in base name command that's available in
bash and it's it's a more efficient way of achieving the same same thing as base name personally
I prefer to use it for the reasons I mentioned before if we have an array and I've got an
example where I'm setting an array colors are equal to red the name the red green and blue but
each name has got an underscore on the front of it so if I then want to strip these off then I
can set color and save them back into the array I can set colors equals open parenthesis dollar
open brace colors open square brackets at sign close where brackets hash underscore close brace
close parentheses so what that's doing is defining colors as a new array array definition
remembers in parentheses and inside this array definition is this substitution pattern which is
dollar open brace colors blah blah blah hash underscore what that's saying is process each
element of the array as it stands at the moment strip off the underscore from the front of each
character and drop it back into the array so if you then echoed the the array this example here you
will find red green and blue being returned without the underscores on the front so this was that
was the prefix that was stripping prefixes then there's similar one that removes suffixes from
the the ends of strings or arrays this one also uses a pattern which should which can contain
asterisks and other things but I'm not going into details of that just now but the difference is
that instead of the hash it uses a percent sign so the the variable DIR is being set to the same
things before and if we then echo dollar open brace DIR percent slash some slash DIR close brace
what that means is strip off slash some slash DIR from the end of it and not too surprisingly what
we returned is slash home slash Dave if on the other hand we echo the the expression dollar open
brace DIR percent slash star close brace then what that's saying is remove from the end a slash
followed by arbitrary commands arbitrary characters I mean so that will strip off the slash DIR
from the end giving you slash home slash Dave slash some if on the other hand we wanted to
remove more from the end of the thing and do it in in using the double percent which the
quimp the double hash dollar open brace DIR percent percent slash slump some slash star close brace
that will remove slash some slash DIR from the end of it leaving you the string slash home slash Dave
so using this slash star business slash asterisks if you prefer then you have a similar effect to
the DIR name command that is built into debash so I gave another example not using arrays this time
where it's a quite common thing to do where you you strip off the extension of a file name so
I've got a variable called file name which I'm setting to great underscore view dot jjpg so what we
want to do is to change its name from dot jpg dot png so that can be done by setting file name
equal to and then a string in double quotes dollar open brace file name percent dot jpg close brace
dot png so what that will do is it will take file name it will strip off the dot jpg as we've seen
from the end and then it will append to the what's left the string dot png so the result will be
great underscore view dot png that's that can be a pretty useful thing to do in fact I use it
sometimes it using image magic they're the convert command in image magic so you can change the
format of a file from jpg to png using using that sort of expression in the command so the next one
and we're getting close to the end now is patent substitution now this is quite a powerful
facility that lets you make substitution within a string and the the general syntax is that the
name of the the variable is followed by a slash and a string a pattern followed by another slash
and the name thing that is that the string that is to replace it so if we take my example then I'm
setting a message to the wonderful string and and is and and don't know what possess me to do
that but there you go then we echo dollar open brace msg slash and slash insect close brace then
it will return the string and insect is an ant okay now that's pretty trivial if you use a percent
sign following the slash then it matches at the end of the parameter so if you can have slash
percent which means which means something completely different than just a straight slash when
you're doing the pattern matching so I've got an example here where dollar msg slash percent
and slash insect close brace doesn't change it to an insect as an ant but it changes it to an
ant is an insect so it's changed the last instance of ant to insect rather than the first then if you
have if you use double slashes after the name of the the variable then it changes all instances
of the pattern throughout the string so I've got dollar open brace msg slash slash and slash insect
close brace and it changes the string to an insect as an insect it's changed both instances of ant
you can do similar things with arrays as well and there's an example in the show notes which I won't
go into because I think I've gone on a bit too long as it is the final one then is case modification
so you can change the case of characters in a in a string in this particular case in this
sorry that's a bad pun in this instance you use an an up arrow symbol to mean to which means
change letters to uppercase I was just trying to remember what that up arrow thing is called
but I can't remember and use a comma to indicate lowercase so the there's various examples
showing how you can change the first character in a lowercase string to uppercase or you can
change all of the characters to uppercase and you can also do specific pattern matches throughout
the string I used a pattern including a square bracket a square bracketed list of characters to do
character matching hopefully you'll be able to work that out from the example this is quite
lengthy so I won't go into into great details here so that's the list and that's it and I hope
you find it useful what I should what I would have liked to have done was to have given you some
examples of using it but of using these various features but I didn't do that because I thought
this is getting a little bit too complex anyway but if you look back at one of my earlier shows show
number one two oh four about some scripts I wrote to download music from the magnitude website
in as part of that I produced a bunch of scripts which were put up on the guitarist website and
give links in there to these bash scripts they're all written in bash in those you will find that
I've used a fair number of these things and there's they're also annotated there's quite a lot
of detail explaining what the various parts of these scripts do so if you're up to poking around with
the the guitarist site you might find some stuff there that's quite useful okay that's it hope you
enjoyed it you've been listening to hecka public radio at hecka public radio dot org we are a
community podcast network that releases shows every weekday Monday through Friday today show
like all our shows was contributed by an hbr listener like yourself if you ever thought of
recording a podcast then click on our contributing to find out how easy it really is hecka public
radio was founded by the digital dog pound and the infonomicon computer club and it's part of
the binary revolution at binrev.com if you have comments on today's show please email the host
directly leave a comment on the website or record a follow-up episode yourself unless otherwise
stated today's show is released under creative comments attribution share a light 3.0 license