hpr-knowledge-base/hpr_transcripts/hpr4201.txt

Episode: 4201
Title: HPR4201: Today I learnt (2024-08-23)
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4201/hpr4201.mp3
Transcribed: 2025-10-25 21:17:55

---

This is Hacker Public Radio Episode 4200 and won for Monday the 9th of September 2024.
Today's show is entitled, Today Alert 2024-823.
It is part of the series today I learned.
It is the 160th show of Dave Morris and is about 22 minutes long.
It carries an explicit flag.
The summary is, some random technical items this time.
Hello everybody, my name is Dave Morris and welcome to Hacker Public Radio.
My show today is one in the series Today I Learned and this time I'm going a bit technical
rather than to random sciencey bits and stuff.
But I've restricted it to two items because I've gone into a modern amount of depth on
each one.
The first one is about the date command, which is a stand-alone program from the GNU origin.
And I discovered a thing this time in the past few weeks that I thought I should share.
So I'm working on a project following on from the time when we had to move all of
the HBR stuff off the original server that we'd been on for years and onto the current
one.
And also at the same time we upgraded the way things are done from a PHP site to a static
site.
Unfortunately during that move some things got overlooked and can get properly transferred
over.
The process of fixing that all up, yeah it's about a year late but it's getting done.
So as I make changes I want to be able to check they're correct.
But to do that I need to see what, see the effects of what I've done on a given show.
And I need there to have been a static site update.
Now I can actually do that in a local copy of the site.
But it gets quite laborious and it's better if we just run regular static site updates.
Ken's been away in the past couple of weeks or so and during that time he set up a
cron job to refresh the site every three hours.
Each show page shows at the top a refresh time and it's in UTC form.
So being a bit numerically challenged I wanted a way of computing the next refresh time
in my time zone from the previous refresh time from that UTC time.
Now the Gnu date command accepts a date and time expression after the hyphen D option
or the alternative is hyphen, hyphen date.
Both of those followed by strings that hyphen, hyphen date is equals string.
That string is some sort of date expression.
So it could be a plane date in the past or in the future and you want to report something
about that what day of the week it is or something like that.
So the contents of this string very flexible but quite complex.
You can include time zone data, offsets, date, day in month names and so forth.
And there's, there are links at the end of this particular section to the Gnu manual
that explains all this.
Actually it doesn't explain what I discovered but maybe that's something I should have worked
out for myself by anyway.
But the first thing I did was to type in a update time from a show to the date command.
So the actual command is date space, hyphen D space then in quotes 16 colon 27 colon 16.
So it was 27 minutes past four in the afternoon then still in this string plus a plus sign
three hours.
So then the close quote and the argument, the main argument to date which is a thing that
defines the format of the date that you want is in this case plus percent capital T.
Now this is a request for the time in a standard form from the date that's being worked
on which which if you don't put the hyphen D business in is the current date.
So I've got back 15, 27, 16.
So started with 1600 and got back a date, a time and hour earlier.
It looked like three hours addition.
So I wasn't clear why this felt and I haven't really found a definitive answer to it but
the the GNU code inside the date command which pauses this stuff is obviously confused.
It's also quite complex and has had fair bit of work done on it because it's very flexible
and what it can do but obviously this is one thing that confused it.
So I tried adding the time zone after the time and so my date command which is both of
these are in the in the notes of course.
So 16, 27, 16 and then I put UTC immediately after it with a space actually.
It doesn't matter you can put it with all with that space plus three hours and I get back
a time of 20, 27, 16 so that seems better except hang on there's four hours difference
between 16, 27 and 20, 27.
Well that's because the time that comes back is a local time for where I am.
I'm in the UK and we're currently in our day life saving time thing annoying as it is called
British summertime BST which is UTC plus one hour.
So that's why the extra hour because this confuses me.
I'm not going to be able to look at that and say, oh, 20, 27, no, it's actually 19, 27.
So what I did then was to add in another option to date, dates got a lot of options.
It's quite an amazing command actually.
The option I added was hyphen under lowercase U. The D was a lowercase D and I've never
said that.
I got back 19, 27.
So that's the British standard time for my the next update based on the previous update.
So I put in here that what I ended up doing.
I run one of these pop up terminal thingies.
This is the Yakua K or some some weird name, I'm not sure what that means.
I've got it set so I hit F12 and it pops up.
So what I do there is at the start of the day, I type in the start time, assuming I'm
working on this thing all day which I have been I'm having a break just now.
So I type in current equals and then time in single quotes.
That's saying a bash variable to that time.
Then I do an expression where I set a new another variable next equal and then I give
it a command substitution so that's dollar open parenthesis.
Then that date or modification of that date expression where after the hyphen lowercase
D I've got in double quotes because I'm substituting into it.
Dollar, curly bracket, curly brace or it's called it brace yeah, current, close brace
UTC plus three hours and I actually add three minutes because it takes a few minutes
for the actual update to run.
I don't know why I did that but it seems to be it's close enough anyway.
So that whole expression with the plus percent T at the enclosed parenthesis then there's
a semicolon because I'm putting several commands in this line two actually.
I then echo what the contents of next will be followed by this is all in double quotes
UTC in a slash and then I do another date computation where I take the next value and substitute
it into the string after a hyphen lowercase D and follow that work via string UTC and
then in this case I put out the time in the format of percent capital T space percent
Z. The Z returns the the nice form of the time zone so that in the abbreviation rather
than an hours offset type thing.
So what I get back from that starting at 627 and 55 seconds is 930 55 UTC 1030
55 VST so I get that.
So then the next iteration I simply set current to next and run the thing again and I get
1233 for the next one and then do it again and get 1536 for the next one.
It's sliding a bit because of my extra three minutes I've added each time but I'm just
recalling the same commands over and over again should have been a script really but
I in my long and verbose way of doing it I'm being lazy if she believed that.
So yeah maybe one day I'll turn into a script but it does what I want just now so I can
say okay I'll rush to finish this one and upload it to the database and then the static
site will run and it will show me whether it worked or not and then sometimes it doesn't
so I have to do it again and wait another three hours to see whether it works but I'm
doing a lot of these as many as I can but I'm with so trouble me nuts.
I'm trying to do five a day sometimes they're quite quite detailed and difficult so that's
what I'm doing the links point to the GNU core utils manual and it shows the levels you
go through to get to the date part the date format input format so that's all about
how you specify this business of taking a date or a time and adding things to it and there's
some examples I find it quite useful and I had never dug that deeply into date before so I found
it quite a helpful thing to do and I've also got some references to these these places to go
and look next time I stumble over some date related things so today I learned number two is another
command it's another GNU command and it's about merging lines of files with paste so what am I
doing this all about well while I was doing my show repair business I came across a need to
generate a list of show numbers separated by comments so I do that moderately often it comes in
useful in reports it comes I have scripts that like that I've programmed perl scripts which
I've arranged so that they will take a list of numbers you can give them a start number in it
and account we can give a start number in an end number or you can give them a list of
comma separated numbers in order for them to do whatever's needed to's particular shows that's
quite useful and it's catered for my for the demands I have to deal with in the past I've done
things like load the values I need into a bash array then I use all sorts of different methods
you can effectively print them out or echo them out and along the way you can add commas to each
element the trouble with that is that it applies to the comma to the end of every element including
the last one it'd been quite nice if you could join them together in some way without
comma on the end and there's often a join command in various scripting languages that does that
and it always puts them in in between things will not the end so how to do this in bash in simpler
way so I stumbled upon this command paste another one of these standalone canoe commands and it's
from the canoe core utils bunch of commands you might do well to go and have a look at those if
you're interested in bash related things because there's lots of it be a whole bunch of things
you already know I'm sure but there's there's quite a few odds and ends that can be quite useful
once you understand them and this is what I'm doing here I've never I knew that paste exists but I
never worked out what it would do and didn't realize it would solve this problem so what paste does
it takes a bunch of options not all that many and then it takes an arbitrary number of files from
one to two n sure there's probably a disadvantage going too high with that number but nevertheless
and what it does by default it merges lines from these files and it takes a corresponding line from
each of the files in the argument list and then displays them separated by tabs and writes them
to stand it out put they can catch them in a file give it two or three it'll give you the first
line of first file first line second file first line of third file or can come catanate it together
separated by tabs then it'll do the second line of each and blah blah blah you can make the the
file arguments a hyphen or at least one of them anyway and what happens then is that it uses
standard output which you have you will have possibly supplied with the output from some program
or other and then piped to paste so you can do more than one of these hyphens but I'm not going to
go into that because it gets a little bit more complicated than I can explain well I think
the delimiters that you occur between the items when they're merged concatenated is defined by a
hyphen lowercase d or hyphen hyphen delimiters equals followed by a list and the list can be
enclosed in quotes or not depending on whether it's got ambiguous things in it so if you wanted to
use comma then you can just put hyphen d comma the fact that it's a list is interesting because
paste will take each of the characters in that list in turn and use it as a delimiters so if it's
got I don't know one two three or something in the list then it will use one for the first
limiter two for the second limiter and three for the third delimiter I don't know where you do
that but there you go so you can think of these merged lines as rows in a matrix or in a spreadsheet
or something where each file provides a column I should say I don't think I've said this in the
note if there's not the same number of lines per file then it will continue until all files have
been paste will continue to all files have been read so you will get some lines only contain
the lines from one file if one file is longer than the others you just run it and you'll see
that that's pretty obvious but the thing I'm that reason I'm talking about matrix is because
you can as it were rotated using the hyphen lowercase s or hyphen hyphen serial option let's
just sort of Boolean thing that switched on or off normally off so in that mode the serial mode
the lines from one file at a time are merged in other words the files are processed
serially rather than in parallel so the paste command can be used to generate the comma delimited
list that I wanted by using this hyphen lowercase s and hyphen lowercase d and there's an example
which is printing one of these some brace expansion expressions that I did years ago
now in the the bash tips series and print f open single quote percent s the backslash n closed quote
of that says print out whatever you get between it as a string in your arguments and then add a new
so the brace expression is open curly brace one dot dot ten close brace okay so you will get out
of that the numbers one to ten and because the print f is in is in force you'll get one per line
remember paste works on the basis of lines so you need to do that you couldn't just echo them
because it would come out as one line and pastes go oh yeah there's one line I'm not
nothing to do no it actually repeat the line yeah you know the one line but wouldn't do any merging
or anything so that's piped into paste with hyphen lowcase s hyphen lowcase d comma in the notes
I've enclosed that in quotes but in this example I haven't and then the argument to paste is
that hyphen which means get your stuff from standard in so what I get back is list of one to
ten with commas in between very simple though I talked about it for a long time but you know
sometimes things thoughts and things like this take a bit longer to explain you can also
instead of providing a file to paste you can also give it a bash process substitution expression
again I've covered this in the the the bash series the bash tips this is very powerful thing you
put a series of commands in parentheses and you proceed the first the open parentheses with
a less than sign so that's effectively meaning you're sending it to standard input and something
to the left on the line is expecting to consume it so here we have a paste hyphen lowcase s
space hyphen lowcase d comma then followed by less than sign then in that print if we use before
enclosed in parentheses get the same result vintage one you you like well I actually prefer the
second one even though it looks a little bit more involved because you can actually use it again
use it multiple times I'm not going to read this next one out but indeed it's horrible but what I
did as my next example was I have three process substitution expressions which are which are
using print f's to generate lists of numbers letters and more numbers in a range 100 to 106 and
then it's giving them all to paste and it's telling it to use vertical bar as a delimiter and you
get this rather jolly matrix of what do people have six seven down in one column and then ABC
blah blah blah and then a hundred and a hundred and one hundred and do in three separate columns you
can see how this is a sort of matrix yes they're all the same length the so-called files are all
generating seven items but no there's the point I'm trying to make is that it's like a matrix
and if you do it this way you can see it quite clearly so I then use that example to add the
hyphen lowercase s option and make it two things sequentially and it effectively rotates the matrix
so you've got one two three four five six seven the first line separated by vertical bars
then ABCD and the next line and then one hundred and one hundred and one etc on the third line
so you can see that if you've rotated it if you're using this serial option then you can use
delimiter as a separator as you print out the whole lot which is how my comma separated list
actually gets to be created so I finish up by listing the command I was actually trying to run
it's a query on a database where I'm keeping stuff about the shows I'm repairing there's a it
starts with an echo which produces a database query which is just once the the next five shows
that I'm going to be working on and it feeds it to SQLite 3 which is command line interface to
SQLite where it's listing the results from a database one per line and the database is called
ia.db into archive.db and then it's sending it to paste hyphen s hyphen d and then a
comma actually encodes this time just to prove that both do the same thing and then its final argument
is hyphen which means get what you work on from standard in so the shows are 1959 1950
due 1951 I'm doing them backwards 1946 1941 processing shows backwards by the way because
the the newer ones are a lot less hassle the older ones can be quite a lot of hassle because the
files are in weird places and sometimes they can't find them they're on the backup whatever so that's
what I've been doing and I just thought you might find this useful so just think to sort of file
away for future reference if you ever want to do this sort of thing or maybe you never will want to
do this sort of thing it's just weirdos like me that do but you might find it useful anyway and again
I've linked to the core utils GNUQ core utils manuals and the paste section is not all that long
but it's enough to know actually the man page paste is pretty good you don't really need much more
but if you want to you can go and have a dig around in this GNUQ core utils manual HTML version
that's it hope you found it interesting bye for now
you have been listening to Hecker Public Radio at Hecker Public Radio does work
today's show was contributed by a HBR listener like yourself if you ever thought of recording
podcast and click on our contribute link to find out how easy it really is hosting for HBR has been
kindly provided by an honesthost.com the internet archive and rsync.net on the Sadois status
today's show is released under creative comments attribution 4.0 international license