hpr-knowledge-base/hpr_transcripts/hpr4687.txt

Episode: 4687
Title: UNIX Curio #11 - Merging Files
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4687/hpr4687.mp3
Transcribed: 2026-07-31 16:16:46 (official HPR transcript)

---

This is Hacker Public Radio Episode 4687, for 2026-07-21
Today's show is entitled, "UNIX Curio #11 - Merging Files"
The host is Vance and the duration is 00:11:21
The flag is Clean, and the license is CC-BY-SA
The summary is "Bringing your data together"
Hi there, I'm Vance and welcome to Unix Curio number 11.
This series is dedicated to exploring little known and occasionally useful, trinkets lurking
in the dusty corners of Unix Lake operating systems.
I frequently find myself reaching for the cut utility when rating scripts to extract
one piece of data from a line or to select specific fields from a log file.
While I am familiar with its counterpart, paste, I don't employ it very often because I don't
typically need its functionality.
This perhaps has to do with the fact that I rarely work with text files containing lists.
For shorter lists, I usually end up using a spreadsheet and for larger ones a relational
database.
Both are valuable tools with their own strengths and weaknesses, but it is good to also
know about standard utilities for working with lists.
After uploading Unix Curio number 8, that's Hacker Public Radio episode 4657, I felt
like maybe I had been too dismissive of the commutality in that episode and should
talk more about tools that are useful when managing lists.
I don't often find myself using paste, but can explain how it works.
Briefly, it is a rough opposite of cut.
Given multiple files as arguments, it assembles the first line from each one separated by
tabs, then the second line and so on.
Instead of tabs, a different delimiter can be chosen with the Hifendee as in Delta option.
Another option is HifendS as in Sierra, which swaps rows and columns so that the content
of each name file would appear on one line.
While paste itself doesn't qualify as a Unix Curio in my opinion, there is one feature that
does.
A Hifend can be given as an argument multiple times.
In this special case, the output is taken line by line from standard input, but is spread
across as many columns as there are hifens.
You can find an example of this in the show notes.
The paste command has limitations.
The files you give it must all be already arranged in the same order, and if any file
is missing a value, it must have a blank line so that subsequent lines will match up correctly.
The files do not necessarily have to be sorted alphabetically, but whatever order they
are in has to be the same.
Check out Hacker Public Radio episodes 962 and 4201 for some more background on the paste
utility.
Our second Unix Curio for today is a command called Join, which has a bit more sophistication.
It operates on two files, which can have multiple columns, and combines them using the
Join field.
By default, the first column or field in each file is the Join field, and only entries
that exist in both files are printed.
The Hifen 1 and Hifen 2 options can be used to join on a different field, and Hifen
O, select specific fields to be output.
To make it so lines with missing entries also appear, you need to use the Hifen A option
but an actual empty string with a separator won't be printed unless Hifen O is also present
and includes the field.
The default field separator character is one or more blanks in the current locale.
For the POSIX locale, this means a space or a horizontal tab.
The Hifen T option selects a different character, and also removes the treatment of multiple
occurrences as a single separator, making it possible to have an empty field in one or
both of the files.
By default, a single space is used to separate fields in the output.
If Hifen T is given, the same character is used for separating fields in both input and
output.
You would need to use another tool, like TR, if you wanted to have a different separator
in the output.
The join utility might be an improvement over paste in some cases, since the join field
makes it a little easier to identify which entries match up across files.
It is limited to operating only on two files, one of which can be standard input, so combining
more than that requires either creating temporary intermediate files or chaining together
join commands in a pipeline.
Another requirement is that all files must already be sorted in the current locale.
The full details of using join are a bit too involved to explain in this episode, so
I will refer you to the show notes, for an example, and a link to the utility's documentation.
The brief description for join is relational database operator.
I won't dispute that, but in my view it offers far fewer capabilities than people would
expect from today's relational databases.
I would imagine that when most people think of those, they have structured query language
or SQL in mind, which offers a lot more flexibility and functions to operate on data.
However, I can see how join could be suitable for simple operations.
Our last Unix Curio for today relates to the sort utility.
While, as you might expect, it is well known for its ability to sort data, it has another
feature that is more obscure.
When used with the hyphen M as in mic option, instead of sorting the files given as arguments,
it merges them together.
All of the files are expected to already be sorted.
Once combined, the list that is output will also be sorted.
The order in which the files are named does not matter.
It is not required for the contents of the first file to start before the second, just
that both are sorted.
Imagine that you organize an annual event and have a separate pre-sorted list of attendees
e-mail addresses for each of the past three years.
You are planning this year's event and want to send out an announcement to all of these
people as they will probably be interested.
The command sort, hyphen M, hyphen U, followed by the names of the three files, would spit
out a combined list that you can use for your e-mail blast.
Because it is likely that some people would have attended in more than one year, I included
the hyphen U option.
It removes any duplicate entries.
It is probably no surprise that the sort utility appeared early on.
It was in 1971's first edition Unix, though it didn't gain the merging functionality
until 5th edition in 1973.
What did come as a shock to me is that both cut and paste didn't show up until 1980 with
system 3 and were actually preceded by join, which was in 7th edition Unix from 1979.
I assume that at least cut would have been around far earlier, given it's usefulness
and how firmly established it seems to be, but I suppose it just seems to have been with us forever.
As mentioned, I don't typically manage data as text files containing lists, and I probably
won't start using the join utility, or these features of paste and sort very much.
But it is still useful to know that they exist and how they work.
Hopefully this episode has taught you a bit about them.
Thanks for listening.
There are a few topics that I'm working on for the next Unix Curio.
A couple of these will require more research than normal, so be advised that it might
take more time than usual before they are uploaded.
Keep an eye out here on Hacker Public Radio for them to appear.
If you have an idea for topic, or want to ask a question, please comment on this episode
on the HPR website.
You have been listening to the Hacker Public Radio podcast, at hackerpublicradio.org.
Today's show was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast, then visit the HPR site to find out how easy it really is.
Hosting for HPR has been kindly provided by anhonesthost.com, the Internet Archive, rsync.net, and the HPR Community Content Delivery Network.
Unless otherwise stated, today's show is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.