hpr-knowledge-base/hpr_transcripts/hpr4677.txt

Episode: 4677
Title: UNIX Curio #10 - Checksums and Hashes
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4677/hpr4677.mp3
Transcribed: 2026-07-31 16:16:38 (official HPR transcript)

---

This is Hacker Public Radio Episode 4677, for 2026-07-07
Today's show is entitled, "UNIX Curio #10 - Checksums and Hashes"
The host is Vance and the duration is 00:18:21
The flag is Clean, and the license is CC-BY-SA
The summary is "Another way to compare files"
Hi there. I'm Vance and welcome to Unix Curio number 10. This series is dedicated to
exploring little known and occasionally useful, trinkets lurking in the dusty corners of
Unix-like operating systems. In Unix Curio number 8, that's Hacker Public Radio episode 4657.
I talked about using standard utilities to compare files. Left unmentioned, however, was a
method commonly used today, the hash function. As I've stated in previous entries,
while I am an engineer, I don't have a background in computer science, so my understanding of the
mathematics is limited. But I can give a practical description of what a hash function does.
It takes an input, performs a set of calculations on it, and produces an output.
As hash functions are practically used, the input is a set of bytes,
such as a file or another piece of data like a password. The output is a numerical value in a fixed
range. Most often expressed as hexadecimal characters. Because this hash value
can always be represented in a certain number of bytes, its length as printed is usually a
constant number of characters, padded with leading zeros if necessary.
This episode will not cover the use of hashes in programming, focusing instead on using them to
validate data. A hash function, or more specifically, a cryptographic hash function,
has an additional property. It should be very difficult to predict what changes to the input
would be required to produce a specific change in the output.
An older related concept is called a check sum. While these are designed to vary,
when the input data is damaged, or digits are transposed, they do not necessarily
have that last property mentioned for cryptographic hashes. You have probably already encountered
a check sum, even if you didn't recognize it. On a 16-digit number assigned to a master card or
visa credit or debit card, the first six digits identify the card issuer, such as a bank,
the next nine digits are assigned to you by the issuer, and the last digit is a check digit.
The check digit is calculated using the values of the previous 15 digits, and it is a simple way
to avoid typos in entering a card number. In another example, every Ethernet frame that your
devices send or receive includes a check sum to help ensure that the contents weren't scrambled
in transit. This is 32 bits long and is called a cyclical redundancy check commonly referred to as a
CRC. A CRC is also used in many other places. For example, the zip file format includes one for each
archive member, and this allows a program extracting files from the archive to identify if any were damaged.
Our Unix Curio for today is another example, the CK sum utility. That spelled CKSUM or in the
phonetic alphabet, Charlie Kilo, Sierra Uniform, Mike. It generates a 32-bit CRC based on the Ethernet
algorithm. It operates on either a named file or standard input, and outputs the CRC value,
the length of the input, and the path name if a file was given as an argument.
Unlike most modern hashing programs, the check sum is printed as a decimal integer and is not
padded, so it can be anywhere from one to ten digits long. The length value is the number of bytes
in the input. Actually specified as the number of octets, as systems could potentially use a
byte that isn't 8 bits long, and this value is also expressed as a decimal integer.
There are two major ways that one could use CK sum to check the validity of a file.
First, if you are transferring a file from one Unix-like system to another, you could run
CK sum against it on both systems and check that the CRC and length are the same.
The utility can also be given multiple file names as arguments, which would generate a list that
can then be compared. The second way would be for someone publishing a file or set of files
to also publish the CRC values, lengths, and names so that people downloading them could verify
that they match. However, I don't think the practice of publishing lists like this
really started until more recent hash functions like MD5 and SHA1 came about, so it is unlikely that
anyone would publish CRC values instead. The advantage of these tools should be pretty obvious in comparison
to CMP, one of the utilities discussed in Unix-Curio number eight. To verify a file using CMP,
you need two files to compare. If you're trying to check a large file that you downloaded,
you would need to spend the time in bandwidth to download a second copy, and then if they didn't
match, you would have no idea which of the two if either was correct. By contrast, CK sum is
quicker to run, doesn't require downloading a massive amount of extra data, and if run against the
original file, makes clear what the correct value is. This utility is a follow-on from a program
called sum, which operated very much the same. I had a bit of trouble tracking down the exact
development history, but what seems clear is that two different variants were popular, a BSD version
and a System 5 version. Both output 16 bit check sums, but used different algorithms, so they didn't
give the same results. Also, the BSD version printed the length of the input data as the number of
1,024 byte blocks, while the System 5 version instead gave a count of 512 byte blocks.
Some sources claim that System 5 sum generates a 32-bit check sum, which could possibly be true
internal to the algorithm, but I have tested several independent implementations of the utility,
and all of them output a 16-bit value for both the System 5 and BSD algorithms.
From what I can tell, the BSD version came first. It was in 3 BSD, but probably appeared even earlier.
An identical copy of BSD sum was included with Unix-32V, which was AT&T's 1979 port of 7th edition
Unix to the Vax, and became one of the ancestors of System 3. The divergence seems to have
started with System 3 released in 1980. Its version of the sum utility changed to a new default
algorithm, though it could be made to use the BSD algorithm via the hyphen R option.
System 5 looks to have kept the same behavior as System 3. It's not clear to me why this
algorithm is universally called the System 5 algorithm, rather than the System 3 algorithm.
Perhaps it is because System 5 saw much more widespread use.
Instead of trying to reconcile these differences, the POSIX committee decided to create a new utility
with a unique name, use a separate algorithm entirely, and avoid the block length dispute by
printing the length in octets instead of blocks. I should point out that POSIX states that the CRC
algorithm for CK sum does not strictly meet the mathematical definition of a checksum.
I don't know enough to say exactly why it doesn't qualify, or to say whether either of the sum
algorithms do. However, in less formal usage, the term checksum has gathered the meaning of any
value used to represent or validate a set of data, so I am fine with using it no matter the
technical details of the algorithm. When two different inputs produce the same checksum or hash
value, this is called a collision. Because the output value has a limited range,
there are an infinite number of possible inputs that could produce a collision.
From a practical standpoint, the possibilities are more limited. The majority of these inputs are
larger than the number of atoms in the universe, which isn't going to be able to fit on any machine.
Unlike a cryptographic hash algorithm, the CRC is not specifically designed to resist and attack
or crafting a malicious input that would cause a collision. However, it should be sufficient to detect
accidental damage. Programs implementing more modern cryptographic hash algorithms are superior
to the checksum utilities in avoiding collisions, whether malicious or accidental, but there are still
three advantages that the older programs have. First, a system running a historical operating system
might not have the hash programs available, but is more likely to have CK sum or sum already included.
Second, the checksum values are much shorter than the hashes output by the newer programs,
making them easier for a user to compare by looking at them. This advantage is not as great as it
might appear at first because a common way to check a hash these days is to save a list of hashes
and file names. The hash programs can use that and do the comparison themselves,
sparing the user from having to validate it character by character.
The third advantage is that CK sum prints its input length in bytes.
This greatly limits the number of inputs that could be maliciously crafted to create a collision.
I did a moderate amount of research on implementations of modern cryptographic hash algorithms
and found that sum such as MD5, SHA1 and SHA2 do use the length of the input
often termed the message length in the literature as part of the material fed into the algorithm.
But none of the hashing utilities present this length to the user as part of its output.
There are two possible reasons for this that seem evident to me.
First, if one is hashing a password, you would certainly not want to give a clear indication of its length.
That would give any attacker a massive head start on guessing the password.
However, it doesn't explain why one would avoid printing the input length for a file that is made
publicly available. Second, it is convenient in many contexts such as database entries or in
software such as Git for the hash to be a fixed length, including an extra value
that can be a variable length would complicate those use cases.
However, the length value could simply be dropped and they would be no worse off than they are currently.
Historically on Unix, password hashing was treated differently from check summing files.
The crypt function was used for passwords while sum and later CK sum were used to confirm a
files integrity. So even rather early on, these two use cases employed algorithms with different
properties, but I haven't died into the history enough to know how intentional this was.
My discussion in this episode focuses on the file use case, so understand that I am largely avoiding
the topic of password hashing. Digital signatures are yet another use case, one that I am just ignoring
entirely. Every few years, some security researcher declares a particular hash algorithm to be broken
and that everyone should move over to a new one, which generally has a longer hash.
While the larger hash space certainly reduces the opportunity for collisions,
this disrupts workflows, such as publishing information about software releases by email,
which still tends to observe a 78 character limit on each line, making it harder to include
a list of hashes with file names next to them. This is in addition to the work of modifying
software and scripts to use the new algorithm and managing how to treat past data.
It seems to me that publishing the input length along with the hash would make it far more
difficult to craft a malicious input that matches both, but I haven't found discussion of that
during my investigation. Perhaps someone out there can record a response episode for
HPR explaining that. Thanks for listening. I haven't yet written the next Unix Curio,
but there are a couple topics I'm working on. Keep an eye out here on Hacker Public Radio for
them to appear. If you have an idea for a topic or want to ask a question,
please comment on this episode on the HPR website.
You have been listening to the Hacker Public Radio podcast, at hackerpublicradio.org.
Today's show was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast, then visit the HPR site to find out how easy it really is.
Hosting for HPR has been kindly provided by anhonesthost.com, the Internet Archive, rsync.net, and the HPR Community Content Delivery Network.
Unless otherwise stated, today's show is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.