hpr-knowledge-base/hpr_transcripts/hpr4647.txt

Episode: 4647
Title: UNIX Curio #7 - Compression
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4647/hpr4647.mp3
Transcribed: 2026-07-31 16:15:56 (official HPR transcript)

---

This is Hacker Public Radio Episode 4647, for 2026-05-26
Today's show is entitled, "UNIX Curio #7 - Compression"
The host is Vance and the duration is 00:18:27
The flag is Clean, and the license is CC-BY-SA
The summary is "Making your files smaller"
Hi there, I'm Vance and welcome to Unix Curio number seven. This series is dedicated to exploring
little-known, and occasionally useful, trinkets lurking in the dusty corners of Unix-like
operating systems. In Unix Curio number four, that's Hacker Public Radio episode 4617,
I teased the subject of file compression. Today, I'm circling back to that. The history of data
compression goes back at least to the 1970s, and in contexts outside Unix and computers,
probably even earlier. Somehow, I find it refreshing to learn that humans have always struggled to have
enough storage space to keep all the data they want to hang on to. One way around this limitation
is to use some form of compression. I am only going to dive into lossless compression for this episode.
That is, a compression method that can be reversed and will spit out the original data bit for bit.
The alternative is lossy compression methods, and they also have their places. You might be familiar
with their use for audio, such as Augforbis or MP3. It's also used for images, such as JPEG.
Lossy compression allows some of the original data to be thrown away,
resulting in a smaller file than is possible with lossless compression. But the intent is for the result
to still sound or look good enough to a human observer.
Also, I'm going to limit my discussion to generic methods used for many types of data.
While FLAC does lossless compression, it is specifically designed just for audio.
I should make clear that I have never studied computer science or information theory,
so this episode will not get into the science behind various types of compression algorithms and how
they differ. But in general, these methods take advantage of the fact that many types of data
have recurring patterns. English text mostly consists of words that often reappear many times.
Source code similarly has keywords and variable names that recur.
Compression is accomplished by representing a piece of data that occurs multiple times with a symbol
that is shorter in length. The first compression program in the Unix world that I could find
is called PAC from 1978. It was shortly followed in 1979 by a similar program called Compact.
Both of these used a technique called Huffman coding, but with some differences between them.
Files compressed with PAC were given an extension of dot lowercase z,
or in the phonetic alphabet Zebra, and Compact gave file names an extension of dot uppercase C,
or in the phonetic alphabet Charlie. Roughly every five or ten years after this,
a new program would come along and achieve lasting popularity.
There were, and still are, two opposing forces facing any new form of compression.
Working in favor was the advantages it provided.
First among these was achieving a better compression ratio, but performance improvements,
such as speed or reduced memory usage, could also be compelling.
The force against any new method was the fact that it was not yet widely supported.
It didn't much help to have a smaller file if the people you share it with couldn't decompress it.
The next major advance in compression arose out of three scientific papers,
two in 1977 and 1978 by Abraham Lempel and Jacob Ziv.
These were called LZ-77 and LZ-78, and one by Terry Welch in 1984, which built on LZ-78.
This last method is typically referred to as LZ-W.
Our Unix Curio for today is a program called Compress that implements the LZ-W method.
Files compressed this way are named with the extension dot uppercase Z.
I had always assumed that this was to honor Jacob Ziv, but now that I've researched the history,
it seems more likely to be a follow-on from how files compressed by PAC were named.
Since PAC did not use any of the Lempel-Ziv methods, I would guess that it used dot lowercase Z,
because that wasn't already taken by anything else, but that's just pure speculation.
I do recall encountering dot uppercase Z files in the wild,
but feel certain that hasn't happened in the last 25 years and maybe even longer.
If you need to expand one of these, Uncompress is the program to use.
GNU's G-Unzip can also handle them.
However, there is a serious problem that arose with the LZ-78 and LZ-W compression methods.
Both of them were patented, and the owner became aggressive in seeking payment from developers and users.
The Compress utility was developed within two months of the publication of Welch's 1984 paper,
and was included in Bell Laboratory's 8th edition Unix before the shakedown started.
The paper did not disclose that a patent had been filed,
and apparently Spencer Thomas and the other developers of Compress were unaware of it.
The utility became popular for a while and was even standardized by POSIX,
but people moved away from LZ-W once the legal threat started.
Another important advance came in 1991 and was called the deflate compression method.
It combined the unpatented LZ-77 method with Huffman coding to achieve a similar level of compression as LZ-W,
actually often better, without the legal trouble.
Deflate was developed for PK-Zip and was soon adopted by the GNU project's G-Zip compressor.
While Phil Katz, the PK in PK-Zip, patented one way of implementing the deflate method,
it was possible to write a compressor and decompressor without infringing.
Also, he apparently never tried to enforce that patent.
As I mentioned in Unix Curio number 4, Zip is both an archive and a compression format.
Each archive member can be compressed with one of several possible methods or stored without compression.
Unlike a tar file where compression can be applied to the entire archive,
in Zip, each archive member is compressed individually.
This often means that a Zip file will be slightly bigger than a tar file with the same contents compressed by
G-Zip, because the Zip format cannot take advantage of duplication that occurs among more than one member of the archive.
The vast majority of Zip files use only the deflate and uncompressed storage methods,
and these are the only options if you want to follow the profile standardized in ISO IEC 21320-1.
Actually, since they both use deflate, G-Zip is able to extract a Zip file in the special case
where it only holds one member compressed with that method.
From the 1990s onward, people paid significant attention to avoiding patent landmines,
so only methods that didn't have that problem became broadly popular.
While the patents on LZ78 and LZW have since expired, I feel like their most successful legacy
was in discouraging people from using these methods, leading to deflate taking the overall popularity crown.
The next step came in 1996 and 1997 with the development of BZip and BZip2 by Julian Seward.
The original method was quickly followed by BZip2, which was the version that achieved true popularity.
They used the Burroughs-Wheeler transform, which does not itself compress data, but rearranges it to make it more compressible.
This is then combined with other techniques.
At least, that's my understanding. I told you, I'm not up on information theory.
This provides a significant reduction in the compressed size of the data compared to the earlier methods.
However, it is slower than deflate both during compression and decompression.
Separate projects have developed parallel versions of GZip and BZip2 that can take advantage of
multi-processor machines, but the original utilities run single-threaded.
Another five years later, in 2001, Igor Pavlov added the Lempel-Ziv Markov Chain algorithm,
or LZMA, an enhancement to LZ77, to his 7-zip compression tool.
This was followed a few years later by LZMA2, a container format that allowed for LZMA compression
to be split between multiple threads. Broad LZMA2 support came to the Unix world in 2009 with the
XZ utility. That's X-Ray Zebra in the phonetic alphabet.
It offers roughly similar compression ratios to BZip2, though it can be better or worse depending on
the data to be compressed. While compression takes even longer than BZip2, decompression is significantly
faster, though still not as fast as GZip. The Linux kernel relatively quickly supported booting
from XZ compressed images because it was a good match for that use case. Compression,
the time-consuming activity, only has to be done once, while the more frequent decompression during boot
happens relatively fast. The last method I will cover is Z-Standard, often written as ZSTD.
This came about in 2015 and is another variation on LZ77 that uses finite state entropy, which means
nothing to me, but you might understand it. It performs about as well as deflate in terms of compression
ratios, but is much faster both when compressing and decompressing data.
I should say that these statements are true with the typical default settings. Depending on the
compression level selected, it can compress more slowly, but compress the data smaller.
However, decompression is always speedier than deflate.
This makes it attractive for some uses, and it is heavily promoted by Meta slash Facebook,
where Jan Collette developed it.
For example, shipping large amounts of actively used data between machines in a data center
can go more quickly when the size is reduced. However, if the compression and decompression steps
take too long, that benefit is lost.
A speedy method can be valuable even if it doesn't result in the greatest reduction in size.
This use case stands in contrast to, say, a compressed backup file, which might only be accessed in a
disaster recovery scenario, or never accessed at all, making size more important than speed.
Both the XZ and Z standard utilities have some built-in support for multi-threading, but the default is to run in a single thread.
While XZ can use multiple threads for decompression, but only if the file was compressed in multi-thread mode,
the reference Z standard utility can only use more than one thread for compression, not decompression.
There are many other methods of lossless compression that have been developed over the decades,
but I believe that these are the ones you are most likely to encounter in the world of Unix-like systems.
This is a personal opinion, and others might choose a different set.
As mentioned, it can be tough for a new method to gain popularity, and 35-year-old deflate is still
probably the most commonly used, despite not being the fastest or offering the greatest reduction in size.
Even systems like FreeBSD, NetBSD, and OpenBSD that do not like to include GNU tools supported it by
developing their own version of GZIP based on the permissively licensed Zlib library.
The show notes contain some data comparing the performance of different utilities discussed
in this episode, starting with GZIP's deflate method.
Technically, the LZW method used by the compressed utility is still standardized by POSIX, so one might
expect it to have the widest support. However, aggressive patent enforcement discouraged adoption,
especially by free and open source software systems. Even though the patent has expired,
it is still out of favor compared to deflate. For this reason, I feel justified in calling Compress a Curio.
Thanks for listening. I don't currently have any more Unix Curios written. While I do have a couple
potential topics in mind, it's not yet clear that they would make good episodes. Any future podcast
entries will appear here on Hacker Public Radio. If you wish to suggest any topics or have a question,
please comment on this episode on the HPR website.
You have been listening to the Hacker Public Radio podcast, at hackerpublicradio.org.
Today's show was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast, then visit the HPR site to find out how easy it really is.
Hosting for HPR has been kindly provided by anhonesthost.com, the Internet Archive, rsync.net, and the HPR Community Content Delivery Network.
Unless otherwise stated, today's show is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.