hpr-knowledge-base/hpr_transcripts/hpr3848.txt

Episode: 3848
Title: HPR3848: Editing Thunderbird email filters using vim.
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr3848/hpr3848.mp3
Transcribed: 2025-10-25 06:39:19

---

This is Hacker Public Radio Episode 3848 for Wednesday the 3rd of May 2023.
Today's show is entitled, Editing Thunderbird Email Filters Using Vim.
It is the 30th show of some guy on the internet, and is about 31 minutes long.
It carries a clean flag.
The summary is, Scotie uses Vim's sessions to quickly edit his email filters.
Hello and welcome to another episode of Hacker Public Radio.
I'm your host, some guy on the internet.
Today we're discussing Thunderbird Filters.
I'm going to be typing while discussing this episode, so if you're here to clicking
in the background, it can't be avoided.
I want to go over a couple of prerequisites with you.
There are five in total, but I'm only going to discuss the first two really quickly.
First one, backup all of your data, especially your Thunderbird directory.
Again, all of your data needs to be backed up.
Step two, confirm your backups.
Make sure that they work and try restoring one of your backups before proceeding.
You can read through the rest of the show notes of the other prerequisites.
Let's begin.
I'm going to be doing this from a GNU Linux distribution.
PapoS is the distribution that I'm using, so I have the apt package manager.
My Thunderbird install is through the package manager, so it's the .dev file.
The install is in the home slash . Thunderbird sub directory.
I have some information for those of you on other distributions, or if you're using
different package managers, there's some information down the show notes.
We're going to be using VIM to manipulate the filters, because the filters are located
within the . Thunderbird directory, but it's going to have like an alpha and numeric
string followed by .default release.
That's if you have a single instance of Thunderbird.
If you have multiple instances, it will be the alpha and numeric string .default release
followed by a hyphen once, or hyphen two, that kind of nomenclature, then further down
into the sub directories, you'll have your IMAP mail followed by the account itself surface
at Google.
It'll say .imap.google or Yahoo or whatever you got within that sub directory.
There will be the message filter rules dot that file.
The message filter rules dot that file is your filters.
We're going to use VIM to navigate to that file, open it up, and then I'm also going to
use VIM to open up the VIM session that I'm using.
But first let's talk about the VIM session.
The VIM sessions, they're just files that store information about your working session.
The one that I crafted for Thunderbird, it's a custom one that just created an empty
file, file name dot VIM, and then I imported or created the rules that I wanted for that
session inside of that file.
So if you're familiar with VIM mappings, I mapped out a bunch of things that I want to do
within that file.
So now I can source that file.
When you source a VIM session, it's basically extending VIM.
So you don't have to carry all of those different functions that you've created into every
single project.
You can have a custom set per project example when I'm doing the old no news.
I don't want those functions present when I'm editing a Thunderbird filter because that's
a lot of mark down in the Thunderbird filter has no mark down in.
So let's begin.
I first set up the work environment by opening up the message filter that I'm going to
be using so you can navigate in your system, find your email account that you'd want to
edit the filters for.
And while I'm on that topic, you're going to want to first create filters because obviously
if you don't have any, then you can't edit them.
So when you use VIM, navigate to the correct account, grab that filter file and open it
up.
I'm going to be making it is directly within the filter file, but the complex work itself
that happens to the section of the filter will happen in another file that I call CB.
In my home directory, I have a file called CB CB is short for clipboard.
It's just a basic text file, no extension or anything like that because Linux does not
require you to have extensions for it to recognize the use of the file.
And I created it using touch, but I'm going to open up CB as well.
So that way I can copy the filter from the filter file.
You know the important part that I'm going to edit, copy that over to CB and then begin
the complex work within CB.
So CB is going to actually have the source of the VIM session where all the complex work
will happen.
Let's begin.
If you're new to Thunderbird in the audience and you want to hear more about creating the
filters themselves, just let me know.
Leave a comment down below and I'll do a show on creating filters.
I also wanted to update Thunderbird's documentation on the subject.
However, it was this long sort of loop around of you, you have to use your Mizzoula account
not the Thunderbird account, which I thought Thunderbird had broken away from Mizzoula when
they created their own website.
So you use your Mizzoula account.
Then you have to, I believe they require you to use Sumo or something like that is some
sort of markup language that they use, which was kind of weird.
And I just wanted to use, you know, either text or, you know, pan doc flavored markdown.
I'll eventually get around to figuring out more about their submission requirements for
updating documentation and then I'll try to update their documentation.
But until then, I'll probably just create my own because it'll be faster.
I can just use the technology that I already have at hand, which is markdown and it's
great.
I probably have to include some pictures and things that add nature as well.
And I plan on doing a video on it in the future.
So we have them open with two buffers that is a filter in one buffer that we're going
to be grabbing the actual filtering properties from.
And we have CB, which is my clipboard text file in the second buffer.
I'll have CB sourced to the VIM session that I want to use.
And later on, we'll go into detail about the VIM session.
All right.
Right now, I want to open Thunderbird and I'll just give you a brief on how my filters
are set up, just a quick little brief there.
So if you open up Thunderbird, you press Alt, it'll bring your menu bar down, then you
can use T to go to Tools and F to go to Message Filters.
Inside of my filters for the account that I'm about to be editing, I have my junk filters
at the very top.
You know, there's this spam assassin.
I turned that on and off depending on how I feel followed by one that I call deleted
that just holds domains in it that I want to delete all messages from.
And I have two more after that called junk filters.
Now the reason I have separated deleted from junk, the deleted filter will take away
mail that I intentionally subscribe to, but no longer wish to receive mail from.
I do not want to classify those innocent newsletters in other projects as junk.
I just no longer wish to receive messages from them.
Now I'm not about to go through the whole unsubscribed.
Why do you want to unsubscribe where we not helpful blah, blah, blah and who knows in
the future, I may want to receive information from them again, but I'd have to sign up
for it again.
So with this method, I just throw them in the delete filter, which will automatically
delete all things from them.
And what I do inside of that filter, I specify the domain.
So you'll have newsletter at newsgroup.com.
I'll remove the portion that says newsletter and only keep the at newsgroup.com.
And that way anything from at newsgroup.com that enters my inbox will be filtered out.
It'll just be deleted.
Now the junk filters that I have, those are just things I don't know how I got on their
mail list, but I no longer want to receive it.
And I want it marked as junk.
So that's why I have the separate deleted filters.
The junk will also delete, but it marks them as junk first.
From there, I have a few tag filters for any bills or obligations that I may have.
When that information comes in, it will be tagged as a bill.
So there's a few strings that I look for in the message.
Because normally whenever it's a bill, it's it's the same string over and over.
And when I'm using a string style messaging filter, meaning I'll filter from the subject
of the message rather than using the address, the message came from for these tags.
I'll pull the string and I usually pick three to four words, three to four words that
appear consecutively.
So they can't just be like the first word, the middle word and the last word as we three
to four words consecutively.
I picked those three to four words that best sum up what the email is about, put that
in the filter.
And that way every time that message appears in my inbox, it will automatically be tagged.
The tag just applies a color over it.
So for a bill, it'll let's just say orange, it'll pop up as orange.
Now also have other filters behind that that will then take those messages and move them
to a directory where they can be filtered out of the inbox.
I just wanted them tagged first.
So that way I can identify the bills from any other, say newsletters that those organizations
may also send out.
Now at the bottom of my filters, I have other subject base filters and these are manual
ones.
And basically like, for instance, my bank directory, I want to keep the important stuff
from the bank, but all the other promotional garbage that they send, I don't want to keep
all of that.
But my main bank filter pulls from the domain.
So anything from that domain that I have verified will be placed in the bank folder, including
all of the promotional garbage that I don't want to see.
The important stuff will be tagged so that I can easily identify it visually and other
stuff that I don't want.
I throw those into a separate manual filter so that it does not automatically run.
That's important.
I'm ready to do an archive on my bank folder.
I run the manual filter on it first to filter out all the promotional garbage, which I
have, you know, I continuously add to it because as they come out with new promotional
garbage, again, select three to four words from the promotional subject of the email.
Add that to the filter that I'm going to use to manually remove everything.
And I run that filter against the bank directory to remove the garbage so that I can then
archive the important stuff.
All right.
Now that we went through all that, let's go ahead and switch over to VAM and then we'll
begin editing.
All right.
Here in VAM within our filter file, one thing you'll notice is all the filters are in this
single file for the email account that we have selected.
When you use a Thunderbird to create filters, it appears as though there's a separate file
being created per filter.
That's what the UI sort of visually implies.
However, on the back end, the config or the file itself is just one file where each figure,
excuse me, each filter is spelled out in between two properties.
The first is the name where the filter begins is called name and the last will be condition.
So everything in between those two properties are part of the filter and you'll see the
word.
So if you search name inside of that file, you'll see it appears numerous times based on
how many filters you have.
Same thing with condition.
I do want to point out you want to close your Thunderbird.
You do not want your Thunderbird running while you're editing the filter.
So if you have it open, now it's time to go ahead and close it before you start manipulating
anything.
Now what we want to look for is the condition.
Those are the actual things that you have said I want this stuff filtered out.
So the domains or if it's a specific email address or a subject, whatever, it's going
to be located inside this section and says condition.
I'm going to go ahead and select my very first filter, which is the deleted filter and
I see the conditions for it here on line seven.
So I'm going to go ahead and use, let me go ahead and give them down there.
I'm going to double-wide on that to yank that line.
Now let's open up VIMS command using decolon to be in so that's a bravo nano or whatever
the other thing is to us, switch to the next buffer where CB will be waiting with a blank
canvas for me.
And then I press P to paste that condition into CB and let me go ahead and source going
to command mode in VIM under the CB file, do SO and then we'll begin to select our session
that we want to use here, which is the Thunderbird filter VIMS session that I have created.
So now that I have sourced that within CB, I now have access to those custom things that
I've created to run on this file.
All right, so one of the things you're going to notice when you put that filter in there,
it's going to, depending on how many things you have in it, mine is quite long.
It's been added to over the years.
So it's going to go out quite far.
It's going to start with condition, have an equal sign followed by the double quotes.
And now if I do the dollar sign, which takes me all the way to the end of the string, you'll
see it'll have all the properties within it separated by the word OR.
There's a capital OR, Oscar Rojo.
And in the very end of the string, there'll be another set of double quotes.
I'm not set, but it'll, it'll be the completion of this set, double quote.
This is all to get back to the beginning.
Now what I'm going to do to start my filter here or to break down this filter, I should
say, I'm going to use the leader key, which is the back slash leader key one there.
And now my filter is completely broken down and separated in to so that way I can see
every single element of that filter on its own line.
Let's talk about what that function for mapping, yeah, it's a map, let's talk about what
that map does.
I'll have the maps for this VM session inside of the show notes so you can see what it's
doing.
Let's talk about it real quick.
So you'll see the very first one is a normal mode, that's what the first in is for.
And then it's in ORE, that's for no recursive map, normal mode, no recursive map, then
you'll see the leader, which is the leader key.
I think the default is back slash one, the number one.
So every time I press back slash one, it's going to perform the function to see that follow
the commands that follow the text there.
So let's talk about what that's doing.
It's going to delete two words, the two words that we're looking at, well, one of them's
not technically word, but for VM sake, we're going to call it a word.
It is the word condition at the beginning of the filter, followed by that equal sign
and the first of the double quotes.
Then it's going to run VM's rejects.
It's like said, but you know, it's basically said, that's the closest thing I can think
of.
It's going to run this rejects, which will look for each of those OREs, which is the
separation in between the properties that you want to filter out and space them onto
their own lines.
Now, one thing I want to point out is taking the space at the beginning of ORE and replacing
it with a new line in VM, the new line is not back slash in when you're doing rejects.
It's back slash R. So if you're going to see R forward slash back slash R. So we're replacing
a space with a new line with that and we're going to follow it up with a carriage return.
So I use global on that because otherwise it would just stop at the first or so use
global to carry that through the entire string because it's a very long string.
So that's our first leader thing that we're doing there.
Now that we have it broken down onto their own lines, let me use a GG to go to the top.
Now I want to make sure that everything is lower case because if you have capital case
letters and lower case letters, it'll mess up with the sort function that we're going
to use next.
And we don't want that.
So we build into this next mapping the ability to sort everything and make sure that it's
lower case before the sort happens so that everything gets alphabetized correctly.
Otherwise you have something with like a capital D placed in front of a lower case A.
Now this part is not necessary.
It's just something I like to do also removing the duplicates.
That's also another feature that I have built into this sort.
So if you look down into the show notes, you'll see the next normal nor recursive map
set to leader S1.
That's my sort function.
You'll see that it goes to the top of the map using GG.
I mean, not map the top of the buffer the file, it uses old to make sure it's in a very
top left, then it uses 2F so that it can find the second comma located within the filter.
So if we switch over real quick.
So once I have the lines broken down, it'll sit on to there.
Once I have the filter broken down where each element is on their own line, it'll it should
say or the capital or you have a set of parentheses with the word from comma contains comma, then
the thing that you want to filter out.
All right, I'm taking a short break.
My air conditioning just came on.
I just wanted you to see what it sounds like with the air conditioning on, but I'm going
to stop now and come back when air condition goes off.
All right.
We're already.
All right.
So in that leader S2, excuse me, in the leader S1, we got our GG top of the file, zero.
Make sure that we're on the very first character, F2, taking us to the second comma, followed
by CV, which is control V, that's how you spell it in your, when you're doing them, mappings
inside of your session file, we got a control V, which gives us a visual blocking.
So in your visual mode, you can do visual sections by using lowercase V and then selecting
the text.
Then you get your visual line, which will cover the entire line using capital V. Right now,
we're using control V. So we can select a specific block of text.
After control V, you see we go to the very bottom of the file using capital G, followed by
a dollar sign, which then takes that block toward the end of the strings.
Then we have you, which takes everything that's selected by that visual block and makes
it lowercase.
Now, the function, I mean, now the process starts over again, going to the top of the
file using GG and zero, the two F again, taking us to the second, comma, with the control
V, visual blocking or selecting a visual block, G, bottom of the file, dollar sign, end of
the block.
Now we run the sort function.
So you can see we're going to command mode using the cobbling, followed by a sort and
we sort using the U flag.
Now this is VIMS sort, not the command sort that you normally do in the terminal.
So you don't need the sort, space, hyphen, U. When you're doing it in VIM, you just use
source, I mean, sort, space, U. We followed that up with a carriage return, which is the
greater than less than surrounding capital CR.
That's how you do a carriage return in a VIM mapping and then we move back to the top
of the file using GG zero.
So that's going to go ahead and take all of those different email addresses or if it's
a string base filter where I'm filtering from the subject, make it all lower case, then
sort it alphabetically.
I don't know if I said this already or because I had to keep coming back and forth whenever
the air condition comes on, but if you have a capital letter in there in your filters,
it'll upset the alphabetical order that sort will use.
So a capital D could be placed in front of a lower case A just because it was capitalized.
We want to avoid that so we make everything, you know, lower case first.
Now once we're done and everything's already filtered up and alphabetized, we should be
at the top of our file.
That's why I have the GG zero at the end of that mapping.
And all we have to do is like say, for instance, if we have 50 lines of filters that have
been broken down onto their own individual line, I'll just type in 51 and do a capital
J and then the capital J will take the line beneath the line that you're on and bring
it up to the line that you're on and separate it with a space.
So you're restructuring the filter by doing the capital J and I don't need an actual mapping
for that.
I mean, it's just, you know, type in the number of times you want it to happen plus the
capital J boom, it's done.
So now that's going to restructure the filter and then we have the last leader, which is
the leader to you should see all of that in the show notes.
The leader to which goes to the beginning of the string, once it's been restructured
using the capital I inserts the word condition equal sign double quote and, you know, rebuilds
the rebuild the filter the way it was and it should give an error when it runs because
that last little bit of red checks on the end is just to make sure that there was no space
before the last quotation I've had that at times where there was like a space at the end
of it.
Sometimes it happens.
There's another thing that happens as well when you're structuring your, when you're
breaking down your filter, the very last item that gets broken down does not have a space
after it and you want to make sure that there's a space after it.
So I have another field, another thing I need to add in here.
I think I forgot to add it.
Let me go ahead and add that real quick.
Okay.
I just added it.
I forgot to put it in when I was constructing the show notes, but that item that is normally
at the end of the filter will not have a space after it.
So when it gets filtered, when it gets sorted alphabetically, it will break your filter
if you don't have a space after it during reconstruction in the section where I mentioned
I used the capital J to reconstruct the filter.
So that's why we have the leader and one.
So before we do the capital J to reconstruct the filter onto one line again, we run leader
and one.
It's going to perform the same type of rejects that was done in leader S1, where it goes
to the top of the file.
It's going to select everything using capital V, then a capital G, going to command mode
and then use the set style rejects to ensure that there is a, that there is a space at
the end of the line.
So if there's, if there's not a space there, it will add one.
And if there's more than one space at the end of the line, it'll reduce it to a single
space and it's going to follow that up with a character turn and take you back to the
top of the file so that each of the individual elements when they're separated onto their
online, you can ensure that every single one of them have a space at the end of the element
so that when you do the reconstruction using capital J, it all fits together properly.
Otherwise, one of those sections that is separated by the capital or, you know what, I'm just
going to finish recording through it because their condition keeps coming on and off.
I'm sorry, if it bothers you with the home of their condition, I try my best to edit it
down.
So that, that's normally where your filter is going to break.
One of those ores will not have a space either before or after it.
So I like to take care of that ahead of time.
The very end of your filter as well before the last quotation marks once you've used
capital J to reconstruct it and then you run the leader to which re-inserts the word condition
equals quotation and then it replaces the other quotation at the end of the filter.
Sometimes there'll be a space before that last quotation and that's why you see that
the rejects in there removes that space before the last quotation at the end of the
strain.
So if you get an error, that's because there was no space before the last quotation,
which in that case is fine.
Nothing to complain about.
You get an error because there was no space there.
If you don't get the error, that means there was a space there and the rejects work.
It took it away.
Now I always double check.
I just do a dollar sign to bounce to the end of the strain to a quick peak.
There's no space.
It's got the double quote there, zero to take me back to the base there.
I save it into the clipboard, you know, command mode W, double Y to yank that.
Then I use the buffer one, you know, go back in the command mode V1 to take me back to
the first buffer, which is my message, which is the filter.
I use P to insert the now reconstructed filter into the file and I delete the old one and
save it.
After doing that, leave it open, load Thunderbird, open up Thunderbird and check your filter and
you do this before you edit every single condition.
Because what will happen is if, for instance, you skip the step and there was a space missing
in between those ores, you'll get an error when you load the filter.
So when you try to run the filter, Thunderbird say something like does not recognize filter,
something simple like that and you'll know, oh, I must have a space missing.
I was thinking about creating a second rejects just to filter in between those ores and
making sure that there was a space in between there.
But I mean, right now it works flawless.
I have no problems.
But if it does come up in the future, I'll probably end up doing that.
I probably just included it anyway, just as a safety feature, something to run once
I've reconstructed the filter completely and done the second leader, a leader too.
So in quick recap, open up your file, your filter file, go to see your blank file, which
in my case is CV, grab the condition that you want, which is your actual filtering properties,
paste it over into your blank file, run leader one to break it down, run leader S1 to perform
the sorting and capitalization corrections or the capitalization first then sort, then
run leader in one to make sure that everything has a space after it, perform the capital
J. So if you have 50 different elements, then you'll do 51 capital J to reconstruct it
all back onto one line and complete the process using leader two, which will add the condition,
equal, etc.
Then you can just copy that back over to your filter, the actual filter file from Thunder
Bird, delete the old line that you yanked from, replacing it with the new one that you've
alphabetized and everything else and you're good to go.
Now what you'll also notice is the line may be shorter, that's because if you had duplicates
in there, the leader S1 with that sort function, the U attached to sort was going to remove
duplicates.
So that's why the line ended up being like one or two short, yeah, that's why it took
out the duplicates, which is good.
Your filter has to become more complex over years and even with those duplicates, it'll
slow things down, especially if you have a Yahoo email address, the Yahoo servers are terrible
and you might think it's Thunderbird being slow but it's not it's actually Yahoo or, you
know, whoever you're pulling your mail from, that's where it's bogging down.
And this is just how I manage my filters.
Now there is not currently a filter for archiving mail, otherwise I'd be using it instead
what I'm working on doing now because I manage, you know, multiple different email accounts
from different places.
After I clean things up and got rid of all the promotional garbage, I do a control A inside
of Thunderbird, inside of the directory with the important emails I want to keep, control
A selects everything, then I just tap the A key and the A key is a shortcut for archive
in Thunderbird.
In the show no time, I have some Thunderbird keyboard shortcuts, I have a link to it anyways
and that way you can familiarize yourself with the shortcuts.
I hope this episode has been informational, I apologize if I stumbled over myself a little
bit, it's because I've been trying to work around the air conditioning and I did not
want you to be bothered by the loud hum from the air conditioning.
Let me know what you think, if you want more on Thunderbird, especially the email filters,
let me know, I'll try to put some time in on that.
Thank you guys for listening to that hacker-public radio, please do a show in response if you
can, or I sure aren't anything else, maybe you've got some cool tips about Thunderbird
that we haven't heard about yet and you feel like we need to know about it, go ahead and
do a show.
I'll see you guys in the next episode, take care.
On the Sadois status, today's show is released on their creative commons, attribution 4.0