Filtering emails with sieve scripts and SourceHut October 31, 2020 on tomleb's blog

One of the annoyances I have had with emails has been filtering and managing messages. This usually rely either on email clients' or the email provider’s filtering feature. Both have their pros and cons, but both are usually extremely limited. This is where sieve filters come in. We will go over how I use sieve to filter my messages, and how I configure and deploy my filter with SourceHut’s CI.

Sieve filters

Sieve is a programming language purposely designed for email filtering. It can be run on the email server as well as on the client side. It is also quite limited for security reasons; it does not have a loop mechanism or ways to run programs. The sieve language is an open standard described in RFC 5228. As such, any email provider¹ that implements it will work similarly, depending on the extensions supported. This also prevents being vendor locked to proprietary systems.

Sieve allows us to filter incoming messages. We can discard messages based on certain criteria like From or Subject headers and move messages into specific folders. If the proper extensions are supported, it is also possible to add flags to messages, like marking a message as seen, or tagging it as important.

A sieve script is composed of three kinds of commands: action commands, control commands and test commands. Action commands are used to describe what to do with the message (eg: whether to keep it or discard it). Control commands are used to control which part of the script gets executed. It is possible to stop execution, require an extension or add a conditional branch. Finally, test commands describe the conditionals that must evaluate to true to take a branch.

Enough of vague descriptions, let’s see an example. Below you can see a simplified sieve script that I use to move messages from a mailing list to a specific folder on the mail server. The example is good enough to introduce few concepts from the sieve RFC.

# example.sieve
require ["fileinto"];

if header :contains  ["From", "To", "Cc"] "~sircmpwn/sr.ht-discuss@lists.sr.ht"
{
  fileinto "mailing_lists.~sircmpwn.srht-discuss";
  stop;
}

fileinto "INBOX";

The line starting with the require control declares a list of extension dependencies. If not all the extensions are supported by the sieve server, then there will be an error when deploying the script. In this case, we are requesting the extension fileinto, allowing us to move a message to a specific mailbox.

Then, we have a conditional using the if control where we test if any of the headers From, To or Cc contains the value ~sircmpwn/sr.ht-discuss@lists.sr.ht. If it does, then the branch is taken. Otherwise, we continue evaluating the rest of the script.

In this case, when the branch is taken, we use the fileinto action to move the message to the nested mailbox mailing_lists/~sircmpwn/srht-discuss. Note that my email provider uses a dot as path separator.

Then, we use the stop control to stop the execution of the sieve script to avoid hitting more rules.

The very last action moves messages to the INBOX mailbox.

Advanced filters

The example above was a simple script that covers basic functionality. However, filters can be more complex to enable multiple different use cases. For example, there are test commands that check the size of the message. There are also test commands that evaluates to true if all tests commands evaluates to true, or if any tests commands evaluates to true.

Below is a more advanced sieve script that I currently use. It makes use of three extensions: fileinto, mailbox and imap4flags. One of which we have seen previously.

require ["fileinto", "mailbox", "imap4flags"];

# 1
if allof(header :contains  ["From"] "builds@sr.ht",
         header :contains  ["To"] "me@tomlebreux.com")
{
  # 2
  if header :contains  "Subject" "build failed" { addflag "$label1"; }
  # 3
  fileinto :create "builds";
  # 4
  stop;
}

1. We use the allof test to execute the block only if both of header tests are true. This means the allof evaluates to true only if the From header contains builds@sr.ht and the To header contains me@tomlebreux.com.

2. Inside the block, we check to see of the Subject header contains the string build failed, which would mean a build from SourceHut failed. In that case, we use the addflag action from the imap4flags extension to add a flag to the message. We add the flag $label1 which corresponds to the Important flag in Thunderbird.

3. We use the fileinto action that we have seen previously to move the message to the builds mailbox. However, this time we use the :create tagged argument from the mailbox extension. This ensure that the mailbox builds will be created if it does not exist.

4. We stop the execution of the sieve script.

Let’s see one last example that I really like to use when working with mailing lists.

require ["fileinto", "mailbox", "imap4flags"];

if header :contains  ["From", "To", "Cc", "Bcc"] "~sircmpwn/public-inbox@lists.sr.ht"
{
  if header :contains  ["From", "To", "Cc", "Bcc"] "me@tomlebreux.com" { addflag "$label3"; }
  fileinto :create "mailing_lists.~sircmpwn.public-inbox";
  stop;
}

This example uses the concepts mentioned in the previous example. In this case, I am moving messages from a mailing list to an assigned mailbox. I am also adding a flag (Personal green flag in Thunderbird) for messages in which I am personally involved. This helps to see at a glance which messages might be more of interest to me.

So far, we have seen the general format of the sieve language and some of the things you can do with it. The next section will cover how to tests scripts, how they can be deployed in general and how I deploy mine automatically using SourceHut.

Deploying Sieve scripts with SourceHut

You can test your sieve scripts with fastmail’s Sieve Tester. This is useful to quickly test scripts with your own email messages.

Alright, now that this is out of the way, let me explain how to deploy the sieve scripts on your email provider’s server. Each email service provider may implement its own way to let users upload sieve scripts². This may be through GUI, or other means. Fortunately, there also exists the ManageSieve protocol documented in RFC 5804. This provides us with a standardized way to manage our sieve scripts if we have carefully chosen our email provider.

A ManageSieve service typically listens on TCP port 4190. I use the sieve-connect utility to connect to a ManageSieve daemon. In the case of my mail provider, the sieve server can be access at sieve.emailarray.com:4190.

Here is an example usage of the sieve-connect utility. The first command validates the syntax. The second command uploads the script on the sieve server. The third command activates the script.

# Syntax checking the script
sieve-connect --server sieve.emailarray.com --user 'me@tomlebreux.com' --localsieve example.sieve --checkscript

# Uploading the script on the server
sieve-connect --server sieve.emailarray.com --user 'me@tomlebreux.com' --localsieve example.sieve --upload

# Activate the sieve script
sieve-connect --server sieve.emailarray.com --user 'me@tomlebreux.com' --remotesieve example.sieve --activate

We have the necessary commands to automate the deployment process of our sieve script. Now we just need to create a git repository and add a SourceHut build manifest. Here is the build manifest I am using.

image: archlinux
sources:
- https://git.sr.ht/~tomleb/sieve
secrets:
# git.sr.ht ssh key
- 469ae73d-769d-4345-924e-1e7191f5de8f
# sieve password
- 3ae40a88-8fbd-4322-a1ef-78ce956a462d
tasks:
- sieve-connect: |
    cd sieve/
    makepkg --syncdeps --install --rmdeps --noconfirm    
- check: |
    cd sieve/
    <~/.sieve sieve-connect --server sieve.emailarray.com --user 'me@tomlebreux.com' --passwordfd=0 --localsieve emailarray --checkscript    
- upload: |
    cd sieve/
    <~/.sieve sieve-connect --server sieve.emailarray.com --user 'me@tomlebreux.com' --passwordfd=0 --localsieve emailarray --upload    
triggers:
- action: email
  condition: always
  to: Tom Lebreux <me@tomlebreux.com>

Without going into too much details, I have three tasks that

1. installs sieve-connect using its PKGBUILD,
2. validates the emailarray sieve script,
3. upload it to the server.

You should now be able to build your own sieve scripts and understand how you could deploy using the ManageSieve protocol. There are a lot more functionality possible because sieve is extensible. Happy filtering!