Fetchmail
 |
This {{#switch: | =article | Talk=talk page | Category=category page | Help=help page | Portal=portal page | Template=template page | User=user page | User talk=user talk page
}} is actively undergoing a major edit for a little while -- huston 15:33, 8 May 2007 (EDT). As a courtesy, please do not edit this page while this message is displayed. The person who added this notice will be listed in its edit history or has placed their signature above. If this page has not been edited recently (several hours!), please remove this template (or replace it with {{underconstruction}}). This message is intended to help reduce edit conflicts; please remove it between editing sessions to allow others to improve this page. {{#ifeq:{{{category}}}|no||}} |
fetchmail is a wonderful program which might just solve a lot of issues that people have when they want to copy data from the IMAP server, whether it be because they're leaving the department and want a copy of their mail, or if you wish to use a mail client which can't access the IMAP server directly (though that is an unsupported path, so aside from using fetchmail to download the mail you're on your own).
fetchmail is installed on a few of the linux machines in the department (Cygnus is one example). Before you run it, however, be sure to read and understand all the information here. Running it improperly can cause mail loops, deletion of all mail on the server, and many other Bad Things (some of which are a minor nuisance, others a major problem and unrecoverable).
First, you will need a .fetchmailrc file. This tells fetchmail what do to in order to get your mail. If you just want to download a copy of your mail to a local file, this .fetchmailrc file doesn't need very much information:
poll mail proto IMAP keep fetchall mda "/usr/bin/procmail -t -m my.pmrc"
The first line tells fetchmail to contact the server "mail", using the IMAP protocol. The second line tells the program to keep the mail on the server, and not delete it from the IMAP spool. The third line tells it to fetch all messages, not just unread ones. And the fourth line tells it what to do with the mail once it downloads it; normally it would hand it off to Sendmail in order to deliver it locally (the program is commonly used by people offsite to grab their mail and store it locally on their own computer). In our case, we want to hand the mail off to Procmail for processing and local saving, so we tell it the Mail Delivery Agent (or MDA) is Procmail. If you're curious, the -t option to procmail tells it to fail softly, so it reports an error to fetchmail if there's any problems, and the -m option tells it to act as a mail filter, using the file 'my.pmrc' instead of the default .procmailrc. There are a couple things to note before you blindly copy this and paste it to a .fetchmailrc file:
- Because of the 'keep' and 'fetchall' commands, this will cause *all* your mail to be downloaded *every time* you run the program. It will also mark all messages as 'read' on the server when it downloads them. While I could've written the file differently so you could use this all the time, I chose this set of commands in case someone doesn't read the warnings, so their mail isn't deleted from the server unexpectedly. If, however, you're reading this much and would like to change the behavior, read on to the bottom of this section and I'll explain how you can do so.
- This set of commands will only download mail in your inbox, not any subfolders of the inbox. I will explain momentarily how you can tell it to gather mail from subfolders as well as the inbox.
- I'll repeat, when this command runs it will mark all your mail as 'read'. Therefore, if you run fetchmail with this RC script, delete the resultant output file(s) and then login to the mail server with Pine, you will have no unread messages. They will still be on the server, so all is not lost; just the status of the mails.
Once you have the .fetchmailrc file, you now need to configure procmail to handle the messages once they're downloaded. The simplest way to do this is with the following 'my.pmrc' file (which is what you should name it, since that's what you told fetchmail to pass on to procmail in the above file):
MAILDIR=$HOME/mail DEFAULT=$MAILDIR/INBOX LOGFILE=$MAILDIR/delivery-log LOCKFILE=$MAILDIR/.default.lock :0 fhw |formail # # Any other rules the user wishes to either include with INCLUDERC, # or hardcode into this file, would go here. # -------------------------------------------------------------------------- # If we're here, the mail didn't match any other rules, so deliver normally. :0: $DEFAULT # If that fails, report an error and throw the mail away. EXITCODE=75 :0 /dev/null
The first four lines of this simple procmail script tell Procmail where to put the mail it's about to get; in this case, it would go in a file called INBOX, in a directory called 'mail' in your home directory. Procmail should create this directory if it doesn't exist. Be forewarned, if this file exists and is a mailbox file, all the messages you're about to download will be appended onto the end of the file. If that's not what you want, then be sure to change the "DEFAULT" line above to point to the name of the file you want mail to go into, and possibly the "MAILDIR" line to change the directory. Another file will be created in ~/mail/, called delivery-log, which contains information from procmail about the messages it received and what it did with them. You can make this logfile more interesting (and much larger) by also specifying "VERBOSE=ON" in the my.pmrc file.
The next two lines (:0 \ |formail) tell procmail to pipe every message through the 'formail' program. This adds a certain header to the top of each message which is required for some mail programs to determine where one email stops and the next one starts (The line "From user@foo.com Wed Mar 3 2003 12:41"). It should not be removed unless you know what you're doing, and know that you don't need that UNIX-mailbox-style format. Most things that don't need it, however, will ignore the From line if it exists, so it's safe to leave there.
Next is a few comments, which serve as placeholders. If you want to get fancy and insert normal Procmail recipes in that location, go right ahead. One example would be if you told Fetchmail to gather messages from every folder on the IMAP server, and want it to filter the messages into multiple files, you would put the proper procmail recipes there. There are other places where Procmail rules are mentioned in the FAQ, so for more information look there. Do note that forwarding messages from Procmail at this point is probably not a good idea; it would be much better to do this directly on the mail server instead.
After the comments is the two lines that deliver the mail to your file (:0: \ $DEFAULT). The first tells procmail to lock the file, this way if somehow two instances get run simultaneously the one will wait until the other is done writing to the file, and the second line tells procmail to write the message to the file named in $DEFAULT (which you defined above).
Should that fail for some reason, the last few lines in the script take care of it. First, they set the EXITCODE to 75, which means when Procmail exits it will be with a non-zero status. Exit code 75 specifically tells the parent process that the message could not be delivered, and that it should try again some other time. This will pass the information on to fetchmail that it should *not* delete the mail from the server (if it's configured to do so), or mark the mail as read. The last line then throws the message away to /dev/null since it couldn't deliver it.
Now that you have your .fetchmailrc file, and a basic my.pmrc file setup, run the 'fetchmail' command. It will ask for your password on the server; once you give it, the program will start to collect your mail from the server and deposit it into a file for you.
ADVANCED TOPICS
Some more information on Fetchmail, if you're so inclined, follows. Make sure you understand the implications of what you're doing; deleting all your mail off the server and asking for it to be replaced may be impossible, so always test the script with all destructive options off first (namely, use the flag '-k' to fetchmail to force it to keep all mail, regardless of what you said in the .fetchmailrc script).
One trick, if for example you wish to keep your mail on the server but pull unread messages from it (and flag them as read on the server), is to remove the 'fetchall' keyword from the .fetchmailrc file. This tells fetchmail to only download unread mail, but mark it as read when it does. In keeping with this, you can then run 'fetchmail -c' to only check for unread mail on the server, and not try to download it.
If you wish to be brave, and remove all your mail from the server keeping it only in a file, then remove the 'keep' directive from the .fetchmailrc file. This tells fetchmail to delete the messages as it downloads them. If you still have the 'fetchall' keyword there, then mail that you've read before on the server (through an IMAP client, such as Pine or Squirrelmail) will also be downloaded; if you remove 'fetchall', then only mail that is unread will be pulled down and deleted from the server.
You can tell fetchmail to get your email from other sources as well, with using other 'poll' directives naming other servers. For example you could tell it to get your @princeton.edu mail from OIT's IMAP server. 'man fetchmail' should give you all the information you need to figure this out.
You can also tell fetchmail that there's a preconnect condition that must be met before it can connect to the named server. Why would you do this? Let's say you want to pull all your unread mail from our server, while at home. But you don't want to use SSL to do it, you'd rather use SSH to tunnel the connection. Then you can tell fetchmail that instead of polling the server 'mail.astro.princeton.edu', it has to poll "localhost port 9000". Set a preconnect condition to do "ssh -L 9000:mail.astro.princeton.edu:143 coma.princeton.edu 'sleep 60'". This tells SSH to connect to Coma, and run the command 'sleep 60', which keeps the pipe open just enough for you to tunnel through it. Next, it will forward the local port 9000, through the SSH tunnel to coma, and from there connect to port 143 on the mail server (which is the IMAP port). Now your username and password will be encrypted from the local machine to coma, and will only travel in cleartext from coma to the mail server (which being on the same network means it's pretty safe). This is a great trick to use if you have to get mail with POP3, which is not usually SSL-aware (so it's an unencrypted connection), but you have a shell account on the POP3 server or another machine on its local network.
If you want to pull mail from more than one folder on the IMAP server, use the 'folders' keyword in the .fetchmailrc file. The keyword tells the program to get mail from the folders listed. Then list the folders you want to retrieve after the keyword, separated with commas. One thing to remember is that Cyrus IMAP keeps all folders as subfolders of your inbox, so if you want to get your inbox and two folders (we'll call them foo and bar), you need to specify:
folders INBOX, INBOX.foo, INBOX.bar
If you want to get *really* tricky, then tell procmail to deliver normally to $MAILDIR/$FOLDER. Edit the 'mda' line in your .fetchmailrc and add the '-p' flag to the call to procmail (this preserves the environment when you call the program). Then do this with a bash script (untested, use at your own risk!):
for F in inbox inbox.foo inbox.bar ; do export FOLDER=$F fetchmail -r $FOLDER done
This will set $FOLDER to inbox and run fetchmail, gathering your inbox and putting the messages into ~/mail/inbox ; then it will run again, getting mail from INBOX.foo, and putting it into the file ~/mail/inbox.foo, and then again with bar, etc. You could do this listing all your mail folders in the "for" line, and it will pull down *all* your mail from the entire server. It will ask for your password every time, but there is a way around that in the .fetchmailrc file (an exercise left for the reader, since it's very insecure to have your password in a file like that). Also be very sure that you know what you're doing if you do this; if you have the "fetchall" keyword in your .fetchmailrc file, and no "keep", you'll not only be copying all your messages to local files but also deleting them from the server, which is not reversable.