Difference between revisions of "Email"

From Peyton Hall Documentation
Jump to navigation Jump to search
(Possibly done for the night - or at least the time being)
(Saving to update inuse tag)
Line 1: Line 1:
 +
{{inuse|2007/05/08}}
 
{{cleanup|date=May 2007}}
 
{{cleanup|date=May 2007}}
  
Line 100: Line 101:
 
=== I created some folders in my client, but they're not in Squirrelmail! ===
 
=== I created some folders in my client, but they're not in Squirrelmail! ===
 
Fear not!  Squirrelmail uses the IMAP subscription model.  Similar to how Usenet worked, before Squirrelmail will let you see the contents of a folder you have to subscribe to it.  In this way, if you have a bunch of folders you don't want to see if you login via Squirrelmail, you can turn them off and they won't be visible.  Conversely, if you create new folders and want to see them, you have to subscribe to them first.  Click on "Folders" at the top of the page, and near the bottom you'll see two sections.  The left side shows folders to which you are subscribed, and the right folders to which you are not yet subscribed.  Select whichever folders you desire to change (one side at a time, please) and then click the corresponding button beneath the section to submit the changes.  You may need to refresh the folder view on the left panel (click "Check email") for the changes to appear.
 
Fear not!  Squirrelmail uses the IMAP subscription model.  Similar to how Usenet worked, before Squirrelmail will let you see the contents of a folder you have to subscribe to it.  In this way, if you have a bunch of folders you don't want to see if you login via Squirrelmail, you can turn them off and they won't be visible.  Conversely, if you create new folders and want to see them, you have to subscribe to them first.  Click on "Folders" at the top of the page, and near the bottom you'll see two sections.  The left side shows folders to which you are subscribed, and the right folders to which you are not yet subscribed.  Select whichever folders you desire to change (one side at a time, please) and then click the corresponding button beneath the section to submit the changes.  You may need to refresh the folder view on the left panel (click "Check email") for the changes to appear.
 +
 +
 +
=== How do I export my email? ===
 +
If you're leaving the department and closing your account, you'll probably want to take your email with you when you go, either as live copies or as an archive.  Or maybe you want to use some mail program that doesn't use IMAP, and needs to see the email folders as flat text files.  There's a few ways to do this:
 +
* [[Pine#Exporting mail|Pine]]
 +
* [[Fetchmail]]
 +
* [[Thunderbird]]
 +
Pine and Thunderbird can be configured so you can copy your mail from Peyton to your new mail server (as long as it's IMAP).  Pine can also (as can Fetchmail) be used to download text copies of your email, in a few different formats (the most common being 'mbox' format).  Depending on your needs, follow one of the links above to see how to get the job done.  For automated downloading of mail, Fetchmail is probably the best choice.
 +
 +
 +
=== How do I filter email? ===
 +
You'll want to either read a bit about [[Sieve]] - the filtering language used by Cyrus IMAPD - and create a Sieve script, or have a look at [[Websieve]] which we run here as a web frontend to the Sieve language.  Most things you might need to do can be done through the interface in Websieve, but worst case scenario you can upload straight Sieve scripts to it as well.
 +
 +
 +
=== How do I manage delivered (and tagged) spam? ===
 +
See [[Websieve#Spam filtering]] for a few examples.
  
  
 
{{update}}
 
{{update}}
−
*[[Exporting Email]]
 
−
*[[Using Fetchmail]]
 
 
*[[Procmail (old message filtering system)]]
 
*[[Procmail (old message filtering system)]]
−
*[[Spam Filtering]]
 
−
*[[Mail Filtering via Websieve]]
 
 
*[[Mailman, the Mailing List Software]]
 
*[[Mailman, the Mailing List Software]]
 
*[[Because it messes up the natural order of reading, and is hard to understand]]
 
*[[Because it messes up the natural order of reading, and is hard to understand]]

Revision as of 14:28, 8 May 2007

Crystal 128 clock.png
 This article is actively undergoing a major edit for 2007/05/08.

As a courtesy, please do not edit this page while this message is displayed. The person who added this notice will be listed in its edit history or has placed their signature above.

If this page has not been edited recently (several hours!), please remove this template (or replace it with {{underconstruction}}).

This message is intended to help reduce edit conflicts; please remove it between editing sessions to allow others to improve this page.

Broom icon.svg This article or section may require cleanup to meet our quality standards.
Please discuss this issue on the talk page or replace this tag with a more specific message.
This article has been tagged since May 2007.

Email has become one of the most important means of communication among people here, in part because they may not be physically anywhere near their collaborators. Also, being a 'non-intrusive' communication is helpful - if you send someone an email and they're not available to reply right away, they can take their time formulating a response instead of needing to on the spot, or promising a return phone call later when time is available. Quite handy for those 4AM questions that pop up. This section will explain a bit about email, including links for setting up various mail clients in the building, as well as some of the details of our email setup here in Peyton Hall.

Email basics

While email has become an important and common method of communication, many don't quite understand the details that go behind pressing 'send' in your client. What follows is a brief explanation of how that all works.


How it works

When you compose an email to send somewhere, your mail client generates a series of text which describes the details of the message. There's a few headers written, including who the mail is from (your From: address), the destinations (To:, CC: and BCC: for blind carbon-copied messages - those which are sent somewhere, but not written in the message itself so other recipients are unaware it was sent to them), the Date: header saying when the message was written, and a Subject: header for the subject line of the message. Some clients will also include other headers, such as priority (which works differently in different mail clients) or headers to identify the client program and version. The email is now ready to be sent to a server that speaks Simple Mail Transfer Protocol, or SMTP.

In some cases, when you send a mail, the message is handed off to a SMTP client program such as Sendmail running from the command line (/usr/bin/sendmail). This client program will take the message and handle speaking SMTP to the mail server. In other cases, your mail client will speak SMTP directly to the server. In both cases, the conversation is fairly short; the client tells the server what information it needs, known as the "envelope headers", so called because they could be different than what the message itself has for recipients and headers (ie, if you're sending a BCC of an email to someone, the envelope recipient is that person, even though their name doesn't exist in the actual headers of the email). Once the headers are passed, the client tells the server it's now sending the data of the message, sends the data, then signals that the message is done. At that point, the mail server can either accept the message completely - usually giving a kind of tracking number for the email - or reject it, which should cause the client to pop up an error that the mail could not be delivered. Should the mail be accepted, it is now the responsibility of the mail server which accepted the mail to either deliver it one step closer to the recipient, or return a message to the sender to say that the message could not be delivered. It is unacceptable to have a legitimate email discarded by a mail server and no "delivery status notification" sent back to the sender of the message.

Once the SMTP server holds the mail, it looks at it to determine who needs a copy, and where they are. Through some clever Domain Name Service (DNS) entries, it can tell where a mail for a certain person needs to be delivered, and will contact that server (unless this SMTP server is configured to send all mail to a "relay host", in which case it will relay the message up the chain and that host will now be responsible for delivery - this is common if you have an ISP which requires all emails be sent through their own servers, and not directly from your home computer). Once the SMTP server knows where to hand off the message, it initiates a connection to that server and repeats the process, adding to the message a few headers to mark the email's travels through itself (in most cases, a "Received:" header to show that it received the message and handed it off, while in some cases there may be a few headers added if the message was scanned for spam & viruses). This process will continue to repeat, until an SMTP server holds the message and must deliver it locally somehow.

When the local SMTP server holds the mail, it may be delivered many ways depending on the local configuration. In many cases, the mail will be scanned for viruses and spam content, and assuming the message passes it will be delivered. Some mail storage systems work with plain text files for holding messages, so the mail server can deliver them internally. Others use the Local Mail Transport Protocol (LMTP) to pass a message on to a mail server daemon which clients will connect to for reading mail (using either Post Office Protocol v3 (POP3) or Internet Mail Access Protocol (IMAP)).

As mentioned before, if a SMTP daemon cannot deliver a message any closer to the recipient for some reason, it has two choices for what to do:

  1. Reject the message, and the sending program (SMTPD or client) is now responsible for notifying the sender somehow
  2. Generate a Delivery Service Notification (DSN) message to the sender, notifying them that their email could not be delivered and why

In some cases, DSNs are sent because the recipient does not exist, or their mailbox is full. Sometimes it may be because a virus was detected in the email, or because the mail appeared to be spam (or had enough characteristics to make it 'spammy'). It could also be that the recipient's SMTP server is not responding, in which case you'll see the familiar "Message could not be delivered for 4 hours; will keep trying for up to 5 days" DSN. These mean just what they say: the message could not be delivered in the last four hours, but the program will keep trying to deliver it (usually once every hour) until it is five days old, at which time it will send another DSN to let you know the mail could not be delivered at all. Because SMTP servers will hold messages like this, mail tends to always go through even if a mail server is down for a while. All those trying to send mail to it will just hold their messages and wait until a later time, so the worst problem you'll usually see is that messages are delayed an extra hour from when the server is brought back online.

If you'd like to know how our mail system here is configured, see the below section "The nasty details".


Using email in Peyton Hall

There's a few ways to read your email while in Peyton Hall, which are outlined here. A word on "support" first, however. Because there are so many different ways to access email, we cannot possibly support every permutation of them. However, we have selected a few programs which are widely used and easily setup so that we can guarantee support with those programs. This does not mean you must run only one of those programs; you're free to use whatever software you like for reading and sending emails. However, if the program you've chosen does not work properly, or has some kind of problems that are not a fault of the mail server itself, you will be on your own to figure out the problem and corresponding solution. See Other clients below for information if you wish to use an unsupported email client.


Supported clients

Pine

From any Unix/Linux machine in the building, there's two main options for viewing your email. The first is Pine, a text-based email program which will work over SSH connections as well as while you're sitting in front of a machine in Peyton. While it may not have all the whiz-bang features of a GUI mail client, Pine is still in use by many and is a good no-frills (or low-frills) program to use for accessing email. Because of the way we have it configured, we can also guarantee that even if nothing else works, Pine should be able to read your mail no matter what - as long as the mail server itself is running. You can find more information about Pine in the article linked above in this section.


Thunderbird

Another supported client in the building is Thunderbird. This is what became of the mail client portion of the Mozilla suite. Thunderbird is a great IMAP client with many plugins and extensions that can help you effectively and efficiently use your email. Since it supports encryption and secure connections, you can configure Thunderbird for use on your laptop, and it will work the same if you're in the building or sitting at home. More information on Thunderbird is available in its own article (linked above).


Squirrelmail

Squirrelmail is an IMAP client which is accessed via a web interface. We run Squirrelmail on the mail server directly, and using a SSL certificate for encryption means your user name and password are not sent in-the-clear for others to see. You can do just about anything through Squirrelmail as you would with any other mail client, with the added benefit that you don't need anything more than a web browser to access it. The downside, of course, is if you don't trust the computer in front of you (for example, in an Internet cafe or web kiosk). But if you're visiting some other institution and have web access, you can read and reply to emails with Squirrelmail. You can use the article link above to see more information.


Other clients

For unsupported email clients, we have some information here which may be useful in getting it setup to access the mail server. If you have information about another client, you're welcome to either add it here, or if it's a large article create a new page for it and link to it in here instead.

  • VM is an email client that runs within Emacs.
  • Netscape Mail, while no longer in active development, may still be used - but you should probably upgrade to Thunderbird instead.


Use your own

If none of these clients fit your needs, you can always pick your own for whatever you'd like to do. The information you'll need to know is:

  • SMTP (outgoing) server: mail.astro.princeton.edu
  • IMAP (incoming) server: mail.astro.princeton.edu
  • Security/encryption must be turned on
    This means that the IMAP port will be 993 instead of 143, and SSL must be turned on. For SMTP, you'll need to tell your client to use TLS encryption. You will also have to supply your user name and password to the client, as it will have to authenticate to the mail server to be able to send mail to recipients not at the astro domain (ie, you're relaying mail through our server from home, and the recipient is in Berkeley).


Accessing email from outside Peyton

As nice as the building is, you can't always be here. So how do you access your mail from somewhere else? Simple!

  • If you have Thunderbird configured on your laptop or home computer, it will work just the same while outside the building as it will while connected to our network.
  • If you have SSH access from wherever you're connecting, you can login to any machine and run Pine.
  • If all you have is a web browser, you can use Squirrelmail


Email FAQ

This section lists some frequently asked questions about email, and their answers (or links to their answers).


How do I get all my mail in one place?

If you have an account with OIT, you also have an email address on their servers. However, many people don't like to have to check multiple accounts, and would rather have their mail all in one location. We'll leave out issues of server availability and uptime :>


To forward OIT mail to here

  1. Go to http://support.princeton.edu/account/ and select "Forward your e-mail" from the list of choices under "Manage your e-mail account". Login with your OIT user name and password.
  2. Click the "Change entry" button.
  3. Select the radio button next to the blank text field, and enter your Peyton email address (username@astro.princeton.edu).
  4. Click "Submit changes" at the bottom of the page.


To forward astro email elsewhere

If you would like your email here to be delivered somewhere else, see the forwarding section of the WebSieve article.


Why don't I see duplicate emails?

Say for example, you're subscribed to two mailing lists. One list, however, is subscribed to the other, so normally you would see two copies of the message - one from the first list, and one from the second. However, Cyrus IMAPD will discard the duplicate. Why? Because duplicate emails take up more space and are wasteful, and rarely do you need to see the same mail twice. Also, in cases where someone accidentally sends out the same message many times, you'll only see it once. Cyrus checks the actual "Message-ID:" header to verify that the message is a duplicate, so in some cases you may see dupes if the mailing list software (erroneously) rewrites the Message-ID.


How do I know I have new mail?

In the days of local mail storage in flat text files, it was simple to run a program that would look at the time stamp of the file to see if you had new messages. However, with an IMAP system, you have to speak to the mail server to find out if you have any new mail. Most IMAP clients, such as Thunderbird, have a way to notify you of new messages, as does Pine (many in the department will leave Pine running all the time, since there's no harm in having multiple IMAP clients access the same mailbox at the same time). There is also a program called 'tkbiff' which attempts to replicate the behavior of 'biff' (the program that would notify you of new mail in a flat file), but:

  • It doesn't work very well
  • It is not very secure
  • It is not supported

So if you get a copy and get it running, that's great - but if it breaks, you can keep the pieces.


Are there any tools for accessing mail (like grep)?

Since mail isn't stored in a text file you can access, using something like 'grep' on your mail spool is not possible. However, there are some IMAP utilities which are available that may help you to do what you need.


I created some folders in my client, but they're not in Squirrelmail!

Fear not! Squirrelmail uses the IMAP subscription model. Similar to how Usenet worked, before Squirrelmail will let you see the contents of a folder you have to subscribe to it. In this way, if you have a bunch of folders you don't want to see if you login via Squirrelmail, you can turn them off and they won't be visible. Conversely, if you create new folders and want to see them, you have to subscribe to them first. Click on "Folders" at the top of the page, and near the bottom you'll see two sections. The left side shows folders to which you are subscribed, and the right folders to which you are not yet subscribed. Select whichever folders you desire to change (one side at a time, please) and then click the corresponding button beneath the section to submit the changes. You may need to refresh the folder view on the left panel (click "Check email") for the changes to appear.


How do I export my email?

If you're leaving the department and closing your account, you'll probably want to take your email with you when you go, either as live copies or as an archive. Or maybe you want to use some mail program that doesn't use IMAP, and needs to see the email folders as flat text files. There's a few ways to do this:

Pine and Thunderbird can be configured so you can copy your mail from Peyton to your new mail server (as long as it's IMAP). Pine can also (as can Fetchmail) be used to download text copies of your email, in a few different formats (the most common being 'mbox' format). Depending on your needs, follow one of the links above to see how to get the job done. For automated downloading of mail, Fetchmail is probably the best choice.


How do I filter email?

You'll want to either read a bit about Sieve - the filtering language used by Cyrus IMAPD - and create a Sieve script, or have a look at Websieve which we run here as a web frontend to the Sieve language. Most things you might need to do can be done through the interface in Websieve, but worst case scenario you can upload straight Sieve scripts to it as well.


How do I manage delivered (and tagged) spam?

See Websieve#Spam filtering for a few examples.


Current event marker.svg This article or section needs to be updated.
Parts of this article or section have been identified as no longer being up to date.
Please update the article to reflect recent events, and remove this template when finished.
Clockimportant.svg

See also Mailing Lists for information about the mailing lists available.


The nasty details

This section explains all the nitty-gritty of the mail system in use here. For casual users, it's probably not important, but the more curious may want to know exactly what programs touch their email on its travels through the Ether.


SMTPD

After running Sendmail for a long time, I had a look at Postfix before one of the mail server upgrades. While Sendmail configuration has become so cumbersome that there's entire books dedicated to the configuration files (O'Reilly's Bat Book), Postfix's configuration is fairly straightforward. With mostly plain-English directives, I was able to setup a Postfix system that mirrored our Sendmail system in a fraction of the time. Whats more, the Postfix system allowed for sending a mail through a filter and then processing it on the other side, and authentication/authorization and encryption for security. Sendmail is only used for secondary systems now, and only because it's the default installation at the time.

We're running Postfix 2.2.x compiled with SMTP AUTH via SASL (which in turn gets information through PAM using LDAP; there's also a local LDAP slave running on the mail server for its own use and to make the machine autonomous). Postfix is configured to use TLS encryption over the standard port (25), allowing for secure authentication to the server for relaying mail.


IMAPD

Originally, all email was delivered to a file named for each user, which was cross-mounted on all machines in the building. While this allowed for the use of things like 'grep' to search your inbox, it had a number of downsides:

  • Mail server downtimes could cause strange NFS locking issues on every machine in the building
  • Lock contention
    If the mail server process and your client were both trying to write to the file at the same time, there was a very real chance that mail would be lost, and/or the file completely corrupted.
  • Remote access to mail was impossible - only option was to login and use Pine

To make the system more robust, I moved everything over to Cyrus IMAPD. Cyrus is a "black box" mail system - end users do not (and in fact, can not) login to the mail server, but only speak to it over IMAP. This makes the system better for us, since it no longer needs to rely on the availability of the home directory server - in fact, our mail server could be physically removed from the building, and plugged in somewhere else on campus, and as long as it can get the same IP address it uses while here, it will continue to function fully. Cyrus IMAPD allows for filtering via the Sieve filtering language (similar to how Procmail works). Cyrus is also compiled to use SSL or TLS encryption depending on the connection.

We use Cyrus IMAPD 2.3.x.


Spam & virus filtering

In this day and age, it's impossible to run a mail server without filtering for spam (and it's irresponsible to run one without virus scanning as well). Early on in the process, I added AMaViS as a filter in the mail processing pipeline to do just that. It scans every incoming and outgoing mail for viruses, and scores mails based on their content to see if they're spammy or not. The spam filtering comes from SpamAssassin, a powerful filter which is very configurable and routinely updated.

We're running amavisd-new version 2.3.x, and SpamAssassin 3.1.x.


Mailing lists

Originally, mailing lists here were nothing more than "exploders"; an alias which mapped to a long list of recipients, so an email sent to the alias would just be expanded to all the people on the list. While effective, these are harder to maintain and not very powerful. By installing and using Mailman, there's a lot more flexibility available in our lists, including the ability to delegate someone other than us as the maintainer and moderator (important in the case of "user-run" lists for projects and such). We can also offer archiving for mailing lists this way, so that all mails can be stored on the server for future reference.

We currently run Mailman 2.1.x.


Squirrelmail

As mentioned above, Squirrelmail allows anyone to read their email with just a web browser over a secure connection. A web-based IMAP client, Squirrelmail has helped those who might not be at a computer capable of doing SSH, or installing Thunderbird for a full-fledged IMAP client. Some people even use it full-time, instead of a regular IMAP client.

Squirrelmail 1.4.x is the version currently installed.


How it all fits

An email coming in from outside the building to be delivered to a local account takes the following path:

  1. External Postfix
    Postfix takes the details of the email, and when the remote client/server finishes sending it does not immediately reply with an acceptance or rejection message - instead, the connection is held open while processing takes place. Because of this, we never accept responsibility for an email until it's been delivered, so if any stage fails it is still up to the sending computer to either retry or notify the original sender of the failure.
  2. AMaViS
    amavisd-new runs the email through a virus scanner, and through SpamAssassin to assign a score to the message. If the virus scan fails, the message is rejected and Postfix will inform the sending system. SpamAssassin tags all emails with their score, no matter the value, for debugging purposes. If the score is above 3.0, the email will also be flagged as spam but still delivered (it's up to the user if they want finer grained filtering at that point). If the score is 5.0 or above, the mail is rejected.
  3. Assuming above passed, mail moves to an internal Postfix queue
    This queue deals with local delivery; either handing the message off to Cyrus, or forwarding it to another computer for processing (SDSS and some other lists are processed elsewhere). The internal Postfix checks to make sure the recipient exists, and if not will reject the mail (which carries through the open pipes back to the sender). At this point, if this Postfix accepts the message, amavisd-new sends back an acceptance to the external Postfix, which in turn tells the sending computer that the email was accepted.
  4. Cyrus via LMTP
    The internal Postfix speaks to Cyrus with the Local Mail Transfer Protocol (LMTP), a lightweight version of SMTP for local intra-network processes. Cyrus will accept the message from Postfix, process it through Sieve (if needed) and save it to a mail folder for later viewing.
Retrieved from "https://docs.astro.princeton.edu/wiki/index.php?title=Email&oldid=456"