Email

From Peyton Hall Documentation
Jump to navigation Jump to search

Email has become one of the most important means of communication among people here, in part because they may not be physically anywhere near their collaborators. Also, being a 'non-intrusive' communication is helpful - if you send someone an email and they're not available to reply right away, they can take their time formulating a response instead of needing to on the spot, or promising a return phone call later when time is available. Quite handy for those 4AM questions that pop up. This section will explain a bit about email, including links for setting up various mail clients in the building, as well as some of the details of our email setup here in Peyton Hall.


Email basics

While email has become an important and common method of communication, many don't quite understand the details that go behind pressing 'send' in your client. What follows is a brief explanation of how that all works.


How it works

When you compose an email to send somewhere, your mail client generates a series of text which describes the details of the message. There's a few headers written, including who the mail is from (your From: address), the destinations (To:, CC: and BCC: for blind carbon-copied messages - those which are sent somewhere, but not written in the message itself so other recipients are unaware it was sent to them), the Date: header saying when the message was written, and a Subject: header for the subject line of the message. Some clients will also include other headers, such as priority (which works differently in different mail clients) or headers to identify the client program and version. The email is now ready to be sent to a server that speaks Simple Mail Transfer Protocol, or SMTP.

In some cases, when you send a mail, the message is handed off to a SMTP client program such as Sendmail running from the command line (/usr/bin/sendmail). This client program will take the message and handle speaking SMTP to the mail server. In other cases, your mail client will speak SMTP directly to the server. In both cases, the conversation is fairly short; the client tells the server what information it needs, known as the "envelope headers", so called because they could be different than what the message itself has for recipients and headers (ie, if you're sending a BCC of an email to someone, the envelope recipient is that person, even though their name doesn't exist in the actual headers of the email). Once the headers are passed, the client tells the server it's now sending the data of the message, sends the data, then signals that the message is done. At that point, the mail server can either accept the message completely - usually giving a kind of tracking number for the email - or reject it, which should cause the client to pop up an error that the mail could not be delivered. Should the mail be accepted, it is now the responsibility of the mail server which accepted the mail to either deliver it one step closer to the recipient, or return a message to the sender to say that the message could not be delivered. It is unacceptable to have a legitimate email discarded by a mail server and no "delivery status notification" sent back to the sender of the message.

Once the SMTP server holds the mail, it looks at it to determine who needs a copy, and where they are. Through some clever Domain Name Service (DNS) entries, it can tell where a mail for a certain person needs to be delivered, and will contact that server (unless this SMTP server is configured to send all mail to a "relay host", in which case it will relay the message up the chain and that host will now be responsible for delivery - this is common if you have an ISP which requires all emails be sent through their own servers, and not directly from your home computer). Once the SMTP server knows where to hand off the message, it initiates a connection to that server and repeats the process, adding to the message a few headers to mark the email's travels through itself (in most cases, a "Received:" header to show that it received the message and handed it off, while in some cases there may be a few headers added if the message was scanned for spam & viruses). This process will continue to repeat, until an SMTP server holds the message and must deliver it locally somehow.

When the local SMTP server holds the mail, it may be delivered many ways depending on the local configuration. In many cases, the mail will be scanned for viruses and spam content, and assuming the message passes it will be delivered. Some mail storage systems work with plain text files for holding messages, so the mail server can deliver them internally. Others use the Local Mail Transport Protocol (LMTP) to pass a message on to a mail server daemon which clients will connect to for reading mail (using either Post Office Protocol v3 (POP3) or Internet Mail Access Protocol (IMAP)).

As mentioned before, if a SMTP daemon cannot deliver a message any closer to the recipient for some reason, it has two choices for what to do:

  1. Reject the message, and the sending program (SMTPD or client) is now responsible for notifying the sender somehow
  2. Generate a Delivery Service Notification (DSN) message to the sender, notifying them that their email could not be delivered and why

In some cases, DSNs are sent because the recipient does not exist, or their mailbox is full. Sometimes it may be because a virus was detected in the email, or because the mail appeared to be spam (or had enough characteristics to make it 'spammy'). It could also be that the recipient's SMTP server is not responding, in which case you'll see the familiar "Message could not be delivered for 4 hours; will keep trying for up to 5 days" DSN. These mean just what they say: the message could not be delivered in the last four hours, but the program will keep trying to deliver it (usually once every hour) until it is five days old, at which time it will send another DSN to let you know the mail could not be delivered at all. Because SMTP servers will hold messages like this, mail tends to always go through even if a mail server is down for a while. All those trying to send mail to it will just hold their messages and wait until a later time, so the worst problem you'll usually see is that messages are delayed an extra hour from when the server is brought back online.

If you'd like to know how our mail system here is configured, see the below section "The nasty details".


Using email in Peyton Hall

There's a few ways to read your email while in Peyton Hall, which are outlined here. A word on "support" first, however. Because there are so many different ways to access email, we cannot possibly support every permutation of them. However, we have selected a few programs which are widely used and easily setup so that we can guarantee support with those programs. This does not mean you must run only one of those programs; you're free to use whatever software you like for reading and sending emails. However, if the program you've chosen does not work properly, or has some kind of problems that are not a fault of the mail server itself, you will be on your own to figure out the problem and corresponding solution. See Other clients below for information if you wish to use an unsupported email client.

Note: You MUST login to your email inbox with a new account before it is created! If you do not login at least once, you'll have no mailbox yet. Instructions are just below...
Note 2: If you do not configure your real name in your mail client it will usually show up as "username@astro.princeton.edu". If you care about people seeing your real name, you should edit it in the client.


How to activate your E-mail Account at Peyton Hall

In order to prevent people who don't wish to use their Peyton e-mail account from having mail pile up and go unread, you must explicitly activate your e-mail account and create your INBOX. If you do not do this, mail to you will bounce.

  1. First login to your account. We recommend using Alpine at the command line of any of our managed Linux workstations. You will need to do this at some point to set your password anyway, so you might as well do this while you're doing that. So, run 'alpine' from the command line to launch the program.
  2. You will get an error that your Inbox does not exist. Ignore it and hit 'L' to open the collection list.
  3. Choose the "Global Folders" collection and you'll see the global folders on the system (LearnSpam mainly).
  4. There if you press 'A' to add a new folder and type 'INBOX' you'll create the inbox.

That should do it.

If you'd prefer, you can also activate your e-mail account from the web by logging into Roundcube at https://webmail.astro.princeton.edu/roundcubemail/

  1. Login to Roundcube
  2. Ignore the error about your mailbox not existing. We're going to fix that below...
  3. Hit the 'Settings' button
  4. Click 'Folders'
  5. Set the Folder Name to 'INBOX' and the Parent Folder should be left as ---
  6. Hit Save

Supported clients

Alpine

From any Unix/Linux machine in the building, there's two main options for viewing your email. The first is Alpine, a text-based email program which will work over SSH connections as well as while you're sitting in front of a machine in Peyton. While it may not have all the whiz-bang features of a GUI mail client, Alpine is still in use by many and is a good no-frills (or low-frills) program to use for accessing email. Because of the way we have it configured, we can also guarantee that even if nothing else works, Alpine should be able to read your mail no matter what - as long as the mail server itself is running. You can find more information about Alpine in the article linked above in this section.


Thunderbird

Another supported client in the building is Thunderbird. This is what became of the mail client portion of the Mozilla suite. Thunderbird is a great IMAP client with many plugins and extensions that can help you effectively and efficiently use your email. Since it supports encryption and secure connections, you can configure Thunderbird for use on your laptop, and it will work the same if you're in the building or sitting at home. More information on Thunderbird is available in its own article (linked above).


Roundcube

Roundcube is an IMAP client which is accessed via a web interface. We run Roundcube on the mail server directly, and using a SSL certificate for encryption means your user name and password are not sent in-the-clear for others to see. You can do just about anything through Roundcube as you would with any other mail client, with the added benefit that you don't need anything more than a web browser to access it. The downside, of course, is if you don't trust the computer in front of you (for example, in an Internet cafe or web kiosk). But if you're visiting some other institution and have web access, you can read and reply to emails with Roundcube. You can access Roundcube at https://mail.astro.princeton.edu/roundcube/. Note that like any other email client, if you do not set your real name in the settings your name will be omitted from emails and they will only have your email address. If you care about your real name being in the "From" line of emails, you will need to edit that in Settings -> Identities -> <username> -> Display Name


Other clients

For unsupported email clients, we have some information here which may be useful in getting it setup to access the mail server. If you have information about another client, you're welcome to either add it here, or if it's a large article create a new page for it and link to it in here instead.

  • Mail.app is the default email client that comes with Mac OS X machines.


Android

Unfortunately, different Android devices use different implementations of the e-mail client, so it's difficult to come up with a catch-all set of instructions. If you have trouble getting outgoing e-mail working with the settings under the section below, try disabling the option to encrypt connections to the outgoing mail server. Our mail server will reject non-encrypted attempts to login, anyway, so it should be safe to try. The problem is that by checking that box, some Android devices force a mode of SSL which our server doesn't support. Instead, our server uses TLS, which works in a different manner (and which Android should support). I've tested this setup on an Motorola Droid X running Android 2.1 (aka Eclair) and it works.

LG Email works for reading incoming e-mail, but many others do not.

Use your own

If none of these clients fit your needs, you can always pick your own for whatever you'd like to do. The information you'll need to know is:

  • SMTP (outgoing) server: mail.astro.princeton.edu port 587
  • IMAP (incoming) server: mail.astro.princeton.edu port 993
  • Security/encryption must be turned on
    This means that the IMAP port will be 993 instead of 143, and SSL must be turned on. For SMTP, you'll need to tell your client to use TLS encryption. You will also have to supply your user name and password to the client, as it will have to authenticate to the mail server to be able to send mail to recipients not at the astro domain (ie, you're relaying mail through our server from home, and the recipient is in Berkeley).


Accessing email from outside Peyton

As nice as the building is, you can't always be here. So how do you access your mail from somewhere else? Simple!

  • If you have Thunderbird configured on your laptop or home computer, it will work just the same while outside the building as it will while connected to our network.
  • If you have SSH access from wherever you're connecting, you can login to any machine and run Alpine.
  • If all you have is a web browser, you can use Roundcube


Email FAQ

This section lists some frequently asked questions about email, and their answers (or links to their answers).


Because it's hard to understand and unnatural to read.

Why shouldn't I top-post an email reply (where the response goes at the top of the email, and the original message gets pushed down to the bottom)?


How do I ...

...get all my mail in one place?

If you have an account with OIT, you also have an email address on their servers. However, many people don't like to have to check multiple accounts, and would rather have their mail all in one location.

NOTE: We DO NOT recommend forwarding mail anymore.  In today's age, forwarding your email does not always work, as senders can set specific rules on the handling of email from their domains, and places such as Google will look at a forwarded message and discard it for not matching those rules.  It is much safer to simply have your mail land in one location and deal with it there, rather than forwarding it everywhere.  Note that you WILL LOSE MAIL in some cases of forwarding, and this is not something we can control at all - it is completely at the decision of the sender's mail servers and your eventual recipient honoring those wishes.


To forward OIT mail to here
  1. Follow the instructions in OIT's KB here
To forward astro email elsewhere

If you would like your email here to be delivered somewhere else, see the forwarding section of the SmartSieve article. Note that using a .forward file will not work.


...export my email?

If you're leaving the department and closing your account, you'll probably want to take your email with you when you go, either as live copies or as an archive. Or maybe you want to use some mail program that doesn't use IMAP, and needs to see the email folders as flat text files. There's a few ways to do this:

Alpine and Thunderbird can be configured so you can copy your mail from Peyton to your new mail server (as long as it's IMAP). Alpine can also (as can Fetchmail) be used to download text copies of your email, in a few different formats (the most common being 'mbox' format). Depending on your needs, follow one of the links above to see how to get the job done. For automated downloading of mail, Fetchmail is probably the best choice.


...filter email?

You'll want to either read a bit about Sieve - the filtering language used by Cyrus IMAPD - and create a Sieve script, or have a look at SmartSieve which we run here as a web frontend to the Sieve language. Most things you might need to do can be done through the interface in SmartSieve, but worst case scenario you can upload straight Sieve scripts to it as well.


...manage delivered (and tagged) spam?

See Filtering Spam for an example.


...subscribe to such-and-such mailing list?

See the Mailman interface.


...learn more about Mailman & mailing lists?

You can browse the documentation at the main Mailman website, http://www.list.org/docs.html.


...know this email was sent by someone?

If the email is not cryptographically signed (say, with GPG), you don't. Emails can be forged, very easily. They can appear to come from anywhere, and with varying degrees of believability; anything from "no way in hell" to "only the NSA knows." If your work requires you verify that an email came from the person who appeared to send it, then you both need to install GPG (and if you use Thunderbird, you'll want the Enigmail extension; see here) and learn how to use it.


...send or receive large emails?

See the article on large emails which describes what they are and what to do.


...find out why a mail got marked as spam?

In order to see why a message was flagged as spam, you must look at the headers of the message. Spamassassin will include a header named X-Spam-Status which includes the score and all the tests which matched the message, and what their result added to or subtracted from the score. If you would like us to look at it and describe it more, you need to send us the message with all headers intact. You can also see below for another way to easily get us the message for inspection.


...teach the spam filter to learn from mistakes?

The spam filter can be taught that a message is spam, or not spam (aka 'ham') and will learn over time. Using Bayesian logic, the filter teaches itself when a message has a very low score that its contents tend to not be spam, and when a message has a very high score that its contents tend to be spam. This way, those messages which are in the middle can be pushed one way or another based on the content of the message and Bayesian inference.

If a message was erroneously flagged as spam, or let through when it clearly was spam, you can help to teach the filter where it went wrong. There is a global folder on the mail server, called LearnSpam, where you can put those messages which were incorrectly classified. You don't need to let us know about a message being there, as we can see them and shouldn't have trouble figuring out if it was wrongly flagged or allowed to pass, unless you want us to explain why a message was flagged. In that case, please send us a message describing the mail you copied there and what you'd like us to tell you about it.

Note: Everyone can copy messages into this folder, but cannot read messages there. Only the sysadmins can read messages placed in this folder, and in the course of training the spam filter we may do so. Rest assured that messages which may find their way to the folder will be kept in confidence, and that only the admins can read messages there (and we only care about the technical aspects of the spam filtering, not really about the content of the message anyway except how it may have been detected as spam).


I created folders in my client, but they're not in Roundcube!

Fear not! Roundcube uses the IMAP subscription model. Similar to how Usenet worked, before Roundcube will let you see the contents of a folder you have to subscribe to it. In this way, if you have a bunch of folders you don't want to see if you login via Roundcube, you can turn them off and they won't be visible. Conversely, if you create new folders and want to see them, you have to subscribe to them first. Click on "Settings" at the top of the page, then click the "Folders" tab. Under the "Subscribed" header, check the box for any folders you want to see, and uncheck the box for any folders you want to hide. Changes get saved automatically, so just click "E-Mail" at the top of the page to return to your Inbox once you're finished.


Why don't I see duplicate emails?

Say for example, you're subscribed to two mailing lists. One list, however, is subscribed to the other, so normally you would see two copies of the message - one from the first list, and one from the second. However, Cyrus IMAPD will discard the duplicate. Why? Because duplicate emails take up more space and are wasteful, and rarely do you need to see the same mail twice. Also, in cases where someone accidentally sends out the same message many times, you'll only see it once. Cyrus checks the actual "Message-ID:" header to verify that the message is a duplicate, so in some cases you may see dupes if the mailing list software (erroneously) rewrites the Message-ID.


Why am I not getting important system announcements?

Make sure you whitelist the address listed in Requesting assistance in whatever mail client you use. When we send out announcements, we will try to always use that as the "From" address. Failure to whitelist the address could mean that mails from help are flagged as spam in your email client, and discarded (or hidden).


People said mail to my new account bounces!

Have you ever logged into your mailbox yet? If you have not, then the mail system hasn't created your account yet. Most people will login within a day or so of getting their account, but if you didn't login yet, then your inbox hasn't been created yet, and mails to your account will of course bounce.


I tried mailing a new user and it bounced!

See above, and tell the user to go login to their mailbox so you can email them. Or perhaps they're not using our mail accounts here, and they use their Princeton account, or somewhere else. Either way, while help could create their account, we can't force them to use it, so this isn't a problem we can fix.


I got this strange email...

While our mail server does a pretty good job of keeping viruses out of your mail, and the majority of people here read their mail on Unix machines that are impervious to Windows viruses anyway, some things occasionally "get through" (especially if a virus starts running rampant before the virus scanner's database is updated, either because the company has not released a new version yet or because the update hasn't run yet). Here's some things to keep in mind:

  1. If you read mail on a Unix machine, and you opened an email with a virus in it, don't fret.
    Just delete the mail and move on. You're fine. Trust me.
  2. If you read mail on a Windows machine, and opened an email with a virus, don't fret.
    Delete the mail, and run your favorite virus scanner to make sure you're fine. More likely than not, you will be uninfected (especially if you don't use MS Outlook). And either way, NEVER OPEN ATTACHMENTS YOU DON'T RECOGNIZE!!
  3. If you get an email that seems strange, but either has no attachments or has a text attachment that says something about a virus being deleted, just delete the mail.
    Some admins subscribe to the theory that all mail should always get through, so if their scanners find a virus in an email they will "cleanse" the mail (ie, remove the virus, sometimes replacing it with a text file stating that a virus was removed) and send the body of the message through unharmed. Due to the way viruses act anymore, I subscribe to the theory that a mail should only go through in its original form; if there was a virus in the mail, then the entire message is blocked. In the cases of non-mass-mailing viruses (more later), a note is sent back to the mail sender telling them that the mail was discarded and why.
  4. If you get emails about a message from *yourself* being returned or blocked due to a virus, and you didn't mail anyone anything, DON'T WORRY ABOUT IT.
    Nobody has hacked into your account, and nobody is sending mail from your machine. One of the biggest virus methods of transport nowadays is through email, and many viruses (those that end with "@mm", meaning "Mass Mailer") will take the address book of someone infected, and send itself to everyone in the list. The trick is, to hide the true location of the virus, they will also pick *another* name in the address book, and use that as the "From:" address. This is quite common, and why things like GPG have come about with digital signatures on email to guarantee that a mail is from the person it claims to be from. It is very trivial for anyone to send mail as anyone else, without ever breaking into anything or knowing another password, and only a trained eye can spot the trickery in the headers (and sometimes only when one looks through the mail logs on the servers in question). Also, a mail such as this does not mean your computer has a virus (unless you actually sent something to the person whose mail server bounced the message, in which case you should run your virus scanner to be sure). If you want to find out who is actually infected, look at the headers of the message, and find the last header that mentions "mail.astro.princeton.edu" - it will usually say something like "Received: from smtp.foo.com by mail.astro.princeton.edu". In the example here, "smtp.foo.com" is who gave the mail to us, so either that machine or one that it trusts is likely infected. You can narrow this down by trying to find someone who has both yourself and the "recipient" of the bounced email in their address book. That is likely the person with an infected computer.
  5. If you want a good resource for virus information, including links to removing some viruses and worms, see the Symantic Threat Explorer.
    This is not only a good place for reading about viruses old and new, but it also explains what they do, how they work, and how to remove them. Also, if you get a mail that you think might be a hoax, you can usually find it here as well.


The nasty details

This section explains all the nitty-gritty of the mail system in use here. For casual users, it's probably not important, but the more curious may want to know exactly what programs touch their email on its travels through the Ether.


SMTPD

After running Sendmail for a long time, I had a look at Postfix before one of the mail server upgrades. While Sendmail configuration has become so cumbersome that there's entire books dedicated to the configuration files (O'Reilly's Bat Book), Postfix's configuration is fairly straightforward. With mostly plain-English directives, I was able to setup a Postfix system that mirrored our Sendmail system in a fraction of the time. Whats more, the Postfix system allowed for sending a mail through a filter and then processing it on the other side, and authentication/authorization and encryption for security. Sendmail is only used for secondary systems now, and only because it's the default installation at the time.

We're running Postfix 2.10.x compiled with SMTP AUTH via SASL (which in turn gets information through PAM using LDAP). Postfix is configured to use TLS encryption over the standard port (25), allowing for secure authentication to the server for relaying mail. We also have it listening on port 587 (mail submission from clients), which requires TLS and SMTP AUTH for all connections (even local deliveries).


IMAPD

Originally, all email was delivered to a file named for each user, which was cross-mounted on all machines in the building. While this allowed for the use of things like 'grep' to search your inbox, it had a number of downsides:

  • Mail server downtimes could cause strange NFS locking issues on every machine in the building
  • Lock contention
    If the mail server process and your client were both trying to write to the file at the same time, there was a very real chance that mail would be lost, and/or the file completely corrupted.
  • Remote access to mail was impossible - only option was to login and use Alpine

To make the system more robust, I moved everything over to Cyrus IMAPD. Cyrus is a "black box" mail system - end users do not (and in fact, can not) login to the mail server, but only speak to it over IMAP. This makes the system better for us, since it no longer needs to rely on the availability of the home directory server - in fact, our mail server could be physically removed from the building, and plugged in somewhere else on campus, and as long as it can get the same IP address it uses while here, it will continue to function fully. Cyrus IMAPD allows for filtering via the Sieve filtering language (similar to how Procmail works). Cyrus is also compiled to use SSL or TLS encryption depending on the connection.

We use Cyrus IMAPD 2.4.x.


Spam & virus filtering

In this day and age, it's impossible to run a mail server without filtering for spam (and it's irresponsible to run one without virus scanning as well). Early on in the process, I added AMaViS as a filter in the mail processing pipeline to do just that. It scans every incoming and outgoing mail for viruses, and scores mails based on their content to see if they're spammy or not. The spam filtering comes from SpamAssassin, a powerful filter which is very configurable and routinely updated.

We're running amavisd-new version 2.11.x, and SpamAssassin 3.4.x.


Mailing lists

Originally, mailing lists here were nothing more than "exploders"; an alias which mapped to a long list of recipients, so an email sent to the alias would just be expanded to all the people on the list. While effective, these are harder to maintain and not very powerful. By installing and using Mailman, there's a lot more flexibility available in our lists, including the ability to delegate someone other than us as the maintainer and moderator (important in the case of "user-run" lists for projects and such). We can also offer archiving for mailing lists this way, so that all mails can be stored on the server for future reference.

We currently run Mailman 2.1.x.


Roundcube

As mentioned above, Roundcube allows anyone to read their email with just a web browser over a secure connection. A web-based IMAP client, Roundcube has helped those who might not be at a computer capable of doing SSH, or installing Thunderbird for a full-fledged IMAP client. Some people even use it full-time, instead of a regular IMAP client.


How it all fits

An email coming in from outside the building to be delivered to a local account takes the following path:

  1. External Postfix
    Postfix takes the details of the email, and when the remote client/server finishes sending it does not immediately reply with an acceptance or rejection message - instead, the connection is held open while processing takes place. Because of this, we never accept responsibility for an email until it's been delivered, so if any stage fails it is still up to the sending computer to either retry or notify the original sender of the failure.
  2. AMaViS
    amavisd-new runs the email through a virus scanner, and through SpamAssassin to assign a score to the message. If the virus scan fails, the message is rejected and Postfix will inform the sending system. SpamAssassin tags all emails with their score, no matter the value, for debugging purposes. The process of spam scanning is:
    1. If the score is above 3.0:
      • X-Spam-Flag is added to the headers with the value "YES".
      • X-Spam-Status header (added regardless) will now start with "Yes", followed by which tests were related to the mail and the score output from each test.
      • X-Spam-Level will contain a number of stars according to the whole number value of the score (3.1=***, 3.9=***, 4.0=****)
    2. If the score is 5.0 or above, the mail is rejected entirely.
  3. Assuming above passed, mail moves to an internal Postfix queue
    This queue deals with local delivery; either handing the message off to Cyrus, or forwarding it to another computer for processing (SDSS and some other lists are processed elsewhere). The internal Postfix checks to make sure the recipient exists, and if not will reject the mail (which carries through the open pipes back to the sender). At this point, if this Postfix accepts the message, amavisd-new sends back an acceptance to the external Postfix, which in turn tells the sending computer that the email was accepted.
  4. Cyrus via LMTP
    The internal Postfix speaks to Cyrus with the Local Mail Transfer Protocol (LMTP), a lightweight version of SMTP for local intra-network processes. Cyrus will accept the message from Postfix, process it through Sieve (if needed) and save it to a mail folder for later viewing.