Windows 7

Troubleshooting Mail Flow

Message delivery is arguably the most important piece of Exchange Server 2013, and it's only fitting that Microsoft has provided a formidable arsenal of troubleshooting weapons to deal with pesky delivery failures. You'll have your pick of tools, from self-serve choices, such as message tracking in the Exchange Control Panel, to several forms of tracing, to the inevitable cmdlets.

However, just because you have an array of choices doesn't mean you have to use them right away. Again, it's important to approach a message-delivery problem with clear eyes and ask probing questions about what you're facing. Remember the example earlier in the tutorial, with the end user who couldn't send email? There were a number of plausible explanations for this, some of which didn't involve message delivery at all! So it's still important to gather the essential information:

Can the user send any emails at all? Is nondelivery restricted to a subset of users?
Does the user receive a delivery status notification? If so, what is the delivery code?
Is the recipient in the same Exchange Server organization or in a different organization (presumably on the Internet)?
How close do messages get to their destination?
What is the messaging path between the end user and the recipient?

These questions, though relatively simple, conceal a bewildering list of possible root causes. Consider the impact on message delivery on the following:

DNS Failure
Mailbox servers can't locate A records and therefore can't reach next-hop servers.
Site Link Failure
No site link exists between sender and recipient.
Transport Failure
Transport services on all of the Mailbox servers in the user's site are inaccessible.
Transport Agent
A transport rule prevents this email from reaching the recipient (because of sender restrictions, content restrictions, or recipient issues).
Mailbox Limits
The recipient's mailbox is full, but nondelivery reports do not reach the sender for whatever reason.
Messages Stuck in Queue
A transient failure has temporarily stopped messages at a back-off location.
Back Pressure
Transport services are temporarily throttling message delivery due to resource constraints.

This isn't even an exhaustive list, but it includes a wealth of possibilities. Now, there are few listed here that you would probably detect by performing the basic troubleshooting steps we covered earlier in this tutorial (like DNS failure or transport failure). We'll begin with a simple cmdlet to check basic mail flow, which is typically the first step in locating undelivered messages, and then move on to message tracking and agent logging.

Using Test-Mailflow

Assuming you've done some of the basic checking (is the user's client connected to a database, are transport services available, and so on), you'll probably want to test that mail is flowing in the organization. There's an aptly named cmdlet for just this job: Test-Mailflow.

The cmdlet's basic function is simply to send and receive email from the system mailbox of the target server, but it can do so much more. The syntax is extremely simple:

Test-Mailflow followed by the source server, then-TargetMailboxServer, -TargetDatabase, or -TargetEmailAddress. The different options mean that you can start with a Mailbox server, and if that test succeeds, focus on the user's database and then the user's email address. If you've deployed multiple databases in a DAG, you should skip the first step and start with -TargetDatabase.

Test-Mailflow EX1 -TargetEmailAddress matcary@netlogon.com

RunspaceId 		: 0848e0b8-4228-4195-b1ee-c4c967ac9a41
TestMailflowResult 	: Success
MessageLatencyTime 	: 00:00:02.5631250
IsRemoteTest 		: True
Identity 		:
IsValid 		: True

The output from Test-Mailflow doesn't need much interpretation. The most important piece is the TestMailflowResult property. If it reads Success, you know that you can reach that server, database, or email address, and you know that email is flowing, at least for some combination of user and database. The next property, MessageLatencyTime, lets you know the time it took for the message to reach the destination server. The IsRemoteTest property simply indicates whether the message left the server (this will also be True if you use the - TargetEmailAddress parameter).

However, if your test fails, the TestMailflowResult property reads *FAILURE*, and that's unfortunately the only indicator you receive-no messages about where the failure might have occurred or other useful information. That's when you need to start figuring out where the messages are stopped, and for that we need to move into different tools. We'll start with the Queue Viewer and then look into message tracking.

Utilizing the Queue Viewer

The Queue Viewer is a tool that is available through an MMC snap-in named Exchange Toolbox. The Exchange Toolbox provides a virtual toolbox in the MMC that contains the Queue Viewer, alongside a number of other useful tools (some of which we'll cover later in this section). A big believer in truth in advertising, the Queue Viewer allows you to, yes, view the contents of the various delivery queues. Obviously, you need to connect to a Mailbox server to use this tool, but you can open the Queue Viewer from anywhere and then connect to the appropriate Mailbox server. The interface for the Queue Viewer is shown.

It's largely unchanged from Exchange Server 2007, but it didn't need any enhancements; it tells you the status of the queues, how many messages are pending delivery, and where the mail is heading, among other things. The list of queues is a bit thinner than in previous versions of Exchange Server (particularly when compared to the positively garrulous Exchange Server 2003), but if there's a problem with a particular queue, it'll be listed here.

Most of the columns in the Queues tab are relatively self-explanatory, but here's a brief rundown of what you'll see on this page:

Next-hop Domain
This is where the mail is heading next, whether a server in a different site, a server in the same site, or an Internet host.
Delivery Type
This indicates where the messages are heading next in their journey to the recipient.
Status
This simply indicates whether the queue state is Active (sending messages at the moment), Suspended (stopped through administrative action), Ready (able to send messages should any arrive), or Retry (unable to send messages). Queues in a Retry state are the most obvious candidates for additional review and analysis, but remember that queues can fail because of the sending server as well as the recipient.
Message Count
This lets you know how many messages are stuck in this queue.
Next Retry Time
This is applicable only to queues in a Retry state and lets you know the next time Exchange Server will attempt to "wake up" the queue for delivery.

You can perform a small number of tasks on the queues, including temporarily stopping them (Suspend), forcing them to connect if they've failed (Retry), or deleting all the messages (Remove messages with or without a non-delivery report). This can be useful for restarting mail flow after you've resolved a problem somewhere else in the environment or for deleting a quantity of undesired email. The Actions pane at the right of the MMC will display only valid actions for the queue you've selected.

The Messages tab has similar information to the Queues tab, and it's generally most useful when you've clicked a queue and then selected View Messages. You can click the Messages tab right away, but it'll show you every message queued on the server, which might take a little while for a busy system. The columns for the Messages tab are similar to those on the Queues tab:

From Address
This is the address of the sender, taken directly from the SMTP envelope.
Status
This indicates the message status, which is generally the same as the parent queue but is also influenced by administrator action (for example, if the administrator has tried to delete the message while it was being delivered, the message will appear as Pending Remove).
Size (KB)
This is the size of the message, displayed in kilobytes.
SCL
This is the spam confidence level (SCL) rating; the values range from -1 through 9, with -1 representing authenticated email and 9 representing email that is almost certainly unsolicited commercial email (UCE, or spam).
Queue ID
This value indicates the queue in which the message appears. If you chose a specific queue and then selected View Messages, this should be the same for all messages and should reflect the queue you chose on the Queues tab.
Message Source Name
This indicates the Exchange Server component that delivered the message to this particular queue. Depending on your architecture, this could be a Hub Transport server in another site, a Mailbox server in the same site, or possibly even an application or client submitting a message directly to the Hub Transport server via SMTP.
Subject
This is the subject of the email, taken from the SMTP envelope.
Last Error
This indicates the last error experienced when attempting delivery of this message. This typically appears only if the message is in a Suspended, Retry, or Pending state.

The Queue Viewer is useful for locating a message that hasn't been delivered, but the message-tracking feature is also useful for this in larger environments, and depending on the Last Error field for the queue or message in question, you may be able to figure out what your next move should be. However, the one drawback to the Queue Viewer is that unless you have a simple topology, you might not necessarily know exactly how a message reached that particular server. For that type of analysis, you need something a little more detailed (like message tracking, also known as Delivery Reports in Exchange Server 2013, which we'll cover next).

Using Message Tracking

With end-user message tracking in the Exchange Control Panel, Exchange Server 2010 introduced a new wrinkle into what used to be a purely administrative task. In a reversal, we now see the tool that was designed for end users make an appearance in the administrative management console, the Exchange Control Panel (for administrators, of course.) The conscientious administrator now has three choices for tracking messages:

Allow the end user to search for messages via the Exchange Control Panel
Track messages via the Exchange Control Panel
Track messages via the Exchange Management Shell

These options are listed in order of power and usability, so we'll start with the simplest first: end-user message tracking.

Self-service Message Tracking in the Exchange Control Panel

Before Exchange Server 2010, the only way an end user could determine the delivery status of a message was by requesting delivery receipts, but there were two drawbacks: many companies would block delivery (and read) receipts from leaving the Exchange Server organization, and many users elected to never send them at all. This left a functionality gap that the Exchange Control Panel helps fill. This option, available in Outlook Web App (OWA), allows end users to gather information about their own messages (or other people's messages if they have the permissions). This can be incredibly useful for environments with lots of tech-savvy users but would require a little investment in training, documentation, and, above all, communication. The security conscious among us need not fear: the message-tracking function in the ECP adheres to the same role-based access control regime as all the other Exchange Server components, so users couldn't use this interface to just browse their way through random users' message history. To access the self-service message-tracking component, simply log into OWA as you normally would, and then select Options. Select Settings and then select Delivery Reports in the center pane. This displays the message-tracking screen shown.

Although the title of the message-tracking pane seems to indicate that it's processing delivery reports, don't worry: Exchange Server hasn't been secretly appending delivery reports to every email your users have been sending! It's simply processing delivery information taken from the message-tracking logs (remember, message tracking is enabled by default in Exchange Server 2013).

Assuming the logs are still available, users should be able to determine information about their own messages, although as in medicine sometimes a little knowledge is a dangerous thing! Users might become so enamored of self-service message tracking that they check the status of all their messages, so any small delay could turn into more help desk calls, not fewer. You'll need to balance the needs of the community with the realistic expectations of delivery performance.

Message Tracking via the Exchange Admin Center

The message-tracking tool, also listed as Delivery Reports in Exchange Server 2013, is very different than the one you encountered in previous versions of Exchange Server. Administrators can search for messages from any sender, to any recipient, with any subject line, using wildcards and filters as necessary to focus on the critical data. However, they are using the same interface used by Exchange Server users that is accessed from the Exchange Control Panel (From Outlook Web App).

To launch message tracking from within the Exchange Control Panel, select Mail Flow in the navigation pane at the left and then choose Delivery Reports in the display panel. This launches the web-based message-tracking tool, which is the same tool an end user would use but with a few additional options. The big difference is that as an administrator you will be able to track everyone's messages and not just your own.

Once you've launched the tool, you'll be presented with what might be a bewildering array of possibilities. You can track on any of a number of fields, including the mailbox to search, the sender, and keywords that appear in the subject line.

Once you've entered all the relevant criteria, click Search to begin searching for messages. Depending on your search criteria, this process could take a significant amount of time.

The Message Tracking Results page is a little confusing when you first encounter it, but it makes sense after you've visited it a few times. Because messages pass through different stages during the mail-transfer process, you should (hopefully) see multiple entries for every message. At a bare minimum, a message should be listed three times, for the original notification to a Hub Transport server in the local site (SUBMIT), the delivery to the database on the receiving Mailbox server (DELIVER), and the ultimate delivery to the recipient (RECEIVE). If the recipient is in a different site, you'll see the delivery (SEND) of the message from one Hub Transport to another, and if there are multiple recipients, you'll probably see TRANSFER, which indicates that a message was bifurcated en route.

The message-tracking tool in the console can be useful, but it's a lot slower than building your own queries with PowerShell. After you've tracked messages a few times with the Exchange Control Panel, you'll probably be comfortable enough to forgo the GUI and just use the shell.

Message Tracking Using the Exchange Management Shell

Since the message-tracking tool in the Exchange Control Panel portion of Outlook Web App uses the Get-MessageTrackingLog cmdlet, there's little to do here but show the actual output of the cmdlet, with no input:

Get-MessageTrackingLog | Format-table EventID,Source,Sender, MessageSubject

There are a few advantages to using the shell over the Exchange Control Panel. Essentially, you get much more flexibility and granularity in your searches. You can initiate searches based on EventID, source event, or even source server. Now, if you were around before Exchange Server 2013, you might remember that these search criteria were available in previous versions in the Exchange Management Console (our previous GUI administrative console), but to have this flexibility today, you must use the Exchange Management Shell.

Now that we've gone through message tracking, you should be well equipped to determine whether a message was delivered and if not, where it stalled.

Exploring Other Tools

If you've used all the tools and techniques we've outlined to troubleshoot a mail issue and your problem isn't solved, you might be facing more than a simple mail-flow issue. If you long for some of the tools that you used in previous versions, force yourself to hone your shell technique. It's true: Microsoft has not yet updated the Routing Log Viewer or the Mail Flow Troubleshooter, both great transport troubleshooting tools from previous versions of Exchange Server; but you can still get by with the Exchange Management Shell and its powerful scripting ability.

If you've deployed transport agents in your environment, you may need to enable pipeline tracing, which essentially records every message to disk for later review. However, pipeline tracing is rather complex and is typically used only in conjunction with a Microsoft support case, so we won't cover it here-not to mention that it would deserve its own tutorial! If you're curious about what pipeline tracing entails, what it offers, and (if you're brave enough) how to enable it, have a look at this page:

http://technet.microsoft.com/en-us/library/bb125018(v=exchg.150).aspx

Note
Many of you may have noticed by now that Windows Server 2008 R2 and Windows Server 2012 servers have IPv6 enabled for their network interfaces by default. In most cases, this is not of much concern to Exchange Server 2013 servers, except for some particular scenarios. The most common that I've come across recently is the lack of IPv6 DNS records. Noticed that some ISPs now allow IPv6 traffic on their networks, yet most network administrators may not be ready for the trouble this might cause.

Here are the specifics. Some large cloud-based email companies, such as Google and Yahoo, in an effort to minimize the flood of SPAM received by their users, reject email messages sent by servers that do not have DNS records in the Reverse Lookup zones. Most organizations have created such DNS records for the IPv4 addresses, but failed to do so for the IPv6 address. Most will not feel any symptoms, until the day the Exchange Server starts to deliver email messages to @gmail.com addresses by using IPv6, and those emails are rejected.

To avoid the flood of users complaining about bounced email messages, do yourself a favor and create reverse DNS records for both IPv4 and IPv6 addresses used to deliver email messages to the Internet.