Zachary Loeber

I eat complexity and am never without a meal.

Exchange: Receive Connector Tango! – Part 2

In part 1 of this series I discussed some basic knowledge requirements to get a better grip on receive connectors in Exchange. I continue that conversation with some examples of improperly configured connectors and the issues they may cause. I finish up the discussion with a script you can use to scan your environment for such configurations.

Receive Connectors and RemoteIPRanges

When you have multiple receive connectors it can seem confusing understanding which one is chosen for a particular inbound email. The general rule of thumb is that the most specific match for a remote IP address is used to select the receive connector that should be used for an inbound email. From my experience this is the least understood aspect about receive connectors. This is also part of why overly generous RemoteIPRanges assignments can cause headaches or unexpected transport behavior. Where I’ve specifically seen problems are in three areas;

  1. Exchange coexistence
  2. Inbound spam filtering
  3. Hardware load balancing

In all cases, the issue was improperly configured receive connectors setup for internal relay purposes. Actually, let me rephrase that as the configuration wasn’t wrong, it was just not well thought out and caused unintended consequences. Here is a rather awful exchange architecture which probably doesn’t reflect any real production infrastructure. But this design diagram illustrates some of the issues I’m about to cover. In the illustration one server is hosting antispam services, all servers are hosting all exchange roles, and two of the servers are also being load balanced for SMTP. Again, no real design would look like this (obviously) but the illustration serves its purposes for this discussion.

Issue 1: Exchange Coexistence

Imagine that you have created a new Exchange server and have gotten as far as to already migrate over client access to it from your old server. Everything works well but the moment you migrate over a mailbox to the new server you find that email starts backing up from that user to users on the old server. Email just refuses to deliver and keep backing up and retrying in the queue on the 2013 server. If you look at the queues you may see the last delivery attempts complaining about authentication:

“451 5.7.3 Cannot achieve “Exchange Server authentication”.”Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts”

Couple this with a send connector which is still using the old server for outbound email and you also have outbound email delivery failure for the migrated mailbox. There may be many causes to this issue but one of the first areas I look at when exchange servers are unable to deliver email to one another (besides obvious issues like disk space and Exchange services) are the receive connectors’ defined RemoteIPRanges. Most SMTP traffic between Exchange servers runs on port 25 via some form of authentication. The default receive connectors have the appropriate authentication set and are configured to listen for all IP addresses, the broadest range of addresses you can define. So by default any exchange server is setup to appropriately forward smtp traffic to every other server. But if another range of addresses (ie. 192.168.1.0/24) that happens to include other exchange servers are configured as allowed remote IP ranges on a connector then that receive connector will be preferred over the default receive connectors. If this is setup on a relay connector that does not have all the authentication bits required to work properly you will start experiencing intra-server SMTP flow issues. The issues you experience may not even prevent mail delivery if you have multiple transport servers in the organization. Exchange transport services are pretty resilient and may even reroute traffic through other servers without you even realizing there may be underlying concerns to address.

Note: Interestingly enough, it is perfectly valid to have a relay receive connector setup with RemoteIPRanges that encompass it’s own address and not impact email transport. The moment you migrate those ranges to another server (say if you are migrating relay receive connectors from an old 2010 server to a new 2013 server) you may find that mailboxes on the old server cannot send mail to those hosted on the new server.

The take away from this is that less is more. It may be tempting to put entire ranges of IP addresses in your relay receive connector but do not do this. Also, if tasked with migrating to new exchange servers you should validate that all these RemoteIPRanges do not accidentally contain IP addresses from existing or new exchange servers.

Issue 2: Inbound Spam Filtering

Similar to the prior issue you can have unexpected consequences when you define the allowed IP addresses to include the default gateway. This is especially true on smaller deployments where the default gateway also happens to be the firewall and you have antispam software running directly on the exchange server. This is is because in bound traffic from the internet will likely be NATing from your firewall and so all SMTP traffic seems to be coming from it. You may experience mixed results when configured this way as the antispam software may still seem to function as it should. But while it may work for blocking spam, it may also be allowing anonymous relay to the outside world as well! This one is especially sneaky as you may not even realize it is happening.

Note: Incidentally, this is also one of the reasons why I recommend a cloud based antispam service as your first line of defense. Besides the queuing of email that they can provide when your servers are down, they are also typically the only external IPs which are setup in the firewall for inbound SMTP delivery to your server.

Issue 3: Hardware Load Balancing

I’ve only run across this once but it is worthy mentioning as it is one of those configurations which has multiple components that all work together to unintentionally reduce security overall. When introducing internal load balancers on your network it is pretty common to configure them in a single arm configuration where all interfaces and IP addresses reside on the same network as the devices which are to be load balanced. When deployed in this manner the load balancers essentially source NAT the traffic so that it appears to be coming from the individual load balancer IP address instead of the original source IP. While this works it can obfuscate the source IP address on the servers which are being load balanced. With client access based on HTTP protocols this can be worked around by injecting X-Referrer fields at the load balancer and configuring the IIS services to log the field. SMTP doesn’t have this luxury. Even if it did, the receive connectors wouldn’t know or care about that data as they only see the source IP address that initiates the connection. How this affects security is easy, if you load balance SMTP traffic to Exchange in this manner and expect to restrict relay IP addresses by using the RemoteIPRanges it simply will not work. What may happen then is that someone will configure the IPs of the load balancers on the receive connectors instead. What then ends up being configured is a wide open relay on your internal network for all traffic bound to the load balancer VIP address. Depending on the relay receive connector rights configured on the connectors (mentioned in part 1 of this article) it may also enable anyone to spoof anyone else in the organization.

Checking for these Issues

All of these configuration concerns are similar enough to be in the same discussion but different enough so that there is no one easy test to run for checking if they are in your environment.

Issues 1 & 2: Exchange Coex and Spam Filtering

In smaller environments simply going in and looking at the RemoteIPRanges which are setup on the receive connectors manually should be an easy enough task to accomplish. In larger environments it can be a bit more difficult to detect which servers have incorrectly configured relay receive connectors. As is my style, I’ve hacked together some powershell to help us automate the process of detecting possible relay receive connectors and which ones may have improperly configured RemoteIPRanges. This script is broken down into several functions. A few of the helper functions are IP address validations which may be generally useful to have in your library. The others are far more specialized for Exchange. I actually don’t even validate Exchange plugins are installed as it is assumed that these will be run directly in an exchange management console. Anyway, here are the helper functions.

The most important function in the script is the one for attempting to intelligently guess the relay receive connectors in the environment. What makes this task a bit difficult is that there are default receive connectors that have anonymous permissions similar to what a relay connector might have. To help eliminate these from further calculations down the line I validate if a receive connector is ‘Wide Open’, that is if it listens indiscriminately for any IP address. So in this script a relay receive connector is determined to be a candidate by the following rules;

  1. Are there anonymous permissions configured on the connector?
  2. Are there specific remote IP ranges which are setup in the RemoteIPRanges property (not just all IP addresses)?
  3. Is the connector listening on port 25?

We then get a list of unique transport servers in the environment and remotely query them for IP address information. Using all of this info we can then start validating if Exchange server IPs are present in the ranges defined in your anonymous relay connectors. While we are at it we can use some of the same logic to see if local gateways are in the relay connectors accepted range definitions as well.

With the functions that have been created we can then go ahead and start creating a quick console reports to see if some of the remote exchange server IPs or their own default gateway IP addresses are able to be found in the RemoteIPRanges of any of the (possible) anonymous relay internal receive connectors.

Issue 3: Hardware Load Balancing

Ideally you will already know if you are load balancing exchange servers behind an HLB and can access the load balancers to validate if SMTP is being load balanced. This is because there is no easy way to automatically detect for this scenario with a script. You can manually test for it with good ol’ telnet though. I probably should find a powershell way to do this but it is so simple to do via telnet that I’ve not been able to justify spending the time on it. Here is an example of testing a server for email relay. Just swap out the recipient to test for relay (or swap out the from address to test for spoofing).

Note: Another possibility for testing a server to see if it is improperly being load balanced would be to use something like log parser against the smtp connection logs. You can try to guess if smtp is being load balanced behind a VIP by looking for many connections from the same IP addresses on the same subnet as the server.

Conclusion

This discussion has gone much longer than I originally anticipated. I hope people get some value from this. I always welcome comments, corrections, and script ideas. As usual, I’ve uploaded the script for this article to the technet gallery for your convenience.