When performing a bandwidth assessment with your pilot users, you can use our guide; Office performance tuning using baselines and performance history. Plan for high availability requirements Create a plan for high availability to meet your needs and incorporate this into your updated network topology diagram. Plan for network security requirements Create a plan to meet your network security requirements and incorporate this into your updated network topology diagram.
Design outbound service connectivity ExpressRoute for Office has outbound network requirements that may be unfamiliar. Specifically, the IP addresses that represent your users and networks to Office and act as the source endpoints for outbound network connections to Microsoft must follow specific requirements outlined below.
The endpoints must be public IP addresses, that are registered to your company or to carrier providing ExpressRoute connectivity to you. The endpoints must not be advertised to the Internet with the same or more preferred routing metric.
The endpoints must not be used for connectivity to Microsoft services that are not configured over ExpressRoute. This occurs when requests to Microsoft services are routed over ExpressRoute, but responses are routed back across the internet, or vice versa, and the responses are dropped by stateful network devices such as firewalls. The most common method you can use to meet the above requirements is to use source NAT, either implemented as a part of your network or provided by your ExpressRoute carrier.
Source NAT allows you to abstract the details and private IP addressing of your internet network from ExpressRoute and; coupled with proper IP route advertisements, provide an easy mechanism to ensure path symmetry. Add the changes for the outbound connectivity to the network topology diagram.
Design inbound service connectivity The majority of enterprise Office deployments assume some form of inbound connectivity from Office to on-premises services, such as for Exchange, SharePoint, and Skype for Business hybrid scenarios, mailbox migrations, and authentication using ADFS infrastructure. When ExpressRoute you enable an additional routing path between your on-premises network and Microsoft for outbound connectivity, these inbound connections may inadvertently be impacted by asymmetric routing, even if you intend to have those flows continue to use the Internet.
A few precautions described below are recommended to ensure there is no impact to Internet based inbound flows from Office to on-premises systems. If the incoming connections are allowed onto a network segment with routing visibility into ExpressRoute without source NAT, requests originating from Office will enter from the internet, but the response going back to Office will prefer the ExpressRoute network path back to the Microsoft network, causing asymmetric routing.
You may consider one of the following implementation patterns to satisfy this requirement: Perform source NAT before requests are routed into your internal network using networking equipment such as firewalls or load balancers on the path from the Internet to your on-premises systems.
Ensure that ExpressRoute routes are not propagated to the network segments where inbound services, such as front end servers or reverse proxy systems, handling Internet connections reside. Explicitly accounting for these scenarios in your network and keeping all inbound network traffic flows over the Internet helps to minimize deployment and operational risk of asymmetric routing.
There may be cases where you may choose to direct some inbound flows over ExpressRoute connections. For these scenarios, take the following additional considerations into account. Office can only target on-premises endpoints that use public IPs. This means that even if the on-premises inbound endpoint is only exposed to Office over ExpressRoute, it still needs to have public IP associated with it. In order to receive inbound network connections over ExpressRoute, the public IP subnets for these endpoints must to be advertised to Microsoft over ExpressRoute.
Carefully evaluate these inbound network traffic flows to ensure that proper security and network controls are applied to them in accordance with your company security and network policies. Once your on-premises inbound endpoints are advertised to Microsoft over ExpressRoute, ExpressRoute will effectively become the preferred routing path to those endpoints for all Microsoft services, including Office This means that those endpoint subnets must only be used for communications with Office services and no other services on the Microsoft network.
Otherwise, your design will cause asymmetric routing where inbound connections from other Microsoft services prefer to route inbound over ExpressRoute, while the return path will use the Internet. This may mean advertising subnets for those endpoints through multiple ExpressRoute circuits. We recommend applying source NAT for all inbound network traffic flows entering your network through ExpressRoute, especially when these flows cross stateful network devices such as firewalls.
Some on-premises services, such as ADFS proxy or Exchange autodiscover, may receive inbound requests from both Office services and users from the Internet. Allowing inbound user connections from the internet to those on-premises endpoints, while forcing Office connections to use ExpressRoute, represents significant routing complexity. For the vast majority of customers implementing such complex scenarios over ExpressRoute is not recommended due to operational considerations.
This additional overhead includes, managing risks of asymmetric routing and will require you to carefully manage routing advertisements and policies across multiple dimensions. Update your network topology plan to show how you would avoid asymmetric routes You want to avoid asymmetric routing to ensure people in your organization can seamlessly use Office as well as other important services on the internet.
There are two common configurations customers have that cause asymmetric routing. In this diagram, all servers that receive inbound requests, such as ADFS or on-premises hybrid servers are in the New Jersey data center and are advertised to the internet.
While the perimeter network is secure, there is no Source NAT available for incoming requests. The servers in the New Jersey data center are able to see both internet and ExpressRoute routes. We also have suggestions on how to fix them. The inbound request from Office retrieves the IP address of the on-premises endpoint from public DNS and sends the request to your perimeter network. In this faulty configuration, there is no Source NAT configured or available at the perimeter network where the traffic is sent resulting in the actual source IP address being used as the return destination.
The server on your network routes the return traffic to Office through any available ExpressRoute network connection. The result is an Asymmetric path for that flow to Office , resulting in a broken connection.
This time Source NAT is available. The response from the server routes back toward the IP associated with the Source NAT instead of the original IP address, resulting in the response returning along the same network path. Route Scoping Alternatively, you can choose to not allow the ExpressRoute BGP prefixes to be advertised, removing the alternate network path for those computers.
This time the prefixes advertised from Microsoft over the ExpressRoute circuit are not available to the New Jersey data center. The response from the server routes back toward the IP associated with the original IP address over the only route available, resulting in the response returning along the same network path. The inbound request from Office retrieves the IP address from DNS and sends the request to your perimeter network.
The computer on your network routes the return traffic to Office through any available ExpressRoute network connection. The result is an Asymmetric connection to Office From the on-premises network and WAN routing, to the perimeter devices, to the connectivity path; ExpressRoute or the internet, and on to the connection to the online endpoint.
It helps to do this paper walk through of routes with a second person. Remember that ExpressRoute will always provide a more scoped route to Microsoft server IP addresses giving it lower route cost than an Internet default route.
Read our guide on managing Office endpoints for example PAC files. The endpoints change frequently, as often as weekly. Pay close attention to the Effective Date in the RSS feed where the changes are announced and a record is kept of all past changes, IP addresses that are announced may not be advertised, or removed from advertisement, until the effective date is reached.
Build your deployment and testing procedures Your implementation plan should include both testing and rollback planning. The following are some high level principles your plan should consider. Stage the network segment and user service onboarding to minimize disruption.
Plan for testing routes with traceroute and TCP connect from a separate internet connected host. Preferably, testing of inbound and outbound services should be done on an isolated test network with a test Office tenant.
Alternatively, testing can be performed on a production network if the customer is not yet using Office or is in pilot. Alternatively, testing can be performed during a production outage that is set aside for test and monitoring only. Alternatively, testing can be done by checking routes for each service on each layer 3 router node.
This fall back should only be used if no other testing is possible since a lack of physical testing introduces risk. Build your deployment procedures Your deployment procedures should roll out to small groups of people in stages to allow for testing before deploying to larger groups of people. The following are several ways to stage the deployment of ExpressRoute. Set up ExpressRoute with Microsoft peering and have the route advertisements forwarded to a single host only for staged testing purposes.
Advertise routes to the ExpressRoute network to a single network segment at first and expand route advertisements by network segment or region. If deploying Office for the first time, use the ExpressRoute network deployment as a pilot for a small number of people. If using proxy servers, you can alternatively configure a test PAC file to direct a small number of people to ExpressRoute with testing and feedback before adding more.
Your implementation plan should list each of the deployment procedures that must be taken or commands that need to be used to deploy the networking configuration. When the network outage time arrives all of the changes being made should be from the written deployment plan which was written in advance and peer reviewed. See our guidance on the technical configuration of ExpressRoute.
After your ExpressRoute deployment is complete the procedures in the test plan should be executed. Results for each procedure should be logged. You must include procedures for rolling back to the original production environment in the event the test plan results indicate the implementation was not successful. Build your test procedures Your testing procedures should include tests for each outbound and inbound network service for Office both that will be using ExpressRoute and ones that will not.
The procedures should include testing from each unique network location including users who are not on-premises in the corporate LAN. Some examples of test activities include the following. Ping from your on-premises router to your network operator router.