Exchange – IIS problems after upgrade (Event IDs: 10010, 3, 1016)


One of our customers has Exchange 2010 (currently SP3). When I was installing latest build (SP3 from SP4 RU 5v2), I experienced problems that server was unaccessible, Exchange web services were not able to serve clients, Powershell was not working and Forefront was crashing taking down MS Exchange Transport service. I spent several hours troubleshooting and here is the overview and solution.
Description:

Configuration:

Lets say that my server was in Windows NLB cluster having:

LB IP: 192.168.1.120 /24
 Production IP:192.168.1.100 /24
 Production IP2 192.168.1.99 /24

From the start server has been installed and configured with the following  IP´s:

LB IP: 192.168.1.120 /24
Production IP:192.168.1.100 /24

and second Production IP address has been added later as requirement for one business Exchange Web Services-related application.

From IIS point of view:

From the begining there was only one application pool with all virtual directories as shown in the following picture (NOTE binding settings).

APP_Pools_Before
Site bindings for Default application web site application pool:

bindings_default_before

After business application has been added new application pool has been added too(NOTE binding settings).

APP_Pools_After

Site Bindings for business application Application pool

Bindings_application_before

Process of upgrade:

The process of upgrade was normal and well known from MS Technet, however after reboot of server I lost connectivity to Powershell and to EWS on the server. Therefore I expected problems in IIS, but how to find it?

Errors in IIS:

IIS error

Errors in Event log:

Event ID 3 Event ID 1016 Event ID 10010

Solution:

I checked IIS:

  • Application pools up and running
  • IIS up and running
  • Server giving error 404 page cannot be displayed if I tried https://localhost/OWA url
  • I checked virtual directory paths and all needed files were in place
  • I checked bindings and voila! Bidning of localhost has been moved from Default Web Site to Application Web Site

Binding on Bussiness application web site has now localhost:

Bindings_application_after

Bindings on Default Web Site is missing localhost record:

bindings_default_after

Move Localhost binding to correct place and restart IIS!

Explanation:

There is a logic how IP addresses are set in IIS. One basic logic says, that WEB server can have as many IP addresses as needed, BUT the default one must be the one with lowest number. For example if I

have 3 IP addresses:

192.168.10.10
192.168.10.11
192.168.10.12

IP address with highest priority is 192.168.10.10 and in some circumstances localhost binding can be moved under application pool, which is listening under IP address with highest priority without notice. This is quite logical, that Powershell and OWA virtual directories stopped to work against localhost binding, because there was no virtual path inside newly created application pool for bussiness application.

Conclusion:

Plan the possibility to extend server with some more application pools and listeners already in design phase and reserve some IP addresses with higher numbers for future needs.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s