RTF content archiving problem when using Mailstore against Exchange 2010 SPx – ErrorInternalServerTransientError

I have experienced problem in one of my customer´s Exchange environment after utilization of Mailstore archiving software. Mailstore is EWS and client based archiving solution for Exchange. All best practice configuration steps can be found here: http://en.help.mailstore.com/MailStore_Help

Environment:

  • Virtualized Exchange 2010 SP3 RUx environment with 2 node DAG, multirole servers. Both running on ESX 5.1. No Firewall and router between production Exchange and Mailstore virtual servers.

Symptoms:

  • RTF content messages cannot be archived using Mailstore via EWS
  • RTF messages can be easily simulated as new meeting request containing inline picture of any size. Meetings should not be answered to have error visible in 100 percent of cases
  • Error message in Mailstore job log as follows
08:36:58.874 [18] INFO Processing message: 23.1.2014 7:42:45 UTC 'FW: Problém s archivací meetingů', UID 1: @mail.domain.cz, UID 2: 
08:36:58.890 [18] INFO Retrieving message...
08:36:58.890 [18] INFO Sending EWS Request (GetMimeContent)
08:36:59.561 [18] INFO Sending EWS Request (GetMimeContent)
08:37:00.403 [18] INFO Sending EWS Request (GetMimeContent)
08:37:01.464 [18] INFO Sending EWS Request (GetMimeContent)
08:37:02.727 [18] INFO Sending EWS Request (GetMimeContent)
08:37:04.194 [18] INFO Sending EWS Request (GetMimeContent)
08:37:05.879 [18] INFO Sending EWS Request (GetMimeContent)
08:37:07.751 [18] INFO Sending EWS Request (GetMimeContent)
08:37:09.825 [18] INFO Sending EWS Request (GetMimeContent)
08:37:12.072 [18] INFO Sending EWS Request (GetMimeContent)
08:37:14.521 [18] INFO Sending EWS Request (GetMimeContent)
08:37:17.173 [18] INFO Sending EWS Request (GetMimeContent)
08:37:20.012 [18] INFO Sending EWS Request (GetMimeContent)
08:37:23.070 [18] INFO Sending EWS Request (GetMimeContent)
08:37:26.330 [18] INFO Sending EWS Request (GetMimeContent)
08:37:29.793 [18] INFO Sending EWS Request (GetMimeContent)
08:37:30.230 [18] EXCEPTION MailboxImportWorker:ProcessMailboxMessageWrapper
: Microsoft Exchange Server nedokázal dokončit úlohu. Detaily: An internal server error occurred. Try again later. EWS Error Kód: ErrorInternalServerTransientError.
  • Moving node to other ESX cluster or moving active database to another node solved error instantly, but after switch back error appeared again
  • User-generated load was also partly the problem

Solution:

We have tried everything from re-creation of throttling policies, moving databases between nodes, updates to latest RU and Mailstore versions, Disabling TCP chimney, RSS and AutoTuning features, re-creation of Exchange databases, re-creation of Mailstore database and many many others.

What has finally helped was to re-create EWS virtual directory and restart IIS:

Get-WebServicesVirtualDirectory SERVER\ID | Remove-WebServicesVirtualDirectory
New-WebServicesVirtualDirectory
Get-WebServicesVirtualDirectory SERVER\ID | Set-WebServicesVirtualDirectory -InternalURL <IURL> -ExternalURL <EURL>

I suspect 2 things. 1 is problematic IIS 7 metabase or utilization of CGI (Common Gateway Interface –http://technet.microsoft.com/en-us/library/cc753077(v=ws.10).aspx ) on EWS virtual directory. Uninstallation of CGI did not solve the problem. Problem has been solved by re-cration of EWS virtual directory on affected DAG node after uninstallation of CGI.

Advertisements

Exchange 2013 / Exchange 2010, Windows Server 2012 – SChannel Event ID:36888 (1203) – TLS/SSL error – The root cause

I have problems in some environments, where these SChannel errors are generated. Well. It took me several days to find reasonable “why” it is logged.

Problem:

The event ID from the picture can be seen from time to time:

EventID-Error

Solution:

Based on several articles I have read and some discussions. First you have to make sure, that the process causing this error is LSASS.exe, which is by the way local security authentication server (authenticating users to winlogon service, using authentication such as msgina.dll and so on). To make sure it is LSASS.EXE. Open Event ID and check the Event ID details, Click on Details tab -> Expand System while friendly view is selected. Check Process ID.

EventID_Details

Then use powershell and run:

Get-Process | select name,id | sort id

Result should give you the name of the processes. It will be lsass.exe.

Why:

Reason is simple. Not standard or corrupted behavior of web browsers or users. The problem behind SChannel and Exchange 2012 is, that sometimes users use HTTP protocol, but on port 443, which expects certificates exchange rather than GET command.

How to test:

Option 1#:

Test is easy. For example you can input URL to your browser address bar, which is obviously wrong and see the results: HTTP://MAIL.DOMAIN.LOCAL:443/OWA – It says to use HTTP protocol (not HTTPS) on the 443 port and it generates errors immediately.

Option 2#:

Run Telnet and test command:

Telnet localhost 443 (to connect to HTTPS)

In Telnet window:

Get /index.htm (on HTTPS SSL must be established first so it will generate errors immediately. Result will not be seen in telnet window)

What is the solution?

Solution #1:

Some IT guys recommend to disable SCHannel logging to get rid of these events, but I cannot recommend that. To be honest. It is better to see, that somebody is trying to connect using HTTP on HTTPS port, because this might be some attempt to DoS attack or info, that users don´t know how to type OWA URL correctly. Shortly it is better to know something is wrong than disable logging.

Solution #2:

I suspect wrong redirect configuration for the websites from HTTP to HTTPS. I would check IIS if redirect is set correctly. For those having this issue without redirect I would suspect problem in web browser area.

Links:

To test SSL via command line:

http://www.bearfruit.org/2008/04/17/telnet-for-testing-ssl-https-websites/

LSASS description:

http://www.neuber.com/taskmanager/process/lsass.exe.html

How to quickly clean mailbox in Exchange 2010/2013

I had troubles and a lot of mess in my test mailbox and didn´t have time to cleanup, so here is, what I did. Basically I used a method, which is also used, when there are problems in production and server / database goes down and you must use Dial tone restore.

  • Gather mailbox database
Get-Mailbox <identity> | select MailboxDatabase
  • Rehome mailbox (set different database to mailbox)
Get-Mailbox <identity> | Set-Mailbox -Database <DB identity>
Get-mailbox x9xxxx | Set-Mailbox -Database MDB12
Confirm Rehoming mailbox "domain.local/Persons/Administrators/test/CZ/X9XXX" to database "MDB12". This operation will only modify the mailbox's Active Directory configuration. Be aware that the current mailbox content will become inaccessible to the user. [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):

Use Clean-MailboxDatabase on old database to see the mailbox in disconnected mailboxes.

Get-MailboxDatabase <old MDB identity> | Clean-MailboxDatabase
  • Your old data will be removed according your Exchange configuration or you can force deletion by command Remove-StoreMailbox <your old data mailbox identity> as well described here:

http://technet.microsoft.com/en-us/library/gg181092(v=exchg.141).aspx

  • Your mailbox is clean.

Exchange 2010 SP upgrade failed (0x80070003)

Let me provide you fresh experience with Exchange 2010 SP3 upgrade.

In the first place, thank you Zbynek, because final solution was his idea!

Problem

Exchange 2010 SP3 upgrade unexpectedly failed for 2 servers from 9. Those servers had separated Exchange roles. So the following error occurred for MBX as well as HUB role.

[10/05/2013 18:58:41.0984] [2] Saving object "EXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" and state "New".
[10/05/2013 18:58:42.0015] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0481] [2] Searching objects "DEXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" under the root "$null".
[10/05/2013 18:58:43.0497] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0497] [2] Ending processing new-PowerShellVirtualDirectory
[10/05/2013 18:58:43.0497] [1] The following 1 error(s) occurred during task execution:
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: A failure occurred while trying to update metabase properties.
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: Microsoft.Exchange.Data.Common.LocalizedException: A failure occurred while trying to update metabase properties. ---> System.Runtime.InteropServices.COMException (0x80070003): The system cannot find the path specified.

As can be seen it was IIS related problem (a failure occurred while trying to update metabase properties) especially with PowerShellVirtualDirectory.

Root Cause

Only suspicion:

  • firewall or application (e.g. an anti-virus) was cutting the connection
  • an application was locking the IIS metabase (e.g. a backup solution)
  • not sufficient permissions

Solution

This solution is intended for separated (CAS, MBX, HUB) as well as multi roles.

1.  Remove corrupted PowerShellVirtualDirectory:

* remove all virtual directories whether CAS role

Get-PowerShellVirtualDirectory EXMBX02\* | Remove-PowerShellVirtualDirectory

2.  Recover Exchange server:

Setup /m:RecoverServer

Appendix

Get remote server names from receive connector

I was asked to make a list of all remote servers (including IP addresses) which are able to use a receive connector in Exchange 2010.

2013-08-21 00_59_27

The remote IP addreses of the receive connector can be found under RemoteIPRanges property of Get-ReceiveConnector cmdlet. Those addreses can be devided based on RangeFormat declaration:

  • SingleAddress (10.10.10.5)
  • CIDR (10.10.10.5/24)
  • LoHi (10.10.10.5-10.10.10.10)

Theoretically if we need to get all IP addresses, we have to know also IP addresses in particular ranges and then we can resolve those addreses in DNS.

Here are two functions. The first one (New-IPRange) is created by Dr.Tobias Weltner and ensures us to find all IP addresses (also from CIDR or LoHi ranges). The second one is my helper (Get-ReceiveConnectorRemoteIPName) which goes through RemoteIPRanges, calls the first function and resolves IP address by System.Net.Dns .NET class. The processing time depends on amount of IP addreses so be careful about your ranges!

function New-IPRange ($start, $end) {
 # created by Dr. Tobias Weltner, MVP PowerShell
 $ip1 = ([System.Net.IPAddress]$start).GetAddressBytes()
 [Array]::Reverse($ip1)
 $ip1 = ([System.Net.IPAddress]($ip1 -join '.')).Address
 $ip2 = ([System.Net.IPAddress]$end).GetAddressBytes()
 [Array]::Reverse($ip2)
 $ip2 = ([System.Net.IPAddress]($ip2 -join '.')).Address
 for ($x=$ip1; $x -le $ip2; $x++) {
 $ip = ([System.Net.IPAddress]$x).GetAddressBytes()
 [Array]::Reverse($ip)
 $ip -join '.'
 }
}

function Get-ReceiveConnectorRemoteIPName ($Identity) {
 $Connector = Get-ReceiveConnector -identity $Identity | select Identity,RemoteIPRanges
 if($Connector -ne $null){
 $IPs = $Connector.RemoteIPRanges | % { New-IPRange $_.LowerBound $_.UpperBound }
 foreach($IP in $IPs){
 $Output = New-Object PSObject
 $IPName = ([Net.DNS]::GetHostEntry("$ip")).HostName
 if($IPName -eq $IP){$IPName="unresolvable"}
 $output | add-member -Type NoteProperty -name “ReceiveConnector” -value $Connector.Identity
 $output | add-member -Type NoteProperty -name “RemoteIp” -value $IP
 $output | add-member -Type NoteProperty -name “RemoteName” -value $IPName
 $output
 }
 }
}

How to use it? Only paste both functions into EMS, that’s it.

2013-08-21 00_44_51

Now you are ready to use both functions especially Get-ReceiveConnectorRemoteIPName:

[PS] C:\>Get-ReceiveConnectorRemoteIPName "EX2010S01\Application Relay"

ReceiveConnector             RemoteIp    RemoteName
----------------             --------    ----------
EX2010S01\Application Relay  10.10.1.16  appolo.ficility.intra
EX2010S01\Application Relay  10.10.2.2   helt01.ficility.intra
EX2010S01\Application Relay  10.10.2.3   unresolvable
EX2010S01\Application Relay  10.10.2.4   unresolvable
EX2010S01\Application Relay  10.10.1.25  kepro.ficility.intra

2013-08-21 01_54_48

Feel free to use Get-ReceiveConnectorRemoteIPName cmdlet in the following scenarios:

[PS] C:\> Get-ReceiveConnectorRemoteIPName "EX2010S01\Application Relay"| Export-Csv -Path "C:\ReceiveConnectorRemoteIPName.csv"

[PS] C:\> $connectors = Get-ReceiveConnector | ? { $_.identity -like "*relay*" }
[PS] C:\> $connectors| % { Get-ReceiveConnectorRemoteIPName $_.identity }

Exchange – One option to restore data from lagged database copy

Recover data from lagged copy:

1. Gather info where user resides

Usually we need to know in which database user resides

2. Check if mailbox is still in disconnected mailboxes

Get-MailboxDatabase mdb13 | get-mailboxstatistics | where {$_.disconnectdate -ne $null}

DisplayName               ItemCount    StorageLimitStatus                                                 LastLogonTime
-----------               ---------    ------------------                                                 -------------
a1			  1962                 BelowLimit                                           5/7/2013 4:01:41 PM
S 		          2075                 BelowLimit                                          6/19/2013 9:26:52 AM
Hän		          185                  BelowLimit                                          4/30/2013 9:19:26 AM

3. Mailbox is not in disconnected state

If mailbox is not in disconnected mailboxes anymore, we have another 14 days before lagged copy disconnected date expires

4. Suspend lagged copy

Suspend lagged copy by command:

Get-MailboxDatabase mdb13 | Get-MailboxDatabaseCopyStatus

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
----                                          ------          --------- ----------- --------------------   ------------
MDB13\SRVMBX1                          		Mounted         0         0                                  Healthy
MDB13\SRVMBX2                          		Healthy         0         2           7/9/2013 11:33:38 AM   Healthy
MDB13\SRVMBX3                         		Healthy         0         2           7/9/2013 11:33:38 AM   Healthy
MDB13\SRVPF1                           		Healthy         0         110355      7/9/2013 11:33:38 AM   Healthy

Suspend-MailboxDatabaseCopy MDB13\SRVPF1

Confirm
Are you sure you want to perform this action?
Suspending mailbox database copy "MDB13" on server "SRVPF1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y

5. Copy lagged database to different location

To perform non destructive recovery we need to create additional copy of lagged database. This database will be restored to particular point in time.

6. Check if database is in clean shutdown

Dump headers of database by command:

eseutil /mh .\MDB13.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.03
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
         Database: .\MDB13.edb

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0xf3fb4807
  Actual Checksum: 0xf3fb4807

Fields:
        File Type: Database
         Checksum: 0xf3fb4807
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,17
 Engine ulVersion: 0x620,17
Created ulVersion: 0x620,17
     DB Signature: Create time:01/24/2012 02:52:12 Rand:391137630 Computer:
         cbDbPage: 32768
           dbtime: 3824249608 (0xe3f16b08)
            State: Dirty Shutdown

7. Determine PIT backup time and move newer logs elsewhere

In my example I want DB to be recovered to 29.6.2013. Be careful! You need EDB file 🙂

PIT

8. Reply logs to database to specified point in time:

Newer logs than specified PIT should be moved elsewhere or deleted (better after recovery process has been done)
Chk file should be removed (checkpoint) to reply all logs present in directory.
The following command will reply logs till PIT to database

Eseutil /r eXX /a

Replay_logs_progress

9. Put DB to clean shutdown

If the database is still in Dirty shutdown mode, we need to run integrity check and hard repair the database

eseutil /p .\MDB13.edb /g

DB repaired

10. Check if DB is in clean shutdown after repair

eseutil /mh .\MDB13.edb

Clean Shutdown

11. Delete all log files since those are not needed anymore

12. Create recovery database

New-MailboxDatabase -Recovery -Name RDB_13 -Server SRVPF1 -EdbFilePath e:\lagged_mdb13\mdb13
.edb -LogFolderPath e:\lagged_mdb13
WARNING: Recovery database 'RDB_13' was created using existing file e:\lagged_mdb13\mdb13.edb. The database must be
brought into a clean shutdown state before it can be mounted.

Name                           Server          Recovery        ReplicationType
----                           ------          --------        ---------------
RDB_13                         SRVPF1   True            None

13. Mount database

Mount database by issuing command:

Mount-Database RDB_13

14. Gather data about mailbox, you want to restore

Display name or StoreMailbox guid. For example use this command:

get-mailboxdatabase rdb_13 | Get-MailboxStatistics | where {$_.Displayname -like "Niitty*"}

mailbox_to_recover

15. Restore mailbox

To restore mailbox use the following command:

New-MailboxRestoreRequest -SourceDatabase RDB_13 -SourceStoreMailbox "Surname, name" -TargetMailbox alias -AllowLegacyDNMismatch

16. Check results

Get-MailboxRestoreRequest "MailboxRestore"

status

Exchange 2010 – DAG – Mapi network issue (MapiAccessEnabled, IgnoreNetwork)

One of our customers has ExRAAS ( Exchange health and remediation check service) every year to audit their environment for health, performance and MS best practices implementation. ExRAAS tools are developed every year and this years tool discovered very interesting issue about DAG networks.

Description:

Our customers DAG has 3 networks:

  • Production – meant to be client network, where only client traffic is enabled, replication traffic is disabled
  • Replication – not routable to MAPI network – custom 5Gbit bandwidth only for log replication
  • Backup – only for VSS backups, no MAPI nor replication traffic should flow there

Problem:

By design DAG is set, that Backup network should be ignored, however if I give Get-DatabaseAvailabilityGroupNetwork command, I can see MapiAccessEnabled parameter in $True, even though this network doesn´t have Clients for Windows Networks feature enabled and according to MS it is not supported network for clients. The magic starts when I set IgnoreNetwork to $false. Right after the change MapiAccessEnabled parameter is in correct value.

Get-DatabaseAvailabilityGroupNetwork DAG1\BACKUP | Set-DatabaseAvailabilityGroupNetwork -IgnoreNetwork $false
Get-DatabaseAvailabilityGroupNetwork | fl

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : BACKUP
Description        : VSS BACKUP Backup subnet - Ignored
Subnets            : {{172.24.188.0/24,Up}, {172.29.99.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,172.24.188.108}, {DC1MBX2,Up,172.24.188.110}, {DC1MBX3,Up,172.24
                     .188.112}, {DC1PF1,Up,172.24.188.104}, {DC2MBX1,Up,172.29.99.109}, {DC2MBX2,U
                     p,172.29.99.111}, {DC2MBX3,Up,172.29.99.113}, {DC2PF1,Up,172.29.99.105}}
MapiAccessEnabled  : False
ReplicationEnabled : False
IgnoreNetwork      : False
Identity           : DAG1\BACKUP
IsValid            : True

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : MAPI
Description        : Production and possible replication
Subnets            : {{192.168.0.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,192.168.0.108}, {DC1MBX2,Up,192.168.0.110}, {DC1MBX3,Up,192.168
                     .0.112}, {DC1PF1,Up,192.168.0.104}, {DC2MBX1,Up,192.168.0.109}, {DC2MBX2,
                     Up,192.168.0.111}, {DC2MBX3,Up,192.168.0.113}, {DC2PF1,Up,192.168.0.105}}
MapiAccessEnabled  : True
ReplicationEnabled : False
IgnoreNetwork      : False
Identity           : DAG1\MAPI
IsValid            : True

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : REPLICATION
Description        : Only replication
Subnets            : {{10.146.231.0/27,Up}}
Interfaces         : {{DC1MBX1,Up,10.146.231.24}, {DC1MBX2,Up,10.146.231.26}, {DC1MBX3,Up,10.146.2
                     31.28}, {DC1PF1,Up,10.146.231.20}, {DC2MBX1,Up,10.146.231.25}, {DC2MBX2,Up,10
                     .147.231.27}, {DC2MBX3,Up,10.146.231.29}, {DC2PF1,Up,10.146.231.21}}
MapiAccessEnabled  : False
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAG1\REPLICATION
IsValid            : True

When I change the Ignorenetwork back to $true, MapiAccessEnabled is set to $True as well.

Get-DatabaseAvailabilityGroupNetwork DAG1\BACKUP | Set-DatabaseAvailabilityGroupNetwork -IgnoreNetwork $true
Get-DatabaseAvailabilityGroupNetwork | fl

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : BACKUP
Description        : VSS BACKUP Backup subnet - Ignored
Subnets            : {{172.24.188.0/24,Up}, {172.29.99.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,172.24.188.108}, {DC1MBX2,Up,172.24.188.110}, {DC1MBX3,Up,172.24
                     .188.112}, {DC1PF1,Up,172.24.188.104}, {DC2MBX1,Up,172.29.99.109}, {DC2MBX2,U
                     p,172.29.99.111}, {DC2MBX3,Up,172.29.99.113}, {DC2PF1,Up,172.29.99.105}}
MapiAccessEnabled  : True
ReplicationEnabled : False
IgnoreNetwork      : True
Identity           : DAG1\BACKUP
IsValid            : True

Conclusion:

This lead to errors in ExRAAS report and to question what is the right way. How should I behave to the network configuration? Better way is to set IgnorenNetwork parameter to $True and just ignore MapiAccessEnabled in $True. This article will be updated after I get info from MS for the resolution. It is also worth to mention, that last best practice says, that compression and encryption should be ENABLED on DAG replication network!

Links:

http://blogs.technet.com/b/schadinio/archive/2010/12/08/exchange-2010-mailbox-dag-based-practice-network-configurations.aspx