Windows server network interface priority, DNS and Exchange “554 5.4.4”

My friend´s Exchange 2013 implementation was experiencing problem with e-mail delivery. E-mails were returned from inside as well as outside of organization with NDR pointing to DNS failure.

I was investigating this issue, but not all e-mails were lost. Only some of those. I knew, that there was a change recently, that new network interface for VPN connection has been added to Exchange server.

Symptoms:

DNS for next hop is not resolvable returning DNS error

“554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain”.

Investigation:

I started to investigate what could cause the problem. It was DNS error, so I started to use NSLOOKUP tool and tried to resolve DNS name for next hop domain. The answer was different, then I thought it would be. I have received Time Out answer from VPN DNS server, even there is another DNS server set in production environment. I realized from my previous Exchange 2010 projects, that priority of network interfaces is added from Highest to lowest based on sequence interfaces are added to system. So for example if you configure Production network interface before Replication, Replication interface will get higher priority and if you use DNS servers in your replication network (for whatever reason), wrong DNS server will answer to your request and this might lead to unexpected failures.

Now the naughty stuff:

If you use IPv6, you are still OK, since IPv6 DNS servers will answer even though Replication interface has higher priority then production one.

If you disable IPv6 on your interfaces, you have troubles. Once interface priority is changed to incorrect order, you will not be albe to at least resolve DNS queries correctly. Incorrect priority might also lead to packet loss.

Solution:

Very simple:

  • Right click network icon in notification area (right low corner) of your server and click Open Network And Sharing Center
  • Click on Change Adapter Settings
  • Press ALT key and select Advanced -> Advanced Settings
  •  In the Advanced Settings window select interface with incorrectly set priority and press arrows to move interface to correct place

Results before priority change

priority_before

priority_before_cmd

Results after priority change

priority_after

priority_after_cmd

Exchange 2010 – All access rights from all mailbox folders

Let’s try this request:22-03-2013 19-42-05
Get all access rights for all mailbox folders.

We should use cmdlet Get-MailboxFolderPermission which requires declaration like alias:\Inbox\folder. So we need only another cmdlet Get-MailboxFolderStatistics and convert slash characters in its folderpath property.

PS] C:\>$folders = Get-MailboxFolderStatistics filip | %{$_.folderpath} | % {$_.replace("/","\")}

[PS] C:\>$folders[7..11]
\folder 0
\folder 0\folder ? test
\folder 0\folder 1
\folder 0\folder 1\ffolder2
\folder 0\folder 2

22-03-2013 20-51-29We should have all folder paths from the mailbox (filip = alias) in $folders variable and be able to use it for Get-MailboxFolderPermission. We can expect few errors for “system” folders such as Top of Information Store, Recoverable Items, Deletions, Versions, Purges but also for folders which contains the slash character in name because of converting.

[PS] C:\>$folders = $folders | %{ Get-MailboxFolderPermission "filip:$_" | select foldername,user,accessrights}`
 | ? { $_.AccessRights -notlike "None" -and $_.User -notlike "*filip*" }

The operation couldn't be performed because 'filip:\Top of Information Store' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : 28783069,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

The operation couldn't be performed because 'filip:\folder 0\folder ? test' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : 18857875,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

The operation couldn't be performed because 'filip:\Recoverable Items' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : ABE546E,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

The operation couldn't be performed because 'filip:\Deletions' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : 8321BED7,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

The operation couldn't be performed because 'filip:\Purges' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : 65847762,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

The operation couldn't be performed because 'filip:\Versions' couldn't be found.
 + CategoryInfo : NotSpecified: (:) [Get-MailboxFolderPermission], ManagementObjectNotFoundException
 + FullyQualifiedErrorId : 8A11CF7B,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolderPermission

Valid objects are in the folders variable:

22-03-2013 20-49-14

Shortly:

  1. $folders = Get-MailboxFolderStatistics alias | % {$_.folderpath} | % {$_.replace(“/”,”\”)}
  2. $folderPermissions = $folders | %{ Get-MailboxFolderPermission “alias:$_” | select foldername,user,accessrights}
  3. $folderPermissions

Exchange – Single public IP address and easy SMTP High availability

This article is about use high availability of very simple kind, when you have single public IP address without possibility to forward traffic to more than one hosts. In my case I have single public IP address in my LAB, I use Steve Goodman´s Exchange 2010 HAProxy (http://www.stevieg.org/e2010haproxy/), which is not compiled for SMTP traffic. I have 2 node Exchange 2013 DAG with CAS/MAILBOX roles on each node.
This configuration simply means, that I cannot use Win NLB, because DAG cannot operate on the same machine as Win NLB does.

Previously

I used single node to route SMTP traffic to and in case of node failure SMTP traffic was held on gateway till the node came up.

Current setup

Well. If I think about DAG itself, it is high available cluster solution for Exchange 201x. For me there are 2 aspects good for SMTP high availability:

  • DAG has its own IP address
  • IP address is assigned to node running Active Manager and quorum

Yes, these 2 things are essential. If I route SMTP traffic to DAG IP, I will have it allways online and available, because if Active Manager is not online and accessible on single DAG node, DAG is in serious problems and most probably some or all databases will not work.

What to do to make it work?

  • On each DAG node create new internet receive connector bound to Frontend Transport Service, which is stateless SMTP proxy (running on Client Access role), routing traffic to Transport Service on Mailbox server role. Bind the connector to DAG IP address.

New-ReceiveConnector -Name “From Internet” -Bindings “192.168.1.55:32” -PermissionGroups AnonymousUsers -TransportRole FrontEndTransport -Usage Internet

Identity                                Bindings                                Enabled ——–                                ——–                                ——-

FRONTEND1\FromInternet                  {192.168.1.55:32}                       True

  • Set receive connectors to access traffic from smart hosts if needed.
  • Set up routing of SMTP traffic to virtual DAG IP address
  • Check firewall to be sure SMTP traffic is allowed to traverse the network
  • The node to which traffic will flow is the one owning DAG IP Address (Active manager). There is info how to determine Active manager in my article: https://ficility.net/2012/09/02/exchange-2010-dag-active-manager-determinemove/

Downsides:

  • SMTP traffic is not load balanced for external traffic

PowerShell – Functions and examples how to work with IP addresses (IPv4)

The IP addreass can be declared by data type System.Net.IPAddress. These examples are equivalent:

[System.Net.IPAddress]$Address = "192.168.23.12"
[IPAddress]$Address = "192.168.23.12"

$Address object contains:

Address : 202877120
AddressFamily : InterNetwork
ScopeId :
IsIPv6Multicast : False
IsIPv6LinkLocal : False
IsIPv6SiteLocal : False
IsIPv6Teredo : False
IPAddressToString : 192.168.23.12

TypeName: System.Net.IPAddress
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object comparand)
GetAddressBytes Method byte[] GetAddressBytes()
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
Address Property long Address {get;set;}
AddressFamily Property System.Net.Sockets.AddressFamily AddressFamily {get;}
IsIPv6LinkLocal Property bool IsIPv6LinkLocal {get;}
IsIPv6Multicast Property bool IsIPv6Multicast {get;}
IsIPv6SiteLocal Property bool IsIPv6SiteLocal {get;}
IsIPv6Teredo Property bool IsIPv6Teredo {get;}
ScopeId Property long ScopeId {get;set;}
IPAddressToString ScriptProperty System.Object IPAddressToString {get=$this.Tostring();}

As can be seen the IPAddress Class has some limitation for advanced tasks, but the following article uses its functionality really nice related to breaking up decimals in low-/high-bytes and creating IP address ranges (http://powershell.com/cs/blogs/tobias/archive/2011/02/20/creating-ip-ranges-and-other-type-magic.aspx)

New-IPRange

function New-IPRange ($start, $end) {
 # created by Dr. Tobias Weltner, MVP PowerShell
 $ip1 = ([System.Net.IPAddress]$start).GetAddressBytes()
 [Array]::Reverse($ip1)
 $ip1 = ([System.Net.IPAddress]($ip1 -join '.')).Address
 $ip2 = ([System.Net.IPAddress]$end).GetAddressBytes()
 [Array]::Reverse($ip2)
 $ip2 = ([System.Net.IPAddress]($ip2 -join '.')).Address
 for ($x=$ip1; $x -le $ip2; $x++) {
 $ip = ([System.Net.IPAddress]$x).GetAddressBytes()
 [Array]::Reverse($ip)
 $ip -join '.'
 }
}

New-IPRange

Do not use  “0.0.0.0”  and “255.255.255.255” because you will wait on 4294967296 addresses :-).

Of course we can specify IP range through  Microsoft.Exchange.Data.IPRange:

LowerBound : 157.144.33.22
UpperBound : 157.144.33.22
Netmask :
CIDRLength :
RangeFormat : SingleAddress
Size : ::1

LowerBound : 157.144.202.0
UpperBound : 157.144.202.127
Netmask : 255.255.255.128
CIDRLength : 25
RangeFormat : CIDR
Size : ::80

TypeName: Microsoft.Exchange.Data.IPRange

Name MemberType Definition
---- ---------- ----------
CompareTo Method int CompareTo(Microsoft.Exchange.Data.IPRange x)
Contains Method bool Contains(ipaddress ipAddress), bool Contains(Microsoft.Exchange.Data.IPvxAddress ipAddress)
Equals Method bool Equals(System.Object obj), bool Equals(Microsoft.Exchange.Data.IPRange other)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
CIDRLength Property System.Int16 CIDRLength {get;}
LowerBound Property Microsoft.Exchange.Data.IPvxAddress LowerBound {get;}
Netmask Property System.Net.IPAddress Netmask {get;}
RangeFormat Property Microsoft.Exchange.Data.IPRange+Format RangeFormat {get;}
Size Property Microsoft.Exchange.Data.IPvxAddress Size {get;}
UpperBound Property Microsoft.Exchange.Data.IPvxAddress UpperBound {get;}

But I would like to  focus on converting numbers to binary and back (http://www.powershellmagazine.com/2012/10/16/converting-numbers-to-binary-and-back/) because it is basic principle for network addressing.

New-IPv4toBin

function New-IPv4toBin ($ipv4){
 $BinNum = $ipv4 -split '\.' | ForEach-Object {[System.Convert]::ToString($_,2).PadLeft(8,'0')}
 return $binNum -join ""
}

New-IPv4toBin

New-CidrToBin

function New-CidrToBin ($cidr){
 if($cidr -le 32){
 [Int[]]$array = (1..32)
 for($i=0;$i -lt $array.length;$i++){
 if($array[$i] -gt $cidr){$array[$i]="0"}else{$array[$i]="1"}
 }
 $cidr =$array -join ""
 }
 return $cidr
}

New-CidrToBinGet-Broadcast (require New-IPv4toBin)

function Get-Broadcast ($addressAndCidr){
 $addressAndCidr = $addressAndCidr.Split("/")
 $addressInBin = (New-IPv4toBin $addressAndCidr[0]).ToCharArray()
 for($i=0;$i -lt $addressInBin.length;$i++){
 if($i -ge $addressAndCidr[1]){
 $addressInBin[$i] = "1"
 } 
 }
 [string[]]$addressInInt32 = @()
 for ($i = 0;$i -lt $addressInBin.length;$i++) {
 $partAddressInBin += $addressInBin[$i] 
 if(($i+1)%8 -eq 0){
 $partAddressInBin = $partAddressInBin -join ""
 $addressInInt32 += [Convert]::ToInt32($partAddressInBin -join "",2)
 $partAddressInBin = ""
 }
 }
 $addressInInt32 = $addressInInt32 -join "."
 return $addressInInt32
}

Get-BroadcastTest-IPinIPRange (require New-IPv4toBin)

function Test-IPinIPRange ($Address,$Lower,$Mask) {
 [Char[]]$a = (New-IPv4toBin $Lower).ToCharArray()
 if($mask -like "*.*"){
 [Char[]]$b = (New-IPv4toBin $Mask).ToCharArray()
 }else{
 [Int[]]$array = (1..32)
 for($i=0;$i -lt $array.length;$i++){
 if($array[$i] -gt $mask){$array[$i]="0"}else{$array[$i]="1"}
 }
 [string]$mask = $array -join ""
 [Char[]]$b = $mask.ToCharArray()
 }
 [Char[]]$c = (New-IPv4toBin $Address).ToCharArray()
 $res = $true
 for($i=0;$i -le $a.length;$i++){
 if($a[$i] -ne $c[$i] -and $b[$i] -ne "0"){
 $res = $false
 } 
 }
 return $res
}

The first IP address is checked regarding IP range (IP,mask or IP,cidr)

Test-IPinIPRange

New-IPv4fromBin

function New-IPv4fromBin($addressInBin){
 [string[]]$addressInInt32 = @()
 $addressInBin = $addressInBin.ToCharArray()
 for ($i = 0;$i -lt $addressInBin.length;$i++) {
 $partAddressInBin += $addressInBin[$i] 
 if(($i+1)%8 -eq 0){
 $partAddressInBin = $partAddressInBin -join ""
 $addressInInt32 += [Convert]::ToInt32($partAddressInBin -join "",2)
 $partAddressInBin = ""
 }
 }
 $addressInInt32 = $addressInInt32 -join "."
 return $addressInInt32
}

New-IPv4fromBinAdvanced examples:

IPAddressExamplesDownload: IPAddressFunctions.ps1 (click for content)

How to load all functions above into Powershell:

FilePathLoadPS1

Exchange – Offline Address Book – OAB download methods, Cached vs Online

Theory

By default OAB is a point in time snapshot of global address list and it is used as cached source of information about Exchange recipients properties. OAB is stored on Exchange servers (see my previous article https://ficility.net/2013/03/04/oab-differences-between-exchange-2010-and-exchange-2013-in-brief/) and downloaded to client once Outlook is configured in Cached mode. I would like to test modes of using address book,
while Outlook is in cached mode. There are several methods to download OAB. These methods depends on registry settings of Outlook (full article here:http://support.microsoft.com/kb/823580)
shortly:

If the following registry key is present (XX.0 means office version – 15.0 for Office 2013), Outlook behaves upon the DWORD value inside:

HKEY_CURRENT_USER\Software\Microsoft\Office\XX.0\Outlook 
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\XX.0\Outlook

DownloadOAB DWORD supported values:
  0 = The Offline Address Book does not download automatically.
  1 = The Offline Address Book uses the Download Full Items download mode. This is the default setting.
  2 = Download the Offline Address Book in any download mode, but download a differential update in the Header only download mode.
  3 = Always download the Offline Address Book and a differential update in one of the following download modes:
Download Headers and then Full Items
Download Full Items
Download Headers

The goal of this article is to test differences in OAB behavior between modes 0 and 1. I want to see how it looks when:

1) Mode 1 – The Offline Address Book uses the Download Full Items download mode.
a)Download OAB and check
b)Change GAL and test downloading OAB instantly
c)Update OAB and check

2) Mode 0 – The Offline Address Book does not download automatically.
a) Test behavior once mode 0 is configured while old OAB files are still on the client
b) Test behavior once mode 0 is configured while old OAB files are removed from client

Test scenarios

1) Mode 1 – The Offline Address Book uses the Download Full Items download mode.

By default OAB is downloaded from server hosting Active mailbox database with Organizational mailbox. (or in 2010 from CAS server distribution point or in 2007 and 2010 from Public Folders)

a) Download OAB and check

I have opened OAB from my mailbox and result is in the Picture

1-Before change

b) Change GAL and test downloading OAB instantly

Creation of a mailbox doesn´t updtate OAB itsetf. To create mailbox use command

New-Mailbox OAB_Test_o1 -UserPrincipalName OABTestO1@salonovi.cz
WARNING: A script or application on the FRONTEND1.SALONOVI.CZ remote computer is sending a prompt request. When
prompted, enter sensitive information such as credentials or password only if you trust the remote computer and the
application or script requesting it.

cmdlet New-Mailbox at command pipeline position 1
Supply values for the following parameters:
Password: ********

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
OAB_Test_o1               OABTestO1            backend1         Unlimited

c) Update OAB and check

To update OAB use command

Get-OfflineAddressBook | Update-OfflineAddressBook

Now I have tested if OAB change is reflected in client computer (should not be)

1-After_change_Before_Download

And now I have downloaded new OAB to client and tested again

1-After change and download

2) Mode 0 – The Offline Address Book does not download automatically

One registry key setting and Outlook client will work online from Addres book point of view, BUT! This setting also requires to clean up OAB files from client computer to behave correctly. I will test both possibilities and try to find differneces in behavior.

First I will set up registry key and restart Outlook

In my lab I dont use any special GPO so yhe setting is done via registry key HKEY_CURRENT_USER\Software\Microsoft\Office\XX.0\Outlook

2-Registry change - Added

OAB files are left on the client

a) Test behavior once mode 0 is configured while old OAB files are still on the client

New mailbox should be immediately visible for client without need to download OAB since information should be available online.

b) Change GAL and test if it appears immediately to client

Change in OAB is not visible immediately, because we have OAB files on the client computer and Outlook use those!

New-Mailbox OAB_Test_o2 -UserPrincipalName OABTestO2@salonovi.cz

2-RAfter change - no download!

c) Update OAB automatically by restarting Outlook

If Outlook is restarted, OAB version is checked / downloaded from Exchange server.

2-RAfter change - after download!

AND NOW THE NAUGHTY stuff! OAB is not updated anymore even you download files successfully from Exchange. In this stage Outlook is stuck somewhere between mode 0 and 1 and updates are not received by client.

d) Update OAB and download change to client manually even mode 0 is used

In this scenario I would like to prove, that setting mode 0 is not the only thing to consider to have Outlook work correctly.

  • To update OAB use command
Get-OfflineAddressBook | Update-OfflineAddressBook
  • Download OAB to client and check if changes are displayed to client.

2 - OAB manual download result

 Changes are only reflected if I try to manually download full or incremental copy of address book. OAB is not downloaded during the client startup!

OAB files are removed from client

a) Test behavior once mode 0 is configured while old OAB files are removed from the client

I added new mailbox again, updated OAB, but for now I have removed all OAB-Related files from client. Changes should appear to client immediately.

To remove OAB files

  • locate:
    c:\Users\<USERNAME>\AppData\Local\Microsoft\Outlook\Offline Address Books\
  • Delete folders
    Example: (c:\Users\lelicek\AppData\Local\Microsoft\Outlook\Offline Address Books\6a285982-48d6-43ee-979b-f84dd5b7d989\)
  • Start Outlook
  • After Outlook startup, folders will be re-creaded, but will be empty. It proves we have mode 0 set to Outlook client.
  • open Offline Address Book
  • Newly created mailbox should be here

2 - OAB folders removed change present

 d) Do change in GAL and test if it appears immediately to client

  • Create new mailbox
New-Mailbox OAB_Test_o5 -UserPrincipalName OABTesto5@salonovi.cz
  • Open Offline Address Book and new mailbox should be again there

2 - OAB folders removed change present again

e) Download OAB manually and test if the changes made to GAL after OAB download will be imediatelly visible to client

2 - OAB manual download

Conclusion

Mode 1

  • Conclusion is , that mode 1 works fine and as it should and user will get updated OAB after regularOAB update schedule or after manual run of Update-OfflineAddressBook command.
  • Note that in Exchange 2010 you must restart File distribution service, to distribute updated OAB to WEB distribution points. In Exchange 2013 it is not needed anymore.

Mode 0

  • Use mode 0 only in case, that you do a lot of changes in GAL and you need clients to see changes immediatelly while taking advantages of Outlook Cached connection.
  • Once mode 0 is used, administrator has to make sure, that OAB files will be removed from client computer (for example by logon script / GPO), otherwise user must use manual OAB update via Send/Receive -> Send/receive groups / Download Address Book.
  • If user tries to manual download OAB while mode 0 is used, Online functionality will STOP working from that time until OAB files are deleted again!