iPhone OS4 and EAS issues

I do remember my first “real” tech job way-back-when.  Exchange 5.5 was cutting edge technology, which is dating me for sure. We had a graphics consultant come into the company for a couple of months to help re-design our website, and that required that this employee have an email address and account with our firm.  Getting them an account was no issue, as Mac systems could easily talk to an Exchange 5.5 server at that point, but once we connected the Mac to the network, it immediately took over the Exchange system. 

I don’t mean that it was easier to control Exchange via the Mac, I mean IT TOOK OVER THE @!#(%$ SERVER!

No one could get email, no one could send us email, nothing worked at all unless you were on the Mac itself.  Finally, after bringing in a Mac expert for a few hours and a ton of troubleshooting with Microsoft, we found the issue. It was a rare hiccup in the Mac networking that caused Exchange to essentially ignore anything else, and we fixed it with no future problems.

Recently, I heard of the same issue happening again, but not rare, and on the iPhone.  After the complete metal breakdown brought on by the thought of the nightmare recurring, I did some investigative work for you, my readers – and for the sake of my sanity.

As reported in Exchangepedia, the issue revolves around a bug in the iPhone OS4 software that hurls a wrench into the Client Access Services (CAS) systems of Exchange 2007 and 2010.  This isn’t an Apple-specific issue, as you could see the same problem on Google devices, but it does appear that it happens much more often with iPhone systems running OS4.  What happens is that the iPhone makes a series of requests to Exchange to perform ActiveSync.  The problem is that there is a relatively short timeout on these operations (30 seconds) and if the CAS system is busy, the iPhone will send the requests again.  And again.  And again… Eventually, the flood of requests becomes a contributing factor in the delay of the response, creating a never-ending loop.  The only way to stop it is to either stop the iPhone or the CAS system and start over, but eventually it will happen again. 

The good news is that there are ways to increase the timeout, and therefore reduce the incidence of this happening in your corporate environment.  The link above has information on how to obtain a profile update that fixes the problem in most cases.

I guess everything old *is* new again.

What’s a Rollup?

Since the news a few days ago that Microsoft has released the Exchange 2010 Update Rollup 4 for download, I figured a couple of lines on what Rollups are would be a good thing. Many are confused as to why these Rollups exist, and when to use them.

An Update Rollup (UR) is a single-installer for many different hotfixes and patches to a single product that Microsoft produces. There have been such UR’s for everything from Windows to Office over the years, and they can make life easier if used correctly. For the most part, you don’t need them for day-to-day operations anymore, but you may use them when building up new systems.

For currently-installed Exchange 2007 and up servers, running Windows Update on a regular basis will get you the critical patches, fixes and updates you need. Just make sure you’ve authorized Windows Update to download software for other software running on Windows (it’s one of your settings in Windows Update) and you’ll find yourself downloading Exchange components as well. You still need to install Service Packs as they’re available, but the individual patches can be updated when you do your regular Windows Updates. There are a couple of cases, however, where this updating process doesn’t meet your needs.

If you’re building a new Exchange 2007 or 2010 system, for example. After installation and (where applicable) Service Pack updating, there may be a large number of individual fixes and patches to be applied. As these will be installed sequentially, and will require service or server restarts in many cases, the process can be time and labor consuming. The UR can apply all patches and fixes up to the date of UR release in one shot, making things faster, easier, and with fewer opportunities for errors.

Many companies prohibit auto-updates, and you may find yourself manually applying fixes and patches for Exchange. If you keep up with this process, then you have no problems, but if you do not; you can quickly find yourself knee-deep in required updates. The UR will make life easier by bringing you up to date quickly, so you can get back to regular patching.

Note that you should still check for updates *after* you go through the UR process. This is because there may be newer patches and fixes released after the UR is built, so while it is cumulative up to the release date, it is not current to today’s date.

Keeping up to date is the best way to avoid security pitfalls and software bugs.  When you’re starting from scratch or trying to catch up, Update Rollups can provide a way to quickly get close to “up-to-date” and help you do so without as many reboots, restarts and errors.

Why would you bring up an Exchange Server offline?

The Exchange Team Blog recently posted an article on how to help speed up the time certain Exchange Server services take to start when the server isn’t connected to the Internet.  You can read the article via this link. The question I’ve been hearing after this article showed up was “Why would I bring up an Exchange Server that isn’t net connected?  Isn’t the point of Exchange to send/receive mail?”  Valid question, but I can think of a few instances where this article can really help folks.

1 – You aren’t sure what’s broken

This is a pretty common one, believe it or not.  When troubleshooting Exchange, sometimes servers can appear to be connected to the Internet, but are not.  They may also require several reboots to properly correct whatever went wrong with the network connection once you figure out what broke.  In both cases, having a way to speed up the Exchange service startup can make troubleshooting faster and more efficient.

2 – You’re testing something out

Many times, administrators will want to see how certain changes impact Exchange Servers without allowing them to receive new mail (just in case they have to revert to a previous version of the database).  This is typically done by disconnecting the server from the network, as that will ensure no mail is transferred in any direction.  If a reboot is required for the testing, shortening service start times makes life easier.

3 – Secured environments

The article specifically mentions this one.  If Exchange Server is used for internal mail routing ONLY, and never expected to send/receive anything on the public Internet, then it may not be connected to the public Internet at all.  Under these specific circumstances, there’s no reason for things like public certificate updates to happen, since the Exchange Server will never use them.  Be very careful with this one, though. You’d be amazed how many times I have found so-called “isolated” Exchange Servers that are chattering away with the outside world.

As you can see, there are reasons why you would bring an Exchange Server up without a connection to the Internet.  In those circumstances, the guidance in the article can help speed up the process of getting your Exchange services started.  Just be careful to follow all the warnings in the articles, as a wrong move could either break Exchange or create a whopping security hole in your email architecture.

Offline Address Book Generation/Update

The OAB is used for Cached/Offline mode clients to perform lookups of contacts and other information. It is periodically updated from the Exchange Server to Outlook clients. A complete overview of how the OAB works in various versions of Exchange can be found on Microsoft TechNet by searching the sections for your specific version of Exchange.

In earlier versions of Exchange Server, a Public Folder was used to house the OAB data and distribute it. In 2007 and 2010, a web-based distribution system was put in place, with the ability to also maintain a Public Folder version in the event you have legacy Outlook (2003 SP1 or earlier) clients. While distribution of the OAB to any Outlook client is fairly automatic, the generation of the OAB for the first time may not fit into your time tables.

Usually, the OAB will update as part of the Online Maintenance Routines that occur by default every day from 1am to 3am server local time. This means that the OAB will not be generated at all until that first maintenance window, and will not be updated with new users until the first maintenance window after they are added to AD. There are several reasons why this may not work for you, but some common ones are:

- You’ve added a VIP to the AD system and management doesn’t want to wait 24 hours for the OAB to update.

- You have created a test system, and either don’t want to leave the system running overnight, or need to perform testing within the first 24 hours of the build.

- You add a large number of users to AD, and want to propagate the information to Outlook clients as quickly as possible.

In these, and other cases, you could force the Online Maintenance to occur, but that would take a significant amount of time, especially if all you want to do is update the OAB. In a production environment, performing maintenance routines during peak hours of operations can also have an impact on user experience, and so should be avoided.

To perform an immediate update of the just the OAB in Exchange Server 2007 or 2010, you can open a PowerShell command window and execute the following:

Update-OfflineAddressBook –id “Name of OAB” –verbose

Where “Name of OAB” is the name of the Offline Address Book you wish to update. Be sure to enclose it in quotes, especially if the name contains spaces. The default OAB for a new installation of Exchange is – oddly enough – “Default Offline Address Book” and you can use that name to perform the initial update on a brand new server.

If you maintain more than one OAB, or if you changed the default name, you must run the command for each OAB in your organization, replacing the name with the appropriate string each time.

The next time Outlook clients look for OAB information, they will see that the OAB has been updated, and download the appropriate changes to their local OAB copy.

Got an Exchange-related question that Mike can try to get you an answer for? Follow Mike Talon on Twitter (@TalonNYC) and send an @ mention with your question.

What the {Censored} Do I Install for Exchange 2010??!!

This is a complaint I have heard over and over again from clients going to install Exchange 2010l and even earlier versions.  The level of profanity depends a lot on the individual, but I have heard some rather prim and proper folks resort to very short words when it comes to Exchange pre-requisites. Come to think of it, they've resorted to some long words as well, but none fit for publication in family-friendly venues.

Exchange prerequisites have typically been confusing and difficult to master.  Starting with Exchange 2000 and 2003, the pre-flight checks built into the setup protocols became intelligent enough to figure out what was missing for the most part.  However, the installer didn’t tell you what those things were unless you started playing with the install options and suddenly got a popup which detailed what wasn’t there.  Exchange 2007 and now 2010 both have extensive pre-flight checks that occur before you do any form of installation, just after you tell the installer which components you’d like to put on the box.  They check for everything from your .NET versioning to Active Directory connectivity, but once they tell you all the different things that are not present, how do you figure out what to install in order to make those things present?

Thankfully, Microsoft has had TechNet pages that detail prerequisites for Exchange installations, starting back in Exchange 2000.  Up until 2007, it was up to you to install each of the components listed, but with the advent of newer server management tools in Server 2008, the process can be automated.

For example, to pre-stage Server 2008 R2 for Exchange 2010, you can invoke a series of PowerShell commands to set up nearly all the prerequisites in one shot.  First run:

Import-Module ServerManager

inside a PowerShell window in order to load the correct command modules into the session.  Then, use the correct sequence of commands in the same PowerShell window to load the prerequisites for the roles you need to run on this particular server.  For example, if you wanted to run Hub/Transport, Client Access Services and Mailbox Services; you would execute the following block of commands:

Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy –Restart

The specific commands (and the tool to call them) will change based on three things:

1. The version of Exchange (2007, 2010, etc)
2. The version of Windows (2003, 2008, 2008 R2)
3. The Roles you intend to install (Hub/Transport, Client Access, Mailbox, Unified Messaging, etc.)

Some things, like installing the 2007 Office System Converter Filter Pack; still have to be done manually, but most of the basic pre-installations can easily be automated using these combinations of command window and PowerShell scripts.

Microsoft has put up all the commands and processes for all the Server OS’s that Exchange 2010 can run on in this TechNet article.  Similar articles exist for previous versions of Exchange too, so you can get your prerequisites onto the server before you even start the Exchange 2010 installation executable.

Outlook Mail stuck? It may not be your profile.

During a recent Exchange 2010 training, our instructor went through a couple of things that could be causing mail to not leave an Outlook or OWA client during normal operations.  When I did a few searches on this topic, I was alarmed to see that many people suggest that the problem is the Outlook offline storage file or PST.  While that does sometimes cause these issues (especially if you’re on Exchange 2003 or earlier), there are a couple of quick checks you should make against the server first.

On Exchange 2007 and 2010, if mail is getting stuck in users’ Outboxes on various clients, check two things:

1 – Is there less than 4GB of disk space available on any volume on the servers that contains Exchange data and/or queues? If there is less than 4GB, Backpressure protocols kick in, and Transport Services won’t move mail anywhere, leaving new messages stuck in users’ Outboxes. This is fixed by freeing up drive space or moving the database and/or queues to another volume with more room.

2 – If the disk space is ok, check to see if the Transport Services (which vary depending on your Exchange version) are running.  Any service that isn’t started can cause mail routing to fail, and messages to stay on the client software.  Obviously, to fix this you either need to get the services up or bring another Hub/Transport system online to handle mail flow.

Additionally, if you are using Exchange 2010, mail may get stuck in the Drafts folder.  That is to say, it leaves the Outbox and immediately appears in the Drafts folder in Outlook.  If you see this behavior, check to ensure that the Mail Submission service is running on your CAS server.  If it isn’t, mail can’t be properly picked up from the Outlook client, and the resulting error shoves the email into the Drafts folder.

Granted, if these issues are not the cause of your issue, or if it only seems to be happening to one user, then rebuilding the OST or PST might be in order.  However before you disrupt clients day-to-day activities, a quick check of the server might uncover a correctable error that can be quickly contained.  In the case of lack of disk space, it can even expose a growing problem that must be dealt with in the near future to avoid much more wide-spread downtime.

Don’t uninstall to fix Exchange!

Exchange Server does, occasionally, act oddly.  Alright, it can do that a lot, but hopefully you won’t run into the phenomenon.  If you should see strange behavior from your Exchange Server, the first thing to remember is that it is not a truly stand-alone application, and can’t be treated as such.  When problems happen with many software packages, the default fix-it is to uninstall the applications, reboot and reinstall.  While that may work with many apps on the Windows Server platform, doing that for Exchange Server will lead to rather disastrous results.

When Exchange Server installs, one of the first things it does is start building structures in Active Directory.  If you have no Exchange Servers at all, this includes extending the Schema and creating an Organization for Exchange and the related security constructs in both the Forest Root domain and the domain you’re installing into.  The thing many folks don’t realize until it is too late, is that much of this mucking about in Active Directory is undone when you uninstall an Exchange Server, which could be a problem if you plan on just fixing something and moving on.

When you uninstall Exchange Server, the uninstallation protocol will remove all objects in AD that were assigned to that server, which could lead to unexpected end results for users, Public Folders and other objects that don’t exist elsewhere in your organization.  This means that an attempt to uninstall to fix a somewhat minor issue could result in an absolute disaster for your Exchange environment overall that cannot be fixed by simply failing over.  The Organization and Forest Root updates will remain, but most of what you did in terms of configuration for the specific Exchange Server will be wiped out.

So, what to do in order to avoid this?  First, hit http://technet.microsoft.com and http://support.doubletake.com to see if there are articles on how to fix the problem you’re seeing.  These documents generally detail much easier fixes for common Exchange problems, and not-so-common ones as well.  Correcting the issues with Exchange still in place is the best route to take when it’s available.

If you do have to reinstall Exchange, your best bet is to shut down the malfunctioning server, build a new one with the same name, and use the disasterrecovery or recoverCMS switches with the Exchange Server installer tools.  This will transfer the settings over to the new machine, instead of removing everything permanently from AD.  Nearly all configuration for Exchange is stored in AD, so using these switches during the installation will allow you to get back up and running quickly, ready to restore the mailbox and folder data from an independent backup.  More information on these switches, the exact syntax for each version of Exchange Server and guidelines for use can be found in various places. http://support.microsoft.com, http://www.msexchange.org and http://technet.microsoft.com are three great resources.

As with most other topics, Exchange acts differently from most other applications when it comes to reinstallation. Always check the support sites before you take any drastic measures, and remember that Exchange is a totally AD-integrated application, so you’re never working on “just the Exchange Server itself.”

Hyper-V Me (Yes, Even for Exchange)

Due to the release of Hyper-V R2, there has been a lot of buzz about using Microsoft’s Virtualization Platform to create virtual machines for everything in the datacenter from file servers to high-capacity databases.  There are quite a few systems that are clearly ok to use virtualization with, but Exchange Server has always been one of those platforms where virtual tech has been a questionable course.  Not the least of the reasons for this is that folks have been confused on if MSFT supports running Exchange in production on a virtual platform.  So, I’ll spell out the current support stance as found on official MSFT sites only.  Note:  There are many blog postings that point one way or the other, so I have hunted down official MSFT sites to quote here (i.e. TechNET, MSDN, etc).  While many blogs can be trusted (like the official Exchange Team Blog at http://msexchangeteam.com – and http://www.BeingExchanged.com of course), many of my readers have corporate policies about needing to see the documentation from a MSFT site directly; and so here it is:

Exchange 2000 and earlier:

Not supported for production environments.  See the end of the TechNET article found here: Link to TechNET

Exchange 2003:

Supported in production.  Warnings are given that you must use Virtual Server (2005 R2 or higher) and that there are some performance limitation to be considered. Link to TechNET

Exchange 2007

Supported for production environments on Hyper-V, but with some limitations. You must be running Server 2008 as the Guest OS, and you have to be running Exchange 2007 SP1. Unified Messaging may not be run on a Hyper-V Guest.  You cannot use both SCR and/or CCR and Hyper-V clustering at the same time. Finally, the virtual machine itself must meet all the hardware requirements (processors, RAM, etc) for Exchange 2007 Sp1. Link to TechNET

Exchange 2010:

Supported for production environments on Hyper-V, with some restrictions:  You must be running Server 2008 SP2 or R2 as the Guest OS, and Unified Communications Roles are not supported at all.  As with 2007, the virtual machine you install Exchange into must meet all the hardware requirements for Exchange 2010. You will also need to choose between DAG availability or Hyper-V failover solutions, you cannot run both at once. Link to TechNET (see section on Hardware Virtualization)

Common notes:

In all supported versions, certain functions of the virtualization systems are not supported with Exchange Server.  Specifically, dynamically expanding virtual disks are not allowed, you have to use pre-configured fixed-sized VHD’s for Exchange Server virtual machines. Likewise, you cannot leverage differencing or VSS snapshot disks if you’re running Exchange Server.

There are several rules about virtual hardware that you must follow as well, so read the related TechNET articles for guidance on things like disk types (SCSI, IDE, etc) and the like.

Hyper-V technology is bringing new opportunities to the virtualization technologies platform in the modern datacenter.  By clarifying what is and is not supported in a virtual environment, Microsoft has begun the process of allowing customers to safely start using virtualization for the Exchange Server platform.  Many users will still choose physical hardware for Exchange (either because they cannot meet one or more of the requirements, or just due to having hardware already provisioned), but these rules help clarify that options for virtualization exist and are supported.

A word on Cluster Groups

Many clients are on Exchange 2003 or 2007 and will need to deal with Cluster Groups in Microsoft Cluster Services (MSCS) or Failover Clustering Services (FCS) including Cluster Continuous Replication (CCR).  So, it is important to understand one very critical restriction of Exchange Clustering that I’ve seen several clients trip over.

When installing a Microsoft Cluster of most flavors, you will configure Groups, which are logical units used to contain Resources like IP Addresses, Network Names, Disks and Services.  By default, a Cluster Group will be created that contains the name, IP address and Quorum Disk for the cluster itself.  It may also contain a networked Distributed Transaction Coordinator (DTC) resource for the cluster as a whole.  It is very tempting to place all other resources in this group, but you should avoid doing that at all costs for 2 significant reasons:

1 – It’s not supported by Microsoft.  For proof, I refer you to This TechNET article.  There is a long explanation of many thing having to do with configuring an Exchange Cluster, but here’s the specific info I’m referring to:

“It is an Exchange best practice to install the MSDTC resource into the default cluster group. However, the MSDTC resource is the only resource supported in the default cluster group. Exchange resources should not be added to the default cluster group, as that configuration is not supported.” [emphasis added]

TechNET and the Microsoft Sites have many other examples of this warning, and it is well documented by Microsoft and the Exchange Product Team.

2 – It makes life more difficult in day-to-day administration.  There may be instances where you want to perform operations on the Cluster Group without interrupting Exchange services for your organization.  You can normally accomplish this by moving the Cluster Group to a Cluster Node that isn’t hosting any Exchange Resource Groups, and perform your activities on that node.  If you have Exchange Resources in the Cluster Group, then this options disappears.  The same goes for many 3rd-Party products (see disclaimer at the end of the blog) which may not accept Exchange Resources that appear in the Cluster Group, as they must treat the Cluster Group and the Exchange Resource Group independently for administrative purposes.

So, as tempting as it is, avoid installing Exchange Resources into the Cluster Group at all costs.  If you already have put Exchange Resources into the Cluster Group, and you don’t plan on upgrading just yet, then seriously consider migrating to a supported cluster configuration when time permits.  Issues that arise from unsupported configuration and limited administration tend to hit without warning, and at the worst possible time.  Taking time to move to a supported platform will keep your organization in the safe zone, and make life a lot easier for you over time.

Look before you leap

There’s a great deal of talk these days about migrating directly to Exchange 2010 instead of jumping first to the 2007 version of the platform.  The arguments for and against this got more complicated when Microsoft did a reversal for support and announced that 2007 would be able to run (at some future point) on Server 2008 R2.  This effectively extended the support cycle of 2007, which means that a slightly older, and more thoroughly tested and widely installed, version is a legitimate option going forward.  That being said, many clients are opting to just go to Exchange 2010 from whatever they’re on now, skipping 2007 if it isn’t already in place.

I’m going to avoid discussing the pros and cons of going to Exchange 2010 directly in this particular post (that’s fodder for quite a few additional posts though).  Let’s just say that you’re jumping from an earlier version directly to 2010 and go on from there.  While most of the process isn’t too bad overall, there are a couple of sticking points that you’ll need to have sorted out before you make the move.  In smaller organizations, these are pretty easy issues to take care of, but on large-scale roll-outs, they can be a problem.

First, make sure your domains are at least at Server 2003 native functional domain levels.  Most of you are already there, but just in case you had a few NT servers hanging around, be ready for this one.  As long as all your servers are 2003 and up, making this change doesn’t have a lot of impact on your domain.  You can find out about the impact of raising Domain Functional Levels in this TechNET article.  So, what if you DO still have Exchange 2000/Windows 2000 servers in your environment?  You need to upgrade to Exchange 2007 as an interim step to 2010.  Since the Exchange 2000 life-cycle is definitely over (and has been for a while), Microsoft offers no method of co-existence between Exchange 2000 and Exchange 2010 – migration or otherwise.

Once your domain is prepped to 2003 or 2008 native mode, you then must plan to keep your old Exchange infrastructure long enough for the migration to occur.  That’s because the Mailbox systems for one version of Exchange can’t talk to any other in most cases.  So a 2003 Mailbox Server can’t talk directly to a 2010 Mailbox Server without some assistance.That assistance is in the form of Bridgehead servers (2003) or Hub/Transport and Client Access Servers (2007). You will need to have at least one of each for each Site you have set up in Exchange/AD.  One note, this has nothing to do with the migration itself, this is all about the period of co-existence you’ll need to go through until everyone is on the 2010 architecture.

After you have upgraded your Domain Functional Level and set up your front-end server systems, you can begin the process of installing the first of your Exchange 2010 server systems.  Start with the Hub/Transport and Client Access roles (unless you are installing them on the Mailbox Role server itself) to allow for message routing to get set up.  This lets mail flow between the 2010 and legacy front-end servers, so when you start moving mailboxes, you don’t lose mail connectivity.

Finally, once you’re sure mail is flowing correctly and that no one is about to get lost in the shuffle, take one more backup of the legacy systems (just to be safe) and start your migration.

You can find a guide to preparing for and deploying Exchange 2010 at this TechNET site.  Along with other blogs and Microsoft sites, this should be required reading before you even download the software and play with it in a lab.  With a migration that includes a period of co-existence with earlier versions of Exchange; careful, proper planning is not a nice-to-have thing.  It is a solid requirement that must not be overlooked or under-appreciated.

Something Old, Something New

As many of you know, Exchange Server 2010 Released To Manufacturing (RTM) today, with TechNET and MSDN releases to follow later this week.  There are a ton of new features, like Database Availability Groups and Archiving and compliance put in place in this version, so it’s a big step for Microsoft.  As you might expect, I’ll be writing quite a few articles on the new features as Exchange 2010 rolls out.

In the meantime, a recent 180 degree turn by Microsoft on support for Exchange 2007 has extended the theoretical useful lifetime of that platform by quite a bit.  It may not be time to start writing off Exchange 2007 just yet =)

As reported by Mary-Jo Foley in this ZDNet Blog post, MSFT has announced that they will continue to support Exchange 2007 into the Server 2008 R2 platform.  Prior to this point, official support for Exchange 2007 would end on the Server 2008 RTM platform, limiting the lifespan of the 2007 product to the 2008 RTM server. That wasn’t going to be tomorrow, or anything like that, but it certainly would be a shorter time-span than if Exchange 2007 got R2 support. 

Yielding to immense pressure from the end-user community, MSFT did acknowledge that not everyone was ready to upgrade to Exchange 2010.  There were no details on exactly when an update for Ex2007 would be available for R2, but suspicions are that it will be done via either a Roll Up or possibly even a new Service Pack early in calendar 2010.  Traditionally, this type of compatibility update had been done as part of a Service Pack, so my bet is riding on that idea.

Either way, the news that Exchange 2007 will live on to Server 2008 R2 is welcome.  This allows organizations who are in the middle of roll outs to not worry about if their servers are installed with Server 2008 RTM or R2, and will allow those migrations to complete on the 2007 platform.  This, in turn, allows upgrades to 2010 to occur without being rushed due to OS incompatibility issues.  There is something to be said for the fact that this decision will slow adoption of Exchange 2010 overall, but it will mean better, more structured roll-outs over time.  Safer, stronger and better planned upgrades are never a bad thing.

More Exchange 2007 OS Choice

A full article is coming early next week, but in the meantime:

Since Microsoft has announced that Exchange 2007 will indeed get support for Windows Server 2008 R2, you’ve got a new choice for running Ex 2007.  So what OS are you going with?

Can I get a Witness?

Continuous Cluster Replication in Exchange 2007 allows for two nodes of a Distributed Failover Cluster (DFC) for Exchange to be held in different physical locations and different physical network segments.  This is a good thing to leverage if you’re not concerned with local High Availability, but can lead to some interesting issues if something goes wrong.  The two nodes will use their quorum resources to find out which server should be in control of the cluster (and therefore assign resources accordingly) – but that doesn’t help if the nodes cannot see each other due to network failure.

One of two conditions would happen if you ran into this situation as described this far.  You could run into split brain, where both nodes thing they’re in charge and bring up Exchange resources.  This can take hours or even days of manual work to fix, and therefore Microsoft has taken steps to prohibit it.  If either node can’t figure out who’s supposed to be in charge, both go offline to prohibit split brain at all costs.

The second potential situation is the opposite, that neither node can figure out who is in control and both therefore shut down.  While this doesn’t put your data in danger, it does effectively shut off your Exchange system, stopping all messaging flow.  Neither situation is good, but by default, if arbitration is not possible via either quorum or other means, this safer situation occurs.  Luckily, there are “other means,” specifically the File Share Witness (FSW).

FSW is a file share (as its name implies) that both nodes can see under normal circumstances.  It must be placed on a server that isn’t part of the cluster.  Usually, you find it on a file server within the environment, but be aware that it will need to be at least Windows Server 2003 SP1 or better.  The FSW should also be placed either locally to the preferred node (the one you want to “win” in the event  of arbitration) or in an independent location that can be seen by both networks where CCR nodes reside.

In a CCR cluster, there are only two nodes, so right off the bat, if an arbitration event occurs, neither node could gain a majority and take over if there was some communication failure or other emergency.  The FSW acts as a third resource that can be polled to find out who is in control. Both nodes will attempt to take ownership of the FSW, but due to the physical placement of the Witness Server, only one will successfully do so.  That node stays online as owner of the cluster, the other node prohibits resources from going live until the emergency has been resolved.  As you can see, placement of the FSW becomes a critical component to the overall success of this arbitration system.

If you have only two physical locations, your best bet is to place the FSW on a server in the secondary site.  This allows the cluster to properly arbitrate to the remote site if the production site goes offline.  If you have more than two locations, then you can place the FSW on a server at a third location, just make sure connectivity to that site is stable and constant to and from both CCR servers.  If that link is unstable to one or more sites, you can create accidental arbitration events when they’re not really needed.  The benefit to putting the FSW at a 3rd site is that you can survive a link outage at either CCR node location without having to manually force one node or the other to take control (called a Force Quorum Operation). 

Here’s an example of what I mean.  If you have only two sites, and place the FSW at Site 2, a network link failure at Site 1 would force arbitration to Site 2 since the CCR node at Site 1 would not be able to communicate with either the node at Site 2 or the FSW hosted there.  In this scenario, there may be no value to failing over to Site 2, but you would automatically fail over anyway.  If, however, the FSW is hosted at a 3rd site, and both sites can see it, then a network fault between Site 1 and Site 2 would not flip everything to Site 2. Since Site 1 is the preferred owner, and can maintain control of the FSW, it will stay in control of the cluster.

You can find out a lot more about configuring FSW for Exchange 2007 via this TechNET article. The use of FSW technology is mandatory for CCR, and will continue to be a good idea for Exchange 2010 and Database Availability Groups as well.  Learning how this technology works today will allow you to create redundant solutions that last through your future Exchange solution sets.

Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Power(Shell)full Stuff

Windows PowerShell was introduced a few years back, but is still trying to find its way in the big, bright world of Windows even today.  This command-line interface allows users of all versions of Windows from XP SP2 on up to navigate through day-to-day operations without walking through layers of GUI interaction to get there.  While somewhat slow to take off in the mainstream Windows admin world, for Exchange Server (2007 and up) it has become an essential part of the Exchange Engineer’s toolkit.

With Exchange 2007, Microsoft removed a great deal of the control functionality from the Exchange Management Console (EMC) in favor of the extensions that Exchange makes to PowerShell, creating the Exchange Management Shell (EMS).  At first blush, this seems to be taking one giant leap backwards in terms of command and control on a Windows Server (DOS, anyone?) but once you dive a little deeper, there are a lot of advantages to be found.

Let’s start with speed.  EMC is slow, very slow.  Waiting for each command to finish loading up in the GUI just to figure out the piece of information you need is painful, to say the least.  The main reason for this is that the EMC doesn’t really do anything itself, it just displays the output of various PowerShell commands in a graphical format.  So each time you ask the EMC to do anything at all, what you’re really doing is waiting for it to create the corresponding PowerShell commands, evoke and run them, and then take the results and spit them back out on the screen for you.  For more complex tasks that are going to take some time anyway (like multi-mailbox move operations), this isn’t such a bad thing to deal with.  The GUI components make that job easier, and don’t add a huge amount of time to the overall process.  But for other operations, like getting a listing of all Storage Groups assigned to a particular server, the overhead in delays for getting the info and formatting it into the MMC3 GUI interface can mean the processes takes several times longer to perform in the EMC than it would at the command line.

Secondly, some techniques could never be performed inside of a GUI in any version of Exchange.  Database consistency checking and error correction were always done from the command line, and therefore it was logical to build those routines into PowerShell and the EMS as the newer versions of Exchange evolved.  By leveraging the way cmdlets (PowerShell code snippets) worked, much more complex database control and corrective action sequences could be piped through a single set of commands.  This lets administrators do more with fewer keystrokes, and opens up a whole new world to 3rd-Party software platforms who found that batch files just couldn’t cut it for what they needed to do.

Finally, the same way that cmdlets can expand what can be done within the framework of a script for things that were always command-line based; they can also allow administrators to automated a lot of their day-to-day work as well.  Prior to Exchange 2007, setting up users and mailboxes was a rather straight-forward process, but required that you interact with a series of GUI’s to get it done.  This took up extra time, and also introduced another level of potential errors every time you opened up another GUI.  Granted, this did not tend to lead to a lot of issues for experienced administrators, but even the most seasoned pro is going to slip up if they have to click in the same spot, over and over, several dozen times a week.  PowerShell allows for the creation of complex scripts that can leverage cmdlets, VB and C# commands and other attributes bound into single executable files.  This means that a set of operations can be crafted, tested and saved, then run over and over as required.  Less moving parts means less chance for errors, and portability means that a more experienced administrator can craft scripts for folks who may need to run repetitive tasks but not have the skills to work with cmdlets yet.  Entire online communities have sprung up to facilitate cooperative efforts on PowerShell scripts and to share what worked and what didn’t for various Exchange-related tasks.

Speed, efficiency and community interaction are just three components of what PowerShell can do to assist the average Exchange Engineer or Administrator.  Since the trend toward leveraging PowerShell and the EMS continues into Exchange 2010 (scheduled for release later this year), getting on-board with PowerShell tools now will build a knowledge-base that will grow with you as you continue to leverage new and better Exchange platforms.

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Net Neutrality and EAS

Mobile messaging and collaboration makes for a big, bright world in business today. What was once the domain of a single service provider (Research in Motion - RIM) has evolved into a robust set of platforms to convey email, appointments and contacts from one device to another.  Blackberries, Windows Mobile devices, iPhones, Android phones and so many other systems can communicate either directly or through a proxy to a Microsoft Exchange platform.  This unleashes the workforce and allows for your people to be where they need to be in order to work, instead of where they have to be just to talk to each other through the email system.

Net Neutrality is the idea that no matter what the network provider offers in terms of services and software, you should be able to use the devices of your choice and the platforms of your choice on those networks.  It’s a great theory, but putting it into practice is causing some issues along the way.  The FCC set forth a set of basic rules that they wanted carriers to follow, and in the greater sphere of comments, they were well received.  They recently added in two more proposed rules that directly impact cellular networks (digital broadband) and the services that run across it – including Exchange Active Sync (EAS).

You can get a rundown of the entire proposed rule set in this article, but the two that directly impact EAS most are:

5. Broadband providers cannot block or degrade lawful traffic over their networks, favor certain content or applications over others and cannot "disfavor an Internet service just because it competes with a similar service offered by that broadband provider."

On the surface, this looks like a standard anti-competitive rule.  In reality, however, many service providers in the cellular world are viciously blocking competing technologies, and their claim is that forcing neutrality will destroy their business.  EAS is a great example of this phenomenon, as not that long ago many providers didn’t allow that traffic on their mobile networks.  Mostly, this was due to the fact that they wanted to pus their own version of enterprise email synchronization (such as Sprint’s ill-fated attempt on the earlier Palm Treo devices).  Eventually, the need to allow this traffic or lose business to other devices and networks overrode the desire to use and sell their own platform, but that took a great deal of time, and lead to quite a bit of bad press and back-end attempts to circumvent the blocks.  By forcing mobile providers to allow all valid and legal traffic, the atmosphere for open communication standards will grow and more people will be able to take advantage of more technologies.

6.Broadband providers must be transparent about the service they are providing and how they are running their networks.

Proprietary networks are nothing new, but trying to create an EAS client for a phone on a network that actively blocks your ability to figure out how it sends and receives data makes this close to impossible.  Some providers have blocked all traffic they do not wish to have on their networks by simply making it very difficult – or nearly impossible – to figure out how a 3rd-party tool can possibly communicate with it.

I’m of two minds on these proposed rules.  On one side, EAS and other technologies require open, transparent communications platforms to work. Exchange can communicate with a whole world of different vendors’ mobile applications, but only if those apps can talk to the Exchange Server.  On the other side, competition drives better software and platforms.  If it wasn’t for all the things you can only do (or could only do) on an iPhone, RIM and Google would never have had the impetus to push their own platforms to new heights, and we’d still be staring at plain-text emails on black and white Blackberry devices.

It’s going to be a very loud fall season as the mobile providers and the FCC battle out these proposed rules.  The end result will have a huge effect, either good or bad, on how flexible and feasible your mobile Exchange platform plans will be. Competition is a good thing, but it cannot be forced on the market at the expense of profits.  There must be a way to balance these scales, and it will need to be found before Net Neutrality can be forged in the mobile marketplace.

 

 

Get back to where you once belonged (Failover Cluster version)

In honor of the re-release of the Beatles stuff all over the world (games, CD’s, maybe iTunes at some point), I took the title of today’s post from their song “Get Back” on the album Let It Be (Remastered).

I am, of course, going to tie this to something in Exchange; specifically Exchange 2007 Standby Clustering. Standby clustering refers to the theory of using a replication engine (like the native CCR or a 3rd-party system like Double-Take Availability – see disclaimer below) to place a copy of the data for the Storage Groups of the production cluster onto a secondary cluster.  Once the data is replicated, you can use the /RecoverCMS commands to recreate the production Exchange Cluster Mailbox Servers (CMS’s) on that secondary cluster.

The solution set for bringing up the Storage Groups and CMS’s on another physical cluster setup in the same or another location is fairly well established.  If a single node fails on a production cluster, other nodes take over the failed Storage Groups and work resumes in a very automated fashion.  If multiple nodes, or the entire cluster, fail you use /RecoverCMS and the associated protocols to manually get everything working on another system – so long as a copy of the data exists to work from.

The problem has traditionally been best expressed by the phrase, “And then what?”

If the original cluster failed completely, the answer was simple.  Rebuild the systems with the same node names, but prepare the systems as though they would be a new /RecoverCMS target system.  However, if you have not lost the production systems, and they’re stable enough to be used again, you would still have to reinstall them without some additional help.  The most common reasons for this kind of outage are routine testing of the failover systems and extended power failures that generators and UPS systems can’t handle.

Microsoft does offer a command set to fix this particular problem, but it is not well known or publicized.  As a matter of fact, during a recent client troubleshooting session, we had a couple or techs from Microsoft on the phone (Premier Support in this case) and they were not aware of this particular method for cluster restoration.

Once you have fixed whatever went wrong, if your production cluster is still viable (and is suitably stable for continued use), you can use a command set called /ClearLocalCMS to remove the original CMS entries from the original production cluster.  Doing so is not without risks, and you should familiarize yourself with this KB article on the subject before you try it. 

/ClearLocalCMS will remove the CMS components off the original production nodes, clean up AD, and disable the virtual computer object for the original cluster CMS.  This ensures that Exchange doesn’t accidentally address the original cluster system, even after the restore process begins.  Once the CMS is cleaned, you can go about restoration of the data using the same tools as you used to get it over to the standby cluster in the first place.

To get back to your original servers, use the /RecoverCMS command in the opposite direction (from DR back to production) and then use /ClearLocalCMS commands to re-prepare your DR cluster for use in the next emergency.

Jumping between clusters is not an automated or easy process, but it does work correctly if you follow all the steps in both directions.  This set of command suites (/RecoverCMS and /ClearLocalCMS) can allow you to get back to where you once belonged, every time.

CCR clustering is still clustering, and so is DAG

As more and more of my readers move to Exchange 2007 and 2010 from Exchange 2003 and earlier versions, I hear a lot about how using the new High Availability tools will finally free them from the yolk of clustering in Windows.  While both CCR and DAG are definite improvements over traditional shared-disk clustering, neither is a departure from clustering entirely.

We’ll be talking about the new HA stuff in Exchange 2010 (along with much more of course) in the webinar Double-Take Software and Microsoft are presenting tomorrow.  I’m the speaker for Double-Take, and Patrick Foley from Microsoft is going to be doing their portion. It’s September 9th at 11am, and you can still register for free by clicking here.

In the meantime, it is important to realize that both CCR (Continuous Cluster Replication) and DAG (Database Availability Groups) are offshoots of Windows Failover Clustering (WFC).  They both change the way WFC works, and by quite a lot, so you may never touch the underlying cluster technology, but it is still there.

CCR – as its name implies – works by allowing you to create a cluster during the installation of Exchange 2007.  This one is a bit easier to see as part of WFC, as you have to create a Failover Cluster first – specifically a Distributed Majority-Node File Share Witness Failover Cluster.  After that, when you install Exchange Server you can specify which server(s) will be the Active node(s) and which will be passive.  This creates the clustered Exchange resources for you, making the overall process of setting up clustering for Exchange a lot easier.  As this one has Cluster in the name, it’s easier to see the WFC roots.

DAG will permit you to create the cluster itself from Exchange 2010 command sets, eliminating the need to pre-create the Failover Cluster prior to getting the Exchange installation rolling.  While this makes the process even easier than in 2007, it still requires that you have two or more servers capable of running Distributed Failover Clustering.  This means that not every version of Windows 2008 is going to be suitable for DAG, but also means that – under the hood – you still need to know how Distributed Failover Clustering works to properly manage the DAG systems.

In both cases, the required level of understanding of clustering is greatly diminished from what was needed in Exchange 2003 and earlier versions.  Most of the guts of the cluster are controlled by Exchange itself, which is a double-edged sword.  On one side you have the fact that folks who don’t have a lot of cluster know-how can now set up HA solutions for Exchange.  On the other side, people who don’t have a lot of cluster know-how are facing troubleshooting clustered Exchange solutions they may not have realized were there.

Both solutions work great for Exchange.  While they don’t eliminate the need for 3rd-party products to help with overall HA (and I’m biased on this one, see disclaimer below), they do make mailbox server protection much more complete.  Just remember that you’re still running on a cluster, and arm yourself with the knowledge needed to keep it running smoothly.