Covering your assets

After last week's Presidential debate, and this week's stock market nose dive, there is a lot of doubt about the US Economy.  There is not much doubt; however, that there will be some litigation to follow after we recover a bit, and lawsuits mean the need to keep and protect email information that falls within your organization's compliance policies.  During times of legal maneuvering, your Exchange servers become targets for everything from your own legal team doing research to opposing counsel or - in more extreme cases - the government going on fact-finding missions through your corporate data systems.

Many companies have compliance policies based on regulations that govern their industries, and usually the internal policies will go well beyond these regulatory measures. Having those policies documented gives you the authority you need to take the appropriate measures when protecting your Exchange-based information; so your first step in determining what to do to cover yourself is consult those internal guidelines.  If you don't have a set of guidelines, the first thing you have to do is get it created, or create it yourself and make sure your company's legal team signs off on it!

If your policies simply detail that all data within a certain time frame must be protected off-site, you have a few great options.  Of course, there are replication/failover solutions that can give you the ability to get information back from a secondary server (see disclaimer below).  If those are outside your budget and/or logistical reach, then tape backup is your only answer.  No matter which method of getting and maintaining another copy of the data you use, be sure to be diligent about it.  Examine the data on the secondary server regularly if you are using a replication solution.  Test the integrity by running an ISInteg check on a monthly or quarterly basis - as this doesn't require you to interrupt your production data system to run this non-invasive check.  For a deeper scan of the database, you can use the ESEUtil system with various switches.  Both of these tools are included with Exchange, and are therefore both free and fully supported by Microsoft.

You can find a description of using ESEUtil here, and for ISInteg, check here.

If you're using tape backup, then perform regular test restorations and then run those utilities on the restored data!  I cannot tell you how many of my clients have come to me in a panic because they were in the middle of an emergency, never did test restores, and the data on the tapes was completely useless.  Test restore at least once per month or once per quarter, don't wait until you're in the middle of a disaster before you make sure the tapes work the way you planned.

If your corporate policies go beyond just protecting the data as a whole within certain time periods (such as "within 7 years from creation"), then you will need to look beyond just backup and/or Disaster Recovery tools.  Continuous Data Protection (CDP) solutions and Archiving solutions are on the menu for your company, and you have a lot to choose from.

CDP systems do as their name implies.  They continuously copy data to another server, but in such a way that not only do you keep all your current information; but changes, versions and information that may have been deleted from the production box.  Where a tape backup system lets you keep a daily (or semi-daily) version on the tape, and a replication solution will let you keep a real-time copy of the latest version of the data, CDP systems maintain a repository of changes.  This can be accomplished by using Exchange Journaling (described here) or by a form of replication solution either within Exchange or at the file level.  Once the information is transmitted to the repository, the internal working of the CDP system will ensure that all changes are independently tracked, so that you can restore information from any point in time that the system as a recovery point for.

Archiving solutions usually do what CDP system do, but with an added benefit for very large Exchange systems.  Archiving tools will remove information from the Exchange server based on data of last access, date or receipt, or many other factors which you control.  Any data not within the scope of the archival system is protected via the incorporated CDP solution, but when data falls into the archival rules, it is removed from the production server with a copy still remaining on the archive server.  This does help by reducing the size of your Exchange databases, but makes it absolutely critical that you provide proper Disaster Recovery protection for the archiving systems, as now that is the only place the archived data exists.

Finally, one note that many of my clients have overlooked.  Using CDP along or Disaster Recovery alone will not be a total solution for most organizations.  It is highly likely that you will need to make preparations for both recovery of different data components (CDP) and recover from the loss of the server system as a whole (Disaster Recovery).  You may be lucky enough to get away with just one or the other, but it's not very likely.

Any and all of these solutions can assist you when it comes time to either figure out your legal liabilities, or defend yourself against incoming lawsuits.  One of the worst things that can happen is to stand in front of a judge or mediator and say that you can't present the information they're demanding because - effectively - the dog ate it.  No matter how valid your excuse for not having the information may be, your company will appear to be trying to get away with something.  So, cover your assets!

Circular (logging) Logic?

Exchange logs have been a thorn in the side of server administrators forever.  Log files contain information about every transaction on the server, and can fill up a hard drive quite quickly. This can lead to the entire Exchange system grinding to a screaming halt, with no recourse but to copy the logs to another drive, reassign that drive letter with the original log disk drive letter and restart. 

Through the years of Exchange development from 5.5 through the current 2007 systems, log files have remained a critical - if somewhat annoying - part of the overall Exchange solution set.  While the size of the individual log file has changed from 5mb to 1mb in 2007, the actual amount of data stored in the logs has stayed pretty constant (so in 2007 you have more log file taking up the same space).

Recently, indications through third-party sources have intimated that Circular Logging (CL) may be making a comeback to help deal with these log files, but at a price.  CL is the process of having Exchange begin deleting old logs as it creates new ones - based on parameters set by the administrator.  For example, Exchange might keep 100 logs, then each time it creates a new log - starting at log 101 - it deletes the oldest log on the server.  This takes care of the disk space issues, but does have an impact.

Without a full set of logs corresponding to the non-committed data, Exchange will be unable to recover from unexpected faults.  While the need to leverage these logs is rare, the lack of the correct logs in an emergency can be devastating.  When a database has been unexpectedly dismounted (such as due to an unexpected reboot), Exchange will compare the last committed transaction in the database to the transactions stored in the log files.  By doing so, Exchange can play forward any missing/incomplete transactions by using the log data to reconstruct them. With CL turned on, there is a chance that one or more of the logs required to replay the transactions will have been deleted to make room for new logs, causing the process to fail, and causing data loss.

Now, there are a few ways to help manage this problem.  Backup solutions like Data Protection Manager from Microsoft, and 3rd Party tools from various vendors, can remove transaction logs that have been properly committed to the backup media (disk, tape, etc).  This way if you have to perform a recovery operation, you can restore all required logs from the backup media and be safe. There are also many 3rd Party Exchange management tools that can store copies of the log files to allow for advanced maintenance and recovery.  However, this should not be taken as a free-pass to turn on CL.

As a sudden rush of data into the Exchange system could create a large number of logs, it would be very possible that CL would kick in and delete logs that have not yet been moved to the backup media.  That would leave you in an unrecoverable position if those logs were to be required for restoration of services. 

Circular Logging is a feature turned off by default in versions of Exchange from 2000 to present.  Given that there are tools to help mitigate log impact on disk space, and that there are very real dangers in using CL overall, it is a feature that should probably be best left turned off going forward.

Update on Wildcard certificates

Not that long ago I talked about using wildcard certificates to allow you to move OWA and ActiveSync services from one physical server to another.  Since single certificates are assigned to a single server, failing over or moving to another server would cause the clients to suddenly lose SSL connectivity, as the certificate would not match up, and ActiveSync devices cannot pop up the error about a non-secure connection. OWA can, but it can be troublesome with end-users who suddenly start seeing security warnings.

Following up on this theory of using a domain-assigned wildcard certificate, research has shown that older Windows Mobile devices (WM 5 or earlier) cannot leverage these types of certificates at all.  WM 6 and higher can leverage this technology, but earlier versions were not coded with the required information to recognize that a certificate could possibly be assigned to more than one physical server or networked device.

So, for those using WM 6, OWA and Outlook Anywhere, you can use wildcard certificates to allow for services to move from server to server as required.  For those using earlier versions of WM (or the 3G iPhone - though the jury is still out on that one), you must use server-specific certificates, and re-configure the devices' ActiveSync connection if the server itself moves. 

None of this impacts Blackberries, as they authenticate via the RIM network, and not directly to the Exchange servers.  If you're using Blackberries, wildcard certificates offer the ability for all other mobile systems (OWA, Outlook Anywhere, etc) to move with your servers in the event of a loss of a particular physical machine, while RIM will handle moving the Blackberry devices.

Long story short, if you're on WM 5 or earlier, be ready for a few support calls when you need to move the services or fail over between servers - even if you use wildcard certificates.  If your users are on the new iPhone systems, be sure to keep a close watch on Apple's forums, as new information is being discovered every day.

Exchange on DC's

Smaller shops have always been faced with a fundamental quandary when the time has come for them to move off Small Business Server and into the world of the independent version of Microsoft Exchange.  One of the biggest considerations is where to put the Domain Controllers for Active Directory.  The first thought most of my clients have is to put it on the same server as the Exchange system, but outside of SBS that's a very bad idea for a handful of important reasons.

For example, if you're ready to make the move from a single server to a cluster, you cannot install Exchange 2003 on a cluster node that is also a DC.  That is just simply not supported by Microsoft, and therefore isn't an issue up for debate.

It's much more likely that you'll be moving to a single-server, however, as if you're just moving off of SBS then the likelihood you need a cluster is reduced quite a bit.  In those cases, while you could install Exchange on the DC, it's still not a good idea.  The Exchange Best Practices Analyzer will even flag this condition as going against best practices, which should be a giant red flag for anyone installing an Exchange Server.

Here's a few reasons it isn't a good idea:

1 - Since Exchange runs many services under Local System accounts with elevated privileges, a successful attack via the Exchange system could easily expose your AD controller to outside influences.  With the potential to gain access to all domain resources if the AD controller is compromised, reducing attack surface is vital, and installing Exchange (or any app, this is not an Exchange-specific issue) will *increase* attack surface.  If you get hit on an Exchange Server that is not a DC, the attacker has access to only that server, not the entire Domain.

2 - 3rd Party integration can suffer.  Since many 3rd Party tools may expect to use Local Accounts and Groups (which wouldn't exist on a DC) or may otherwise need to control aspects of the Exchange System that may not be available or accessible on a DC, integration with Anti-Virus, Backup and Recovery solutions can suffer if the server hosts both roles.

3 - Performance impacts will be clearly visible.  Since the Exchange Server constantly uses the DC and vice-versa, placing both on one physical server can effectively double the amount of work that server must do.  Some visible effects of this are slower overall Exchange performance; longer response times for Domain commands (such as logging in, changing passwords, etc.; and much longer reboot times.

Putting the Active Directory and Exchange Server systems on one physical machine may seem like a way to save money, but in reality it will cost more in downtime and headaches than it will save on server hardware.

 

[Note: Small Business Server contains a highly modified version of Microsoft Exchange Server that is specifically designed to run on an SBS server.  It does not permit all 3rd Party applications, limits functionality and is also limited to only being able to run on SBS.  Therefore, running the SBS version of Exchange on an SBS DC is both a supported installation and a best practice. This article specifically applied to non-SBS versions of Exchange Server.]