Exchange Server is a lot like Soccer

With the World Cup in full swing, I found it interesting that Exchange Server has quite a lot in common with the world of Soccer (ok, fine, Football for everyone not in the US). This is especially true if you consider the ball to be an outgoing email message.

You start out in the center, after the outgoing mail has cleared all your internal message hygiene and routing systems. First, you have to have an open connection to the receiver’s SMTP systems, or the message can’t go anywhere. If you find that opening, you can rocket the message toward the recipient, but many things can get in the way.

You might think you’re aiming for the goal, but due to DNS issues or just TCP nuttiness, you could find your message going right out of bounds instead. When you have an eye on the goal, there’s always the chance that the recipient’s mail systems might not be properly configured to talk to yours, resulting in an off-sides call that brings everything to a screeching halt.

Let’s say that things are configured correctly and you have all the routing information down. The recipient’s guardian systems (anti-virus, anti-spam) might be tuned up high enough to block the message, sending it rocketing right back at your face.

Finally, if you manage to not get caught by their defense (because your message isn’t spam or infected, of course) and you have the right route to the recipient’s mailbox, then the message hits the goal and everyone cheers while you run screaming around the… wait… ok, I suppose the analogy might not be perfect.

On the flipside, you have all of your defenses set up to block any mal-formed messages, spam, and infected objects from ever reaching your goal. Creating a whopping offense but no defense is just as dangerous as the other way round.

Finally, if a player gets injured – or even ejected – you need to have other players pre-trained and ready to go.  Much like you’ll need to ensure you have High Availability set up in your environment in case a server gets taken out of play. 

Luckily, the messages reach goal more often than the ball does in professional soccer. But as you watch the matches, think about the analogy, and you might be surprised just how easy it becomes to explain Exchange Server methodology to a non-techie. At least it is if your listener is watching the match =)

“I need an Exchange Server,” - it’s not quite that simple.

A partner called me today, distraught that he couldn’t get a simple answer to his client’s question.  What was the question?  “What do I have to buy to build an Exchange system for 200 users.” The frustration stemmed from the conversations that my vendor friend that had with the client about things like server class, storage, and about a dozen other criteria he wanted to be sure of before quoting out the hardware and software necessary to get the job done.  From the research the client had done on their own, they believed it should be as simple as buying “a server” and “some disk” and doing the Exchange install.

This particular vendor knows I work for Double-Take Software, and that I am one of their Exchange SME’s, and so reached out to me to see if I could come up with an answer on the failover/availability portion of the question, since that was also going to be a requirement for this environment.  He was less than thrilled with my answer, which fell somewhere between “are you insane,’ and, “I’m not touching this with a 40 foot pole.”

Eventually I settled down, and started walking through all the things that were wrong here, and trying to find some way to get the questions simplified enough so that the client didn’t feel like they were being asked to… well… architect a complete Exchange solution for their environment. It was quite the process, let me tell you.

In order to help those who have never worked with Exchange Server before, let me spell out a few of the things you will need to know before you move forward.  For those who work with it every day, I hope you can use this as a plain English explanation of what’s involved so that you can explain it to those who may need to sign the checks, but don’t know the tech – should you ever run into that problem.

First things first. “Exchange Server” isn’t a server.  It’s a platform of software tools that all work together to provide email, collaboration services and shared data services to end users.  That means that you can’t just buy “an Exchange Server,” and in most cases it’s not one server at all, but two or more acting in concert.  To say “I need an Exchange Server for 200 people” is like asking a car dealer for “a car that can go 10 miles each day.”  Do you want a car that can hold 2 people, or 4?  Are you planning on only moving people or do you have pets?  How about cargo space, are you planning on carrying groceries for those 10 miles, or hauling bricks?  Oh, and do you want a car that can go 5 days between trips to the gas station, or do you want an SUV that’s going to suck up the whole tank during each 10 mile trip?

To beat that analogy up even further, Exchange is only the engine, not the car.  Much like other types of engines, it works better in some car chassis than others.

So, what do you plan to do for your 200 users?  Do they use a lot of email all day long, or is it just light email a few times a week?  What do they email, just regular text emails, or are they shipping multimedia presentations back and forth to each other and outside the firm? Are they all in one office, or spread out across multiple offices all over the place?  If they are spread out (or if they access mail from home, etc.) what level of security do you need to provide?  Are they doctors (HIPPA) or officers of a public company (SOX)?

Now let’s talk about some other things that may go on in your environment.  Do you need to be able to schedule conference rooms, company vehicles or other shared resources?  How about generic email boxes, like for users to request information, open support tickets or contact a salesperson or media contact?  In other words, is it 200 users that need 300 mailboxes all told, or is it 200 mailboxes?

How big were you planning on letting these mailboxes get?  Left to their own devices, users can accumulate gigabyte after gigabyte of data.  I know some sales folks who routinely have about 6-8GB of data in their email boxes.  And speaking of all that email, how long do you plan on keeping it?  Do you need to archive older mail (which you should do) or do you need to permanently delete things over a certain again (which you may need to do)?

As you can see, things are quite a lot more complicated than “get me an Exchange Server for 200 people,” and we’re not done yet.

Let’s talk Exchange Ecosystems.  That’s Microsoft’s term for all the servers and services that surround the Exchange Server in most environments.  Don’t think you need that?  You may want to think again.

Anti-Spam, Anti-Virus and other message hygiene tasks could be done on the Exchange Mailbox server, but typically for systems over about 100 users, they’re done by independent servers well before the email reaches the mailbox.  That means either creating servers dedicated to those jobs, or – more likely at this size – leasing services from a 3rd-Party provider of message hygiene solutions.  If so, do you want to host an appliance at your site, or outsource everything?

Speaking of outsourcing, 200 users is still well within the range of Exchange Hosted Services or other forms of Exchange Server Hosting.  Then you can just outsource everything and tell the vendor what features you want.  Can you do that in your industry and with the regulations/company policies you’ve got in place?

People rarely stand still for long.  I’m writing this blog posting on a train heading to Boston, but I have full access to my entire Exchange mailbox and other corporate resources through Outlook Web Access, Outlook Anywhere via my air card, and of course my Blackberry device.  Do your 200 users all need to be able to have that kind of access? If not, how many do and how many do not?  Do you need to configure Outlook Anywhere (the components of Exchange that enable many mobile devices like Windows Mobile and iPhones)?  Do your mobile folks like Blackberries instead?  If so, do you want to host your own Blackberry Enterprise Server, or outsource that too?

While we’re on the subject of people talking to each other, do your users need to share more than email?  Do they routinely cooperate on documents, lists, wikis and other things too?

At this point, you may be ready to throw your hands up, throw the idea of creating an Exchange Server out the window, and go home.  Don’t give up yet, because here’s where we tie it all together:

Exchange Server 2010 is the currently shipping version of the Exchange Server Messaging and Communications Platform.  It contains the ability to set up mailboxes, message hygiene and routing for email, communication with mobile devices and connections to other tools.  It’s available in two “flavors” – Standard and Enterprise.  With 200 users, you’re almost definitely looking at the Enterprise version, which is good, because it can help a lot with the high availability questions.  Most clients will also need additional availability solutions (see disclaimer below, I’m biased) and backup solutions as well.

Also, you’ll need Client Access Licenses for each user/mailbox.  There are two types, Standard and Enterprise.  Enterprise CAL’s allow you to do things like grant mobile access and get archiving and cross-mailbox search, so you will most likely need those for at least some of your users.

Next, what kind of hardware will you be needed?  Well, if the 200 users don’t do a lot of email work, you can get away with a lower-class machine than most.  You may even be able to squeeze everything on to one server, though putting the front-end systems (called Hub/Transport and Client Access Services) on their own server isn’t a bad idea.  You’re right on the edge of needing to split everything out anyway, and if you’re planning on growing (which hopefully you will) then you’re going to need to do it sooner rather than later.

As for what those servers will look like, there’s a lot of debate on that topic, so for this article I’ll use the official specs from Microsoft.  You can find them (and a lot more info) on the Microsoft TechNet site on Exchange Performance and Sizing.

Your front-end server should have between 4 and 12 cores, which you can get by using 1 to 3 quad-core processors, which are the standard these days.  The Mailbox server needs more horsepower, so you’re looking for from 4 to 24 cores there.  RAM-wise, front-end servers should be getting about 4GB-8GB each.  At this level you’ll probably only have one front-end server though.  Mailbox servers need 4GB to start, and then need 3MB/mailbox if your users are light email users.  If most of your users are heave emailers, with lots of attachments, huge contact lists and tons of calendar appointments, you should be looking more toward 30MB per user mailbox.

Disk space is another concern.  Microsoft has made Exchange 2010 much more efficient with disk than its predecessors, and so you can use a lot of different types of disk options.  You will need to be able to hold all the information not being archived or deleted, so think in terms of 200% of what you’re allowing your users.  So if each user can store up to 3GB of data in their mailbox, you should be allocating 1.2TB (3GB x 200 users x2) of storage space for them.  This will allow you to allocate space for Exchange’s transaction logs (internal log files used to track what Exchange is writing to the databases), the databases themselves, and the necessary queuing systems that shuttle mail back and forth, with a little room to grow.  These days you can get that with just internal disk, but many clients choose to go with external arrays so they can scale over time.  Definitely check the TechNet website for recommendations on what RAID to use and other disk settings; as this is just how much disk you need, not how you should create or allocate it.

Exchange 2010 also comes with native archiving and compliance search features built in.  This will, of course, require more hardware, so have a look at the overview of these tools on the TechNet site and plan accordingly.  Keep in mind that if you use native or 3rd-Party backup/availability tools, you’ll also need another server and appropriate disk space to hold the backup copies of the data.

External to the Exchange Server itself, but just as important, are the Ecosystem servers.  SharePoint Server allows your teams to share documents, calendars and projects easier than the native tools in Exchange Server could alone.  SharePoint is a whole other server platform, but one that you should look into if your users tend to share stuff on a daily basis.

Blackberry and Good Technologies are two common tools that work side-by-side with the native Outlook Anywhere (formerly known as ActiveSync) systems in Exchange.  If you have Enterprise CAL’s, your users can access email via Outlook, Outlook Web Access, or a host of mobile devices like iPhones just by using Exchange Server.  If you want to add extra security and sync features, then you use Good.  If you want your users to have access to the popular Blackberry mobile devices – and centrally control and manage them, then you’ll need Blackberry Enterprise Server (BES).  Good can often by installed to the Exchange Mailbox Server, but it’s a better idea to give it its own server so it doesn’t bog down the mailboxes.  BES can either be provisioned on its own server at your site, or you can lease space on a hosted BES service provider’s server.  If you host your own, then of course you will need a server to put it on.

Summing up, we’ve gone from “I need an Exchange Server for 200 people,” to, “I need several servers of different sizes to host all the components of Exchange Server for my organization.”  If you have never touched Exchange Server before, this can be bewildering, frustrating and expensive – especailly seeing as you’ll probably goof up at least once (we all do).  The good news is that there are two options if you just can’t wrap your brain around all of this.

One is to get help.  There are a huge number of service providers who can assist you in planning, procuring and implementing all these moving parts.  Many can also train your team to maintain what they built, and to build out new stuff when needed.  If you plan on hosting everything yourself, this is the much better course of action to take, as going it alone quickly gets more expensive as you fix mix-ups than the consulting costs would have been.

Alternately, you can talk to Microsoft and their partners about having someone else host and administer your Exchange solution.  Hosted Exchange Services is the product offered directly by Microsoft, and many partners also host Exchange services on their own.  Take a look an article or two before this one in this blog for some details on that.  Up to 500 users/mailboxes, it is quite suitable to go the hosting route, and even up to 1000 users it can make a lot of sense.  It’s still Exchange Server, you still get Outlook, Outlook Web Access and Outlook Anywhere – and you may be able to get Good and BES too.  You’re trading management and hardware costs for a monthly cost per user and/or per amount of storage space they use.

So, what does it boil down to?  The short answer is that there is no valid short answer here.  You need to take a look at your users and your datacenter facility.  You need to know what they want to do with their messaging systems before you can even begin to install Exchange Server components.  Exchange Server is no longer something you can just buy “off the rack” and toss into your environment.  Plan, implement and manage it correctly, or it will be nothing but a nightmare until you eventually do the planning and rebuild it right – which always costs a lot more.

Official versus 3rd-Party Exchange Hosting

Clients have been asking me lately about if Exchange Hosted Services (EHS) is a generic term that anyone who hosts Exchange 2007 and 2010 servers can use, or if it is specific to Microsoft?  The answer is that it could be either, but for the most part the term Exchange Hosted Services – spelled out as such – is a Microsoft product that they sell, host and maintain.

EHS is a relatively new product line from Microsoft Online Services (MOS), which has been offering workspace hosting for the rest of the Office platform for a while. You can store and share Word, Excel and other documents online and control them either personally or via a company controlled platform.  You can also rent Dynamics components and other products through the MOS program, but email was something that didn’t immediately become available.

Last year, Microsoft rolled out the ability to create an Exchange infrastructure, hosted in the cloud and based on Exchange 2010 technology.  This meant 25GB mailboxes, multiple-site resiliency for server failure events and the ability to allow centralized administrative control via the MOS consoles.  The concept has been around for quite a while, of course, but now there is a significant difference.

While any Microsoft Partner in the hosting business can host Exchange servers along with anything else, they cannot offer EHS by that name.  EHS is entirely controlled by Microsoft directly.  They own the datacenter space, control the servers, client management, billing and everything else.  Since Microsoft owns the overall experience, they have a very tight control over what features are offered, but also provide direct support for each of those features. So the trade-off is that you may get more feature sets from a 3rd-Party vendor, but not direct support via Microsoft Support Services.

3rd-Party partners cannot call their solution “Exchange Hosted Services,” at least not in those exact words.  Many partners will call their systems Hosted Exchange platforms or something along that vein, in order to differentiate.  You will probably be able to get the same level of service and redundancy, but be sure to always read the fine print.

When in doubt, ask the sales rep if the systems are hosted in Microsoft Datacenters or in independent, 3rd-Party datacenter facilities.  Only EHS is hosted in Microsoft Datacenters, and so this is an easy way to determine if you’re dealing with Microsoft’s EHS or a 3rd-Party.

Addendum:

A mea culpa is in order.  This article is not meant to give the impression that EHS is superior to 3rd-Party Hosted Exchange Servers.  Each has significant benefits and also at least some limitations, and you should explore all options before deciding which you will use for your email solutions provider.  For example, EHS is supported and managed by Microsoft, but does not give you any direct control over your Exchange Server, as you are instead leasing Exchange Services on Microsoft’s servers.

I’m a strong believer in the fact that no single solution can provide the right fit for every organization.  3rd-Party vendors (including the one I work for – see the disclaimer at the end of the blog) provide strong solutions with multiple options and feature sets.  Many also work in conjunction with Microsoft’s own products and services, so you might not have to give up one to use the other.

I’m glad this blog is part of your investigations, and I’m glad I can be part of your search for a solutions provider that includes multiple sources for reviews, features and functionality.  Always do your homework; only then can you make the best choice based on the needs of your company and the features of the various providers.

Don’t uninstall to fix Exchange!

Exchange Server does, occasionally, act oddly.  Alright, it can do that a lot, but hopefully you won’t run into the phenomenon.  If you should see strange behavior from your Exchange Server, the first thing to remember is that it is not a truly stand-alone application, and can’t be treated as such.  When problems happen with many software packages, the default fix-it is to uninstall the applications, reboot and reinstall.  While that may work with many apps on the Windows Server platform, doing that for Exchange Server will lead to rather disastrous results.

When Exchange Server installs, one of the first things it does is start building structures in Active Directory.  If you have no Exchange Servers at all, this includes extending the Schema and creating an Organization for Exchange and the related security constructs in both the Forest Root domain and the domain you’re installing into.  The thing many folks don’t realize until it is too late, is that much of this mucking about in Active Directory is undone when you uninstall an Exchange Server, which could be a problem if you plan on just fixing something and moving on.

When you uninstall Exchange Server, the uninstallation protocol will remove all objects in AD that were assigned to that server, which could lead to unexpected end results for users, Public Folders and other objects that don’t exist elsewhere in your organization.  This means that an attempt to uninstall to fix a somewhat minor issue could result in an absolute disaster for your Exchange environment overall that cannot be fixed by simply failing over.  The Organization and Forest Root updates will remain, but most of what you did in terms of configuration for the specific Exchange Server will be wiped out.

So, what to do in order to avoid this?  First, hit http://technet.microsoft.com and http://support.doubletake.com to see if there are articles on how to fix the problem you’re seeing.  These documents generally detail much easier fixes for common Exchange problems, and not-so-common ones as well.  Correcting the issues with Exchange still in place is the best route to take when it’s available.

If you do have to reinstall Exchange, your best bet is to shut down the malfunctioning server, build a new one with the same name, and use the disasterrecovery or recoverCMS switches with the Exchange Server installer tools.  This will transfer the settings over to the new machine, instead of removing everything permanently from AD.  Nearly all configuration for Exchange is stored in AD, so using these switches during the installation will allow you to get back up and running quickly, ready to restore the mailbox and folder data from an independent backup.  More information on these switches, the exact syntax for each version of Exchange Server and guidelines for use can be found in various places. http://support.microsoft.com, http://www.msexchange.org and http://technet.microsoft.com are three great resources.

As with most other topics, Exchange acts differently from most other applications when it comes to reinstallation. Always check the support sites before you take any drastic measures, and remember that Exchange is a totally AD-integrated application, so you’re never working on “just the Exchange Server itself.”

Bury the Berries? Not yet.

Research in Motion (RIM) changed the game when they introduced their Blackberry devices to provide mobile email solutions for Exchange and other email platforms.  Over the last few years, many other companies have stepped up to produce handheld messaging devices, mostly centered around mobile phones and carriers.  There are so many good competitors these days, that some companies are considering if they want to ditch the Blackberry Enterprise Server (BES) platform in exchange for… well… Exchange.

Exchange 2003 and up allowed for Exchange Active Sync, a series of mobile technologies that allow Exchange servers to communicate directly with mobile devices using push synchronization to deliver email, contacts, calendar items and other information.  Windows Mobile phones are natively equipped with the Active Sync technology; and the iPhone, PalmOS and WebOS, and various versions of software for Android phones have also licensed the Active Sync technology for those platforms. This means that these devices can communicate directly with an Exchange server and send/receive mail, sync contacts and tasks, and answer calendar requests without the need for an intervening server platform like BES.

However, there are two key reasons that I am (at least for now) sticking with my Blackberry device.  First, there is battery life. Talk time is a pretty standard 4-6 hours no matter what OS runs on the phone you’re holding.  But standby/messaging time is another story all together.  With push email active, I have yet to see a Windows Mobile phone go more than 6-8 hours without needing a charge.  If I turn off Bluetooth, push email and just about everything else that uses a transmitter, I can stretch that number to 12-14 hours, but then what’s the point of carrying the device?  Blackberry devices routinely go 24-36 hours of messaging, even with Bluetooth active (but not transmitting).  Android devices can go a bit longer than Windows Mobile, as do iPhones and WebOS systems. They typically run the middle-of-the-road between those two extremes.

Secondly, there is security.  BES allows for a remote device to be wiped natively.  Windows Mobile devices can also be remote-wiped if you’re on Exchange 2007 and up, but iPhone and other OS devices must use 3rd-party software to accomplish this goal.  Remote-wiping is used to provide security if a device is lost or stolen. Once activated, all personal/corporate data on the phone is deleted, rendering it both useless and safe.  Good Technologies has reinvented itself over the last few years as a great provider of remote-wipe and encryption systems for Android and iPhone platforms, but it is one more thing you have to manage in an environment that (based on who would use it) already has a BES server running.

Based on these two factors, I cannot envision organizations ditching BES in favor of an Active Sync solution set exclusively.  Battery life makes these devices difficult for mobile personnel to use successfully, and the need for additional security systems only adds to that concern. For non-mobile users, battery life isn’t as much of an issue (as they’re going to be near a power outlet in many cases).  Security, though, is still a huge concern.  Unless they are on Windows Mobile devices exclusively (no iPhones, no Droids, etc), they’re going to need additional server solutions to be safe.

My prediction is that BES will stay in the Enterprise space for quite some time.  We may see Windows Mobile, Android an WebOS take over more of the marketplace, but until these two issues are significantly improved, they will remain the minority for corporate mobile messaging.

Manage your migrations to minimize co-existence

Exchange 2010 is beginning to get traction out in the real-world, which is by no means a bad thing.  As folks begin to upgrade, though, there is a chance that you’ll end up with extended co-existence between different software versions and packages, and that can make life a living hell for technical staff. Multiple versions of multiple platforms can be difficult to manage at best, and if that management must continue for months (or longer) you could be setting yourself up to fail right from the start.

Co-existence of platforms is a necessary thing. Especially in larger organizations, the likelihood that you will be able to move all your users and systems from one software package to another in a weekend is slim at best. So, no matter if you are moving from a totally foreign system like Notes, or just upgrading between 2003 and 2010 on the Exchange platform, you will almost definitely need to have both systems running for a period of time as you move users and 3rd-Party tools.  The key is to keep that time period as short as you can, and here’s why.

When multiple versions of a messaging platform (or multiple platforms) you must make sure you’re patching both independently, staying up to date on multiple security threats and dealing with end users who have to have changes made across both system sets.  Since it is highly unlikely that you will have the benefit of extra staff during the migration, that means that the existing staff suddenly find themselves doing twice the work.

If the same number of people are forced to do twice the work, things fall through the cracks.  Patches get applied incorrectly or not at all, threats are left unaddressed and shortcuts abound.  It’s easier to open a hole in the firewall than try to deal with two sets of rules to manage connectivity. It’s easier to only focus on news and updates for the newer systems than to try to keep up with the influx of information for both platforms. The upshot of this is; that the longer you have both systems co-existing, the more opportunity you’ll find for something to break on one platform or the other.  This endangers both systems, and possibly everything else in your environment.

So, when planning for your migrations, work as much as you can to ensure that you do not require extended periods of co-existence if they can be avoided.  Sometimes you will need to have extended migration timelines – it’s unavoidable – but wherever you can do migrations quickly, you should.

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

New Poll: Exchange Versions

Net Neutrality and EAS

Mobile messaging and collaboration makes for a big, bright world in business today. What was once the domain of a single service provider (Research in Motion - RIM) has evolved into a robust set of platforms to convey email, appointments and contacts from one device to another.  Blackberries, Windows Mobile devices, iPhones, Android phones and so many other systems can communicate either directly or through a proxy to a Microsoft Exchange platform.  This unleashes the workforce and allows for your people to be where they need to be in order to work, instead of where they have to be just to talk to each other through the email system.

Net Neutrality is the idea that no matter what the network provider offers in terms of services and software, you should be able to use the devices of your choice and the platforms of your choice on those networks.  It’s a great theory, but putting it into practice is causing some issues along the way.  The FCC set forth a set of basic rules that they wanted carriers to follow, and in the greater sphere of comments, they were well received.  They recently added in two more proposed rules that directly impact cellular networks (digital broadband) and the services that run across it – including Exchange Active Sync (EAS).

You can get a rundown of the entire proposed rule set in this article, but the two that directly impact EAS most are:

5. Broadband providers cannot block or degrade lawful traffic over their networks, favor certain content or applications over others and cannot "disfavor an Internet service just because it competes with a similar service offered by that broadband provider."

On the surface, this looks like a standard anti-competitive rule.  In reality, however, many service providers in the cellular world are viciously blocking competing technologies, and their claim is that forcing neutrality will destroy their business.  EAS is a great example of this phenomenon, as not that long ago many providers didn’t allow that traffic on their mobile networks.  Mostly, this was due to the fact that they wanted to pus their own version of enterprise email synchronization (such as Sprint’s ill-fated attempt on the earlier Palm Treo devices).  Eventually, the need to allow this traffic or lose business to other devices and networks overrode the desire to use and sell their own platform, but that took a great deal of time, and lead to quite a bit of bad press and back-end attempts to circumvent the blocks.  By forcing mobile providers to allow all valid and legal traffic, the atmosphere for open communication standards will grow and more people will be able to take advantage of more technologies.

6.Broadband providers must be transparent about the service they are providing and how they are running their networks.

Proprietary networks are nothing new, but trying to create an EAS client for a phone on a network that actively blocks your ability to figure out how it sends and receives data makes this close to impossible.  Some providers have blocked all traffic they do not wish to have on their networks by simply making it very difficult – or nearly impossible – to figure out how a 3rd-party tool can possibly communicate with it.

I’m of two minds on these proposed rules.  On one side, EAS and other technologies require open, transparent communications platforms to work. Exchange can communicate with a whole world of different vendors’ mobile applications, but only if those apps can talk to the Exchange Server.  On the other side, competition drives better software and platforms.  If it wasn’t for all the things you can only do (or could only do) on an iPhone, RIM and Google would never have had the impetus to push their own platforms to new heights, and we’d still be staring at plain-text emails on black and white Blackberry devices.

It’s going to be a very loud fall season as the mobile providers and the FCC battle out these proposed rules.  The end result will have a huge effect, either good or bad, on how flexible and feasible your mobile Exchange platform plans will be. Competition is a good thing, but it cannot be forced on the market at the expense of profits.  There must be a way to balance these scales, and it will need to be found before Net Neutrality can be forged in the mobile marketplace.

 

 

Big Brother is Watc…[Recalled]

The big kerfuffle over Amazon’s Kindle Team removing several books without warning from users’ devices (story and link at the end of the blog) has lead me to think about the various time clients have used the Recall Message feature. Typically, they have found that it behaved just slightly better than opening their office door and screaming “NOBODY READ THAT!”

There are a lot of great reasons to use Recall Message in Outlook and Exchange.  Perhaps something was sent before it was finished, accidentally clicking Send without typing in that last bit of information.  Maybe you meant to add an attachment but did not – in which case may I suggest some of the great tools at Sperry Software.  You may also have accidentally hit Reply All instead of Reply, or added in a group email list via AutoComplete instead of the one person you wanted to send email to.  All of these are legitimate reasons to really want to get the email out of everyone’s Inbox.

Years ago, Microsoft attempted to build the functionality of yanking back an errant email on command.  It required (and still requires) that you be using Outlook in Exchange Mode, and a Microsoft Exchange Server as your email platform.  When first introduced, this worked great.  There was no such thing as a preview pane, you were always using an Exchange Server and MAPI for communication, and folks were probably online and connected since there was no Cached Mode.

The issues arise when you look at what is required for a Recall attempt to succeed:

• The user must be online and connected to the Exchange Server
• They must not be in Cached or Offline Modes
• The message must be unread in the recipient’s Inbox (including unread by the Preview Pane)
• The recipient cannot have moved the message to another folder (unread or otherwise)
• They must be using MAPI. Even POP3 or IMAP on an Exchange Server won’t count.

So, as you can see, since the majority of email users these days do not meet one or many of these requirements, Recall Message will fail for the majority of your users.  What they will see is the original email (unchanged) and a notice saying you would like to recall the original email.  This can often be just as embarrassing as the original gaff, as it acts as a glaring exclamation point that someone made a goof.

One more note, Recall Message will not impact the copy of the information on a Blackberry or other 3rd-party Smartphones and mobile devices.  So even if the user’s desktop has all the requirements met, they’ll still have a copy of the message (and the recall request) sitting on their mobile device.

In the days of Exchange and Outlook with MAPI connections being the only conduit to email, Recall Message was a great way to manage errant information.  Today, with so many ways to connect to any mail system, including Exchange Server, this feature has lost its usefulness and created a false sense of security.  Remember, if there is a chance that someone’s using a non-Outlook client or not meeting all of the requirements above, you will not recall the mail.  An ounce of prevention is your only cure these days.

As for the Amazon gaff, for those who did not see the news:  Amazon recently pulled all the books by a major author and deleted them off all Kindle devices that held them.  This was due to someone posting the books on the Kindle store, without the publishing rights needed to earn money from them.  Once Amazon realized what was going on, they of course stopped selling all the books from that vendor.  Unfortunately they also removed all the digital copies that were stored on users’ Kindle devices without mentioning it to the Kindle owners.  They did refund everyone’s money, and they did send out an explanatory email, but not until several days later after the blogosphere detonated with Amazon hate postings.

Frankly, Amazon’s user agreement does say that they reserve the right to do exactly what they did.  I can’t fault them for heading off trouble (in the form of a copyright litigation) at the pass. However, it could have been handled a lot better, with more information supplied immediately to end users about what was going on and why.  Amazon has promised to make this process much more transparent so that next time there is no confusion.

The reason there was such an outcry about the process this time (as it has happened in the past, quietly and without fanfare or hate mail)?  The author in question was George Orwell, and the books were the likes of 1984 and Animal Farm.  Both of these books deal with subjects like retroactively removing information that didn’t meet the criteria of an overarching body of government or law.

From Dictionary.com:

Irony: an outcome of events contrary to what was, or might have been, expected.

Diversity takes another one for the team

Belts are tightening everywhere – and though it appears that budget freezes might be thawing out slowly over the next 6 months or so, there are cost-cutting measures all over that are still being brought to bear.  One of the more interesting ones (that wasn’t in my company, that is) was the report that Microsoft was going to be ending its policy of letting employees expense fees for any Smartphone besides Windows Mobile devices.  So no more Pres, no more BlackBerries and no more iPhones.

You can read a blurb about it in this Network World article

These are tough times, and I will not fault any company for tightening the expense account reins.  Standardizing on one mobile technology is a fairly common practice for most companies, and not paying for other technologies is also normal in the business world. So, I’m not quite upset that MSFT has decided to only reimburse for WinMo devices.  Even so, part of me is a bit miffed on reading this one.

MSFT isn’t like other companies – in so many ways.  Specifically here, they make a product line (Exchange Server) that interacts and interoperates with iPhones, BlackBerries and other mobile devices either directly or indirectly.  The diversity of devices that their previous policies made possible was a boon to developers internally who needed to make sure that the latest changes to the Exchange line allowed the more popular devices to continue working correctly.  It’s much easier to see that Exchange Active Sync broke when the guy next to your cube suddenly finds his iPhone stops getting email.

Since real-world businesses are adopting devices like the iPhone and new new Palm Pre (at least if you listen to Palm they are), knowledge of how those devices interop with Exchange Server is critical.  MSFT would be technologically better off having folks using those devices every day right there in Redmond.  The easiest way to ensure that happens is to make them fiscally available to employees.

Granted, when push comes to shove, cost cutting is cost cutting.  Sometimes it’s just necessary to hold back on expenses in order to keep everything else running smooth.  So I do not fault MSFT for making this policy and putting it into effect.  I think that it may have a larger impact than just bottom-line savings, but it is a responsible, viable method for holding back costs while not resorting to more layoffs.

As we put notch after notch in our belts, and as we shakily get back to our financial feet again, there will be more of these cost cutting measures.  A lot of them will be much worse than this, but most will not have such a direct impact on the development of a product platform.  So, hopefully the Exchange team will continue to have Pres and iPhones and BlackBerries running around, even if the company doesn’t officially subsidize them anymore.

Exchange gets tired.

I’m not talking about getting old here, but rather about exhaustion, which can happen in even newer versions of Exchange Server. Checkpoint exhaustion is a function of the ESE database system set used in Exchange, and designed to stop the system from corrupting itself if too many database transactions get held up in a log file by anything.

The normal sequence of events is well known for most relational databases.  Information added to a database is sent to a so-called Lazy Page Writer to minimize disk access by aggregating I/O operations.  When this process happens, a write-ahead logging system tracks the operations, so that if they need to be replayed, they can be safely re-generated if the Lazy Writer data is lost.  As the transactions are committed to the database itself, the log systems note this by checkpointing those transactions, essentially noting that they’ve been safely committed to the database.  For most versions of Exchange Server this happens every 20MB or so (that’s 20MB of log data, not database writes).

When things do not go according to schedule, the logging systems can recover data that was in the Lazy Writer but not yet committed to the database with emergency systems.  Some, like the normal startup process for an Exchange database, are done automatically.  Others, like ESEUtil rescue operations are done manually, and hopefully avoided entirely.  But, the point at which the logs logjam up so much that the system forces emergency procedures can be more difficult to explain.

The short answer is that Exchange can log up to 1,008 transactions before Checkpoint Exhaustion hits.  This is the point where Exchange can no longer just keep writing to the logs and still know that those transactions can be safely recovered even if the Lazy Writer doesn’t have the transactional data anymore.  If the system sees that the logs are tracking more than 1,008 transactions that have not yet been checkpointed, it will forcibly dismount that database – in this case Storage Group and related Stores.

Now, when does Checkpoint Exhaustion happen?  That’s a bit trickier to explain, but there are a few definite times you might see it.

1 – The system is horrifically under-powered in terms of how fast it can get data out of the Lazy Writer and onto disk.  This is rare, but can happen if there is a failing RAID controller or other disk-device.

2 – Something has placed a lock on log files or databases.  In this case, the offending application is stopping Exchange from writing to either a log or database, or at the very least stopping the checkpointing process from happening regularly. Since this means that Exchange cannot successfully checkpoint the database, logs continue to build and build without being “caught up.” This is also not common, but can occur with management tools that go awry.

3 – Backup systems don’t work as expected.  With most Exchange-aware backup tools, the backup system prevents Exchange from checkpointing during the backup operation.  Normally, this is a good thing.  Checkpoint prevention means that no new data will be officially committed while the backup is in the process of…well…backing up.  This means that a point-in-time tape or disk backup can be transactionally intact, even if it cannot maintain strict write integrity native to the backup solution.  Since the Lazy Writer and the log files both have copies of the information, as long as the block is lifted at the end of the backup, all is safe and all is well.

However, if the backup barfs and never lifts the lock, you could have a major issue.  Since no new checkpoints can be set, unchecked log data starts piling up, and eventually you’ll hit the 1,008 transaction limit and see a forced-dismount happen on that Storage Group.  Luckily, restarting the Exchange services nearly always clears this problem, and since the logs still have a copy of all non-checked data, you’re not going to lose anything but time.

Checkpoint Exhaustion was built to solve a particularly thorny problem.  The earlier versions of Exchange Server that are still supported had a hard-coded limit to 1,024 transactions before the database would fault (not just go offline).  That meant that if the Checkpoint Depth hit that level, you could lose data.  So the exhaustion systems allow the database to safely shut down without losing track of any data – which is a not-so-great but definitely better than data loss type of solution set.

So, to avoid exhaustion, there are several things you can do.

1 – Make sure you’re not overloading your Exchange Servers.  Stay within best-practice guidelines for Microsoft Exchange versions, which can be found on TechNET. and be sure you periodically check for hardware issues.

2 – Try to minimize the use of applications that could lock the Exchange Server file sets.  3rd-party non-Exchange indexing for example

3 – Make sure your backup runs are completing properly. One bad run that never got caught can snowball into an exhaustion situation easily.  If you do have a bad backup run, declare a maintenance window of 20-30 minutes and restart the Exchange services, or even gracefully bounce the box.

 

Don’t forget, if you have an Exchange related topic you’d like to ask about, drop an email to miketalonnyc@gmail.com .  Next update should be on Monday or Tuesday of next week, back to my usual schedule. 

A special hello to Larry Meister at Mercury Networks in Rochester, NY, thanks for inviting me to speak at the White Hat Security Day conference.  It was a great event, as usual!  If you’re looking for a partner for your IT needs in Western NY State, Larry is your guy.

Outlook, can you hear me? Can you feel me near you?

Might be showing my age and/or taste in music with that particular title (and if you’re totally confused by it, check out This YouTube video), but I think that it’s a great way to describe an annoyance that can happen if you’re using versions of Outlook before Outlook 2007.  Since a large portion of the users of Office are on the 2003 version (and many even earlier than that), resolution to a new server in the event of a disaster recovery event is a subject that is just as confusing as the famous rock opera I’m making use of in my title today.

When Outlook 2007 was introduced to the world with Exchange 2007, a lot was made (and rightfully so) of the new AutoDiscover features that this platform brought into the Enterprise Email marketplace.  The long and short of the AutoDiscover solution set is this:

When an Outlook 2007 client cannot find its home server – either because it is a brand new install of Outlook or because the home server has moved or been replaced – the Exchange 2007 AutoDiscover system can help Outlook 2007 find its home.  If the Outlook client can see an Exchange Server (or be directed to one by Active Directory), the Server can tell Outlook where the mailbox information for the user’s profile exists, and direct Outlook to connect to the appropriate CAS or Mailbox systems and get connected.  All the user/Admin has to do is tell Outlook the user’s email address and password, and AD with Exchange 2007 will handle it from there.  So if you’re installing Outlook for the first time, you don’t have to manually configure the Profile anymore – a great boon to Admins everywhere.

This system also kicks in if you perform Database Portability during a disaster, and have replicated the database with SCR; or have used a 3rd party disaster recovery/availability solution (see disclaimer below for all my bias information on that one =).  Once the Exchange system is responding again, AD can ferry the Outlook 2007 client to the new home for that mailbox, requiring only that the end-user close and reopen Outlook to complete the process.

However, what many folks do not realize right off the bat is that this solution set is ONLY available if you have both Outlook 2007 and Exchange 2007 as your messaging platform.  All users who need to take advantage of AutoDiscover must be using that combination of tools, and no other.  As you might expect, POP3 and IMAP systems do not AutoDiscover, but the majority of my clients were unaware that Outlook 2003 and earlier also cannot take advantage of this system, even if you have upgraded to Exchange 2007 as the messaging platform of choice.  It’s also worth noting that AutoDiscover doesn’t officially work in Exchange 2003 – no matter what Outlook version you are on.  Before I get blasted by mail on this one, I know some folks have sometimes seen it to work on Outlook 2003 with Exchange 2007, but it bombs more than it works, and officially it’s not supported.  For proof, I direct you to this article by the MS Exchange Team.

Since the code to perform AutoDiscover wasn’t in Outlook 2003, users on that client software will not be able to dynamically re-link to the new Exchange server unless the original mailbox server is still responding.  If it is, then Outlook can find the new server via the original server and re-home itself.  If not, Outlook must be manually re-directed to the new server.

Of course, there are ways around this.  You could update DNS to re-direct anyone calling for “Server 1” to the IP Address of “Server 2” – effectively re-routing all client software including POP and IMAP.  Outlook 2003 will still need to be re-profiled unless you take over the Service Principle Name (SPN) of “Server 1” on “Server 2,” but it will be a smoother transition.  Using a 3rd party tool (see disclaimer below) you may have the option of automated DNS and SPN updates, which will allow even legacy Outlook clients to jump to the new server with no more intervention than is required on Outlook 2007 with Exchange 2007 – even if you’re whole system in on the 2003 versions of those software platforms or earlier.

So, you are not without lots of options if you have any legacy servers and/or clients – or non MAPI clients – in your environment. You just need to be aware that the Exchange 2007/Outlook 2007 solutions for AutoDiscover services are not backward compatible, and plan accordingly.  Right now it looks like Exchange 2010 will have AutoDiscover that is backward compatible to Outlook 2007 only, so this soon-to-be-released platform is not going to solve this particular problem unless you’re planning on upgrading everything else in the environment to at least Outlook 2007 first.

Since I like to let folks continue to discover on their own, here’s a link to the White Paper from MSFT on AutoDiscover.

Finally, Update Roll-Up 8 for Exchange 2007 is out there, which can make life easier if you’re doing a fresh install of Exchange 2007 and want to get up to date with patches and fixes post SP1 quickly.  You can get it at this link.

The Dread Pirate Re-Seed – Part 2

Last week, we talked about re-seeding operations between two nodes of an Active/Passive CCR Cluster.  Since both nodes will most likely be inside the same logical network (though are not required to be in Server 2008), even unexpected re-seed operations on a CCR Cluster should be relatively painless.  Though it’s a full copy to target, it is happening over a LAN, and therefore faster and less resource-intensive than a WAN copy would be.

This week, let’s look at how the game changes when you work with Server (or Standby) Continuous Replication (SCR) when you have re-seed operations.  Just to be clear, re-seeds are not the normal method for data protection with SCR. Normally, when a log file is closed and a new prime log (usually E00 for the first Storage Group) is created, the closed log is shipped over to a target Exchange 2007 if SCR is enabled.  Once on the target, the log is held until the target reaches the number of logs specified in the log replay caching system – 50 by default.  After that, Exchange commits the logs to the database on the target, effectively providing SQL-like Log Shipping (or Continuous Data Mirroring, Asynchronous) between two Exchange 2007 Servers.

Whenever the two servers are not in communication with each other, there is the potential for operations to occur on the source that are not seen by the target, and vice versa.  Since this could easily result in a corrupted database, Exchange will check to ensure it knows what state the data on both servers is in before resuming normal SCR operations.  If the state is known, then SCR continues – transmitting all logs not yet on the target, committing all but the last 50 logs, and continuing on its way as normal.

If the state is unknown, a re-seed operation will need to occur.  More specifically if there is a gap in log file enumeration for some reason, you will require a re-seed.  You can see the reasons why re-seeds are required at the TechNet website listed here.

The one that is of most concern to Exchange Admins is that if the two servers are not in communication during a backup window, you will have to re-seed the database before normal SCR operations can continue.  When speaking of a local SCR pair this is not a big issue, as connectivity will be much more solid than WAN performance, and even if a re-seed is required, it will be relatively fast.  But across a WAN, it is more likely that you can occasionally suffer WAN outages that do not interrupt business operations.  If these outages extend past the time of the backup window, the backup tool will most likely truncate logs committed to tape without SCR being able to transmit those changes to the target Exchange Server.

Since minor network outages are a common (though hopefully not very common) issue with modern networks, the likelihood of requiring a re-seed due to this sequence of events is something that could be considered a normal part of Exchange SCR operations.  If your databases are small, then it won’t be an issue.  For larger databases, remember that a re-seed operation is a full copy of all data files to the target device from source, which could be problematic depending on your WAN throughput.

So with SCR, re-seeds do become a definite issue to contend with.  Since the occurrence of re-seed operations should be limited, you may be able to keep the systems successfully in sync without too much trouble. However, if you have larger databases or smaller WAN pipes, re-seeds can create problems for your network, especially if you use a backup tool that truncates your logs, or use circular logging for any reason.

Of course, there are alternatives to SCR for WAN protection and availability (see disclaimer below, I’m biased here), so you are not out of luck in terms of WAN operations with Exchange 2007.  Locally, CCR is a spectacular choice for nearly any sized Exchange system, with the exception of very large (over 1TB) datasets that may just take to long to re-seed even over a LAN.  Remotely, if you have properly planned for re-seeds then you will also be able to successfully utilize SCR, but if you do not – or cannot – plan for these required operations, then you’ll risk a failure in your failsafe system, which is not a great situation to be in.

In short, re-seeds are a normal part of SCR operations. They should not happen every day (or even every week) but you will most likely experience them over the course of your SCR lifetime, so plan for them now.

 

Next week is TechEd 2009!  I’ll be out in Los Angeles with the Double-Take Software crew, so please stop by and say hi. I’ll also be blogging from the event, so keep an eye on this column and follow me on Twitter at http://twitter.com/talonnyc.

Public airing of Exchange laundry...

Last week we finished up talking about what happens to Private Mail Stores during Online Maintenance (OM) on your Exchange servers.  While most people really tend to focus on what happens in those Private Stores the Public Stores are just as important, especially if you're not on Exchange 2007 yet.

For all versions of Exchange, Public Folders experience most of the same processes that their Private Mailbox cousins do, with the Exchange system scavenging items that are marked for deletion and have passed deleted item retention to start.  The system will also get rid of expired folders - a similar process to deleted mailbox systems in the Private Mailbox Stores - and perform an online defragmentation of the databases.  General maintenance and cleanup for Public Folders should be considered to be the same priority as OM on your Private Mailbox Stores, even if you're on Exchange 2007.

With Public Folders in Exchange 2000 and 2003, and even in 2007 in some circumstances, there is another vital process that goes on each day. It is not technically part of OM, but should be considered on par with the OM process.

As many people notice, the first time you boot up a newly-installed Exchange server, Outlook will complain about not being able to find "a required file" when you do a send/receive operation.  This is because - unless you're using a native Exchange 2007/Outlook2007 environment - Outlook cannot find the Offline Address Book (OAB) which doesn't get created in the Administrative Public Folders until around 5am the morning after the Exchange server goes online.  While not officially part of the OM process, the creation and subsequent updates of the OAB do happen overnight - 5am by default, just after the OM process is typically completed. 

OABgen (the process that - wait for it - generates the OAB) should be considered part of your OM systems. If you do not allow this to occur regularly, your OAB doesn't get updated and therefore your end users don't have the latest information - causing no end of frustration for you and your technical staff.  You could manually create/update the OAB using tools inside of Exchange, but the automated process is still your best bet for making sure these critical systems get updated regularly.

And one final note.  For those who believe they can get away with no Public Folders and/or PF maintenance because you are on Exchange 2007, be aware of one very important caveat.  All Outlook clients must be on Outlook 2007, and no Entourage or other MAPI clients can be used. That's because all MAPI systems before Outlook 2007 required the use of Public Folders for administrative information, like the OAB and other components.  In Outlook 2007, that's handled by Active Directory and Exchange itself, but if there is even one legacy client in your environment, you'll still need at least one PF tree, and therefore need to perform regular Online Maintenance on those Public Folder Stores.

Next week, back to regular blog topics - and speaking of which, if you have a topic you'd like me to cover, go ahead and drop me a line at MikeTalonNYC@yahoo.com

Mailbox Separation Anxiety

As we have been discussing over the last few weeks, Exchange 2000-2007 runs through quite a few operations during the Online Maintenance (OM) process.  We've talked about how databases get cleaned up, and how messages get deleted (temporarily and then permanently).  This week, let's look at Deleted Mailbox Retention (DMR) and what it can do for you.

DMR is the process that Exchange uses to remove mailboxes that have aged past their expiry date in the Private Database.  These expired mailboxes can be created by different operations within Exchange, either directly by the administrator of Exchange, or by operations in Active Directory that inadvertently cause a mailbox disconnect.  Disconnected mailboxes are simply mailboxes that not longer have an associated user or object in Active Directory. The simplest way to do this is to delete a user account in AD, which will disconnect the mailbox assigned to that user as part of the process.  However, this isn't the only operation that can cause mailboxes to become marked for deletion.  There is an odd side-effect to moving mailboxes that can happen if you jump between AD sites and don't finish fast enough (You can read about it here), and of course; if you get hit with AD corruption it could disconnect mailboxes as well.

Once a mailbox is disconnected, the first OM run that happens after the disconnect will mark the mailbox for expiry in a user-defined number of days. The default is 30 days, so if you run maintenance daily, the mailbox will stay in the database, but disconnected, for 30 days after it is disconnected from the mail-enabled object (like a user) in AD.  It also means you have to take when you do OM into account for your deleted item retention numbers.  For example, if you only do OM once per week, then you will need to add up to 7 days to your overall retention time for planning purposes.  Since the 30 day clock doesn't start ticking until the mailbox is marked during OM, you will extend the amount of time data remains in the database.  This is critical for regulatory compliance, as holding data for an extra week could put you over the line in terms of meeting your regulatory guidelines.

The other part of this OM process happens when Exchange scavenges the database for expired mailboxes.  If it finds one, and the mailbox hasn't been re-assigned to another AD object, Exchange will delete the data from the database permanently and create white-space in the database that gets cleaned up by OM to be reused by Exchange.

So, here again, it becomes vital that OM is permitted to run on a regular basis.  If not, deleted mailboxes will not be removed from the database unless you manually launch the cleanup wizards.  This is more than just a regulatory problem, as without removing the old information and freeing up white space, your databases will grow much more rapidly than if you are regularly cleaning up your mail stores.

Next week... Public Folder operations during Online Maintenance, so stay tuned!

Update now - no I mean it, NOW

Though I'm known for keeping a level head during these kinds of scenarios, there is an emerging vulnerability in both Exchange Server and Internet Explorer that you'll have to address immediately to keep your enterprise safe.  I wish I was kidding, but no, it's true.

The long and short of it is that specially crafted messages could be sent to an Exchange server that - even if only previewed by a user - can allow an attacker to take control of the Exchange server itself with full Exchange Admin privileges.  The details can be found at this link, but I'll summarize here.

If an attacker crafts a message in certain ways via TNEF (Transport Neutral Encapsulation Format), and that message is then either previewed or opened by an end-user (which is the point of email after all), then the attacker can gain elevated privileges on the Exchange server and wreak havoc. While that sounds like a major problem, there is good news.  The attack would require a sophisticated attacker creating just the right message and then finding a live mailbox to send it to.  Not only that, but it would need to get past the SPAM and CAPTCHA systems if you use them.  Finally, eventually the anti-virus vendors will jump in as well, scanning for messages crafted with the exploit.

The really bad news is that end-users do NOT have to open an attachment this time.  They can unwittingly infect the mail servers by previewing the message body or opening the message itself.  Since Outlook clients and now even Outlook Web Access both Auto-Preview by default, end-users can follow all best-practice safety protocols and still end up infecting the organization through no fault of their own.  Add to this that some anti-virus tools may use MAPI viewers to scan mail, which means that the system designed to shield you could accidentally infect you instead.  The same goes for backup agents that use MAPI clients to do brick-level backups.

There is one more bit of scariness to be told, though information on this last peice is more sketchy than the other exploit vectors. It would appear that the TNEF message may not need to be even previewed by an end user, but could in fact attack a server just be being received by the server itself.  Even until we know more for sure on this last point, I think the previous points - which are verified by this author and others - are bad enough that you should jump on this right now.

Microsoft has released a patch - also available via this same link, under "Security Update Deployment" and so you can protect yourself and your organization.  I strongly urge you all to do this immediately, and to have your end-users run Windows Update (Start|Windows Update on Vista, Microsoft Windows Update Online for everyone else) to get critical Internet Explorer patches as well.

Now, this is not only your Exchange server that you are protecting.  If an attacker gains privileges on your mail server, they can use it as a launching pad for attacking other servers with the same exploit or any other they'd care to try.  That means that if you don't patch your systems, you could inadvertently be responsible for attacks against other servers all over the net.

And so, I am asking my readers to protect the community at large as much as they can, by spending the time necessary to download and install the appropriate patches.  Yes, this will mean a maintenance window for a reboot.  Yes, it's annoying that we've gotten hit with yet another critical vulnerability.  But, yes, it is our responsibility to make sure that the attacks stop at our own borders, and that our own mail servers do not become the launching point for the next Code Red.

If I cannot bring you comfort, then at least I bring you hope

As we get ready to celebrate the beginning of a new year (in many countries), now is a good time to plan for your Exchange Servers and what you'll be doing in the year ahead.  While this is not the best time to plan out new infrastructure or massive overhauls immediately, you can start planning what you'll do later in the year on that scale.  You can also plan for what you'll do right off the bat in January for things like maintenance and updates/upgrades.

I plan to dedicate entire columns to these topics in the next few weeks, but here's the overview to get you thinking (once the champagne wears off...):

Offline Maintenance is a good thing to do regularly for Microsoft Exchange.  About once or twice a year you will probably want to take the time and walk through the process.  Microsoft says that this is not a solid requirement, but it can help identify long-trending problems deep within a database, so doing it once or twice per year is time-consuming, but worth it.  Plan now for downtime, so that you don't have to worry about when you'll have the ability to do it later. 

Online Maintenance is automatic, but you may want to look at your Exchange Server profile and see when the best time to have it done really is.  By default, the online maintenance system will run every night, but if you're also performing backups overnight, the two systems can overlap, causing the maintenance to be halted and never finish.  Take a look at your event logs to see if both systems are operating normally, or if the online maintenance run is terminating early.  If it is, you may want to change the times it runs, or even only run it on certain days.  Unlike with Exchange 2000, later versions of Exchange can hold up admirably well with even weekly maintenance runs, but be aware of what won't get done until maintenance by checking out Microsoft's Knowledge Base and/or waiting until my upcoming column on the topic.

Test, test, test, test, test......then test again! Now is the perfect time to set out your goals for testing of recovery/availability solutions (see disclaimer at the bottom of the page).  Backup systems, failover systems and other recovery solutions MUST be tested, otherwise how do you know they're doing what they say they're doing.  Best bets: do a test restore of Exchange data (at all the levels available from your tool like database, mailbox, mail item) to a different directory on the Exchange server and mount the DB in a Recovery Storage Group to ensure it is working properly.  This should most likely be done once per month, but at least once per quarter.  It doesn't require downtime, and doesn't require end-users to do anything.  Perform a failover test (if you use a failover tool) or a total restore test at *least* once per year.  2x per year or once per quarter is better, but of course since this type of testing will impact end-users, scheduling can be an issue.

Housekeeping is never a bad thing to look at after the clock strikes midnight on January 1.  Are you reaching your database limits in Exchange Standard edition?  If so it's time to either start ratcheting down mailbox limits or else starting to plan your upgrade strategy.  If you're already on Enterprise edition, are you pushing the limits of the free disk space you have on your drives?  Plan now to move the databases to another partition with more space.  Have your Stores grown to alarming proportions?  Might be time to start planning out new Stores and even Storage Groups to distribute the load.

Finally, give a look through the event logs and such to see if there are repetitive errors or other trends that can indicate problems brewing.  One more common example of this is recurring event ID's 1018, 1019, and 1022.  These indicate that there could be problems brewing in the database itself, though it is very possible that the error events are the only indication you see until the DB crashes.  If you're looking for a tool to help wade through the log information, check out Splunk, an IT search solution set that can make your life a lot easier.  [Due Disclosure: Splunk is currently working on partnerships with Double-Take Software.]

Have a healthy, happy and safe New Year's celebration.  Here's hoping 2009 exceeds everyone's wildest dreams in terms of greatness, and that Exchange never ceases to help us all live in these very interesting times.

Until next year.....

Mike Talon, Chief Columnist, BeingExchanged.com

PS: The title of this posting is taken from the song "Closing of the Year" from the movie soundtrack to "Toys."  The lyrics of the first verse are incredibly relevant to our combined experience this year, and I offer them here:

If I cannot bring you comfort,
then at least I bring you hope
For nothing is more precious than the time we have, and so
We all must learn from small misfortunes
Count the blessings that are real
Let the bells ring out for Christmas
At the closing of the year

- The Musical Cast of Toys Featuring Wendy and Lisa

Get the message? If not, it may not be your fault.

Microsoft Exchange engineers have always come under fire when a message that was sent to someone in their organization didn't get delivered, for whatever the reason may be.  The same goes for messages that bounce back for no apparent reason.  It could be a problem with the user's account (disabled, misspelled, etc) or it could be a problem with the server (offline, not connected to the net, etc).  It could also be a problem totally outside your organization, and that's what I'd like to talk about today.

In this ever-more interconnected world, routing of email can often depend more on external factors than internal ones.  Email messages can get mis-routed or even totally lost well before they reach your system, or can get purposely removed from the Internet somewhere along the way through no fault of your own.  If you're finding that more and more outgoing email is lost without any visible reason, the most common problem that is beyond your immediate control is getting accidentally blacklisted.

Blacklists are specific shared lists of either email servers or - in some cases - individual email addresses that spam control companies make available to end-users.  The most common of these is the DNS BlackList (DNSBL) which can either track IP addresses of servers known (or thought to be known) to send spam, or can track the domain names of such servers.  You can find out more about DNS BlackLists from Wikipedia at this link.  These systems collect information on servers and domains that send out spam and/or dangerous files (like viruses), and share that information with anti-malware companies (like McAfee and Kaspersky among others).  The anti-malware companies then use those lists to update their anti-spam and anti-virus solutions to protect consumers.

So if your server's IP address or domain accidentally ends up on these lists, suddenly external servers won't send mail to your servers anymore and won't accept incoming mail from you either.  It isn't easy to get on these lists by accident, but it could happen to you.  One or more internal users may get hit with a virus that sends tons of spam mails out from your email server, or someone totally outside your sphere of influence gets their computer taken over by a virus that forges headers to make it look like your servers are sending spam - and that's just two examples of things that could get you accidentally listed.

Once you're on these lists, it can be a very time-consuming process to get yourself un-listed.  Generally speaking, it will be a manual process of contacting each organization that has you listed on their blacklist system, and proving that you don't deserve to be there.  You can start your search at w3dt.net, which will assist you in figuring out which companies and filters have got your IP listed as up to no good.  Once you have that information, contact those organizations directly and either ask what caused you to get blacklisted or - if you already know that info - what you've done to stop the problem.

Be prepared to show that you identified who was sending the spam and why (and that you stopped it), or that you've isolated and removed the virus.  You will need to be patient and persistent, and will probably need to make several calls over a significant period of time (days to even weeks) to make sure that the problem gets solved.  Each organization that has you blacklisted will want their own proof that you're not a threat anymore, and in reality - provided they're not abusing their power - I can understand why they're being so careful.  Spam and viruses are a huge problem in the net today, and making sure they protect their clients is their business model.

Once you have the blacklist fixed to show you're not a spammer anymore, it can still take several days or longer for end-users of those services to get the updates to their spam and virus control systems.  So, if you do end up on a blacklist, you have quite a road ahead of you, but well worth it to clear your good name!

Getting blacklisted accidentally is a horrible thing.  Blacklists are a good thing overall for the benefit of email users all over the world, but when they suddenly stop you from emailing anyone (or getting email yourself), they can be a giant headache that you'll have to deal with effectively to get back in business.

Exchange on DC's

Smaller shops have always been faced with a fundamental quandary when the time has come for them to move off Small Business Server and into the world of the independent version of Microsoft Exchange.  One of the biggest considerations is where to put the Domain Controllers for Active Directory.  The first thought most of my clients have is to put it on the same server as the Exchange system, but outside of SBS that's a very bad idea for a handful of important reasons.

For example, if you're ready to make the move from a single server to a cluster, you cannot install Exchange 2003 on a cluster node that is also a DC.  That is just simply not supported by Microsoft, and therefore isn't an issue up for debate.

It's much more likely that you'll be moving to a single-server, however, as if you're just moving off of SBS then the likelihood you need a cluster is reduced quite a bit.  In those cases, while you could install Exchange on the DC, it's still not a good idea.  The Exchange Best Practices Analyzer will even flag this condition as going against best practices, which should be a giant red flag for anyone installing an Exchange Server.

Here's a few reasons it isn't a good idea:

1 - Since Exchange runs many services under Local System accounts with elevated privileges, a successful attack via the Exchange system could easily expose your AD controller to outside influences.  With the potential to gain access to all domain resources if the AD controller is compromised, reducing attack surface is vital, and installing Exchange (or any app, this is not an Exchange-specific issue) will *increase* attack surface.  If you get hit on an Exchange Server that is not a DC, the attacker has access to only that server, not the entire Domain.

2 - 3rd Party integration can suffer.  Since many 3rd Party tools may expect to use Local Accounts and Groups (which wouldn't exist on a DC) or may otherwise need to control aspects of the Exchange System that may not be available or accessible on a DC, integration with Anti-Virus, Backup and Recovery solutions can suffer if the server hosts both roles.

3 - Performance impacts will be clearly visible.  Since the Exchange Server constantly uses the DC and vice-versa, placing both on one physical server can effectively double the amount of work that server must do.  Some visible effects of this are slower overall Exchange performance; longer response times for Domain commands (such as logging in, changing passwords, etc.; and much longer reboot times.

Putting the Active Directory and Exchange Server systems on one physical machine may seem like a way to save money, but in reality it will cost more in downtime and headaches than it will save on server hardware.

 

[Note: Small Business Server contains a highly modified version of Microsoft Exchange Server that is specifically designed to run on an SBS server.  It does not permit all 3rd Party applications, limits functionality and is also limited to only being able to run on SBS.  Therefore, running the SBS version of Exchange on an SBS DC is both a supported installation and a best practice. This article specifically applied to non-SBS versions of Exchange Server.]