Happy Thanksgiving!

To my US readers (and those around the world joining us this holiday):

Happy Thanksgiving to you and yours, may it be healthy, safe and wonderful for you - no matter how you are celebrating.

Some folks just take the day as a day off.  Others share turkey with friends and family.  Still others share a vaguely turkey-shaped mound of tofu instead, and at the other end of the spectrum, mega-carnivores can gather to share a Turducken.

No matter what your plans - even if that's no plan at all - catch your breath, take some time to relax, and try not to remember that the Exchange servers will still be there on Friday morning =)

Why 64-bit?

Of all the new requirements for the latest version of Microsoft Exchange (2007), the requirement for the systems to run on 64-bit hardware and Windows has been met with the most confusion and in some cases contempt.  Granted, Exchange 2007 will install on x86 machines, but that configuration is strictly for testing, and is not supported at all for use in a production messaging environment. The reasons behind the move to the new Instruction Set are many, but the big ones are:

1 - It's faster.  Hands down, an x64 server running on the appropriate hardware will out-perform a 32-bit server every time.  This is due to both the architecture of the server being able to handle more I/O operations per second, and just due to the fact that you'll usually have faster hardware when you upgrade the physical machine to an x64 architecture.

2 - You can push it farther.  Windows 2003 x86 editions have much lower limits in terms of the amount of RAM you can put into the box.  For Exchange, this is a devastating limitation.  Even with the boot switches which allow for extended memory utilization, Exchange 2003 and 2007 (x86) can only use about 3GB of RAM at their maximum, though the Windows system may be able to address more than that number overall.  64-bit Exchange 2007, conversely, has a recommendation of 2GB baseline RAM with another 2-5MB for each Mailbox, meaning that 2000 mailboxes would have you looking at 6GB RAM, and more likely much more than that.  According to the Microsoft Exchange Team Blog, you should be aiming at 32GB per Mailbox Role server, something simply impossible on a 32-bit architecture.

3 - You get more mailboxes with less hardware.  32-bit Exchange servers don't have a fixed limit on the number of mailboxes you can put on a single server, but many Enterprise users have found a logical limit of around 2000-3000 users per server, if all the hardware is maximized out.  The reason for this is that 32-bit systems have a fixed limit of a little over 500MB of Kernel Mode memory, less than that if you use the /3GB switch at boot.  Each Outlook connection uses a small amount of this memory, so eventually you end up with Kernel Exhaustion, and new connections either get delayed (temporarily freezing Outlook)or denied (Outlook errors).  64-bit Windows has up to a whopping (at least by today's standards) 128GB of memory.  Though it will be limited to 40% of total memory on the box, with the right configuration you could have thousands of Outlook connections on x64 and never miss a beat.

In the end, the move to Windows 2003 or 2008 x64 and possible new hardware for your Exchange 2007 server will benefit you and the end users in your organization.  Especially in these days of having to do a lot more with a lot less, biting the bullet and investing in the new hardware platform will allow you to extend your investment much farther than you were able to on x86 architecture.  That means fewer hardware upgrades spread over a longer period of time, and fewer physical machines required to support the messaging systems you need.

Under Pressure (da da dum diddy dum dum)...

First off, welcome to the new subscribers who hopped on-board with BeingExchanged!  Also, congratulations to everyone who recognized the riff I used in my Subject line, and for those who don't, it's referring to This Song, and don't we all feel old?

Back Pressure is a phenomenon in Exchange 2007 that can grind your email flow to a halt with little or no warning, so it's a great thing to begin to understand now, before you run into it in the real world.  The basic idea is that Exchange Servers through the years were famous (sorry in advance to the guys from Microsoft) for collapsing under heavy I/O load and/or lack of disk space.  Back Pressure allows the Exchange 2007 system to halt some of the processing of data temporarily to allow the system to keep running without the danger of I/O overload.

Specifically, Back Pressure allows Exchange Servers to stop accepting new messages into transport queues, thereby allowing system resources to free up over time and for those queues to empty out.  This translates into no mail coming or going from your organization while Back Pressure is in effect, which will have quite an impact on your operations.

While the Back Pressure system could impact your mail organization as a whole, it's good to point out that only Hub/Transport and Edge Services servers are impacted.  Messaging queues and the disks they sit on are what the Back Pressure systems monitor, not database locations.  Pure Mailbox Role (MBX) servers won't experience Back Pressure, but they will be impacted by it if mail cannot flow in or out of the MBX servers due to mail being unable to transverse the other roles.

When the Back Pressure system kicks into gear, the most noticeable impact is that Outlook clients will have all their outgoing mail stuck in either the Outbox or Drafts folders.  OWA will show everything in Drafts and other client software will show that mail isn't being sent.  Of course, no mail is coming in either, but that's a little more difficult to "see" unless the user was expected a message.  On further examination, you will see that no mail is coming into or going out of mail queues on the Hub Transport or Edge servers, a dead giveaway that Back Pressure kicked in if your connectivity is working just fine but no mail is moving.

There are three ways Back Pressure can be brought online on an H/T or Edge server:

First off, if the logical disk that contains logging and other information for either role runs below a set level of available space, Back Pressure will be invoked.  This is done to keep Exchange from "crashing" a disk, or filling up all space available and causing problems within the Exchange system or (if it's also the system drive) Windows itself.

Next, if the amount of physical memory or other system resources run low; Back Pressure comes online.  Since mail isn't coming in or going out, the resources will return to a normal level once whatever process is spiking finishes or is terminated.

Third, the number of pending operations in the memory queue (as opposed to the overall amount of memory in use) can build up if something is slowing down Exchange like a virus, DDOS attack or just a sudden influx of mail transfer.  If the depth of these transactions exceeds a limit set within Exchange, the result is Back Pressure stopping mail flow, allowing the existing transactions in memory to commit through to disk without new transactions coming in.

Each of these metrics has a different number of resources assigned as the "break point" where Back Pressure kicks in.

For disks, Exchange uses a formula to determine what space must be present and free before Back Pressure is invoked.  Specifically the formula is 100*(hard disk drive size - fixed constant) / hard disk drive size. The "fixed constant" number is 4GB for the release version of Exchange 2007, 500MB for the SP1 version. If the total available free space on the drives which contain queuing systems drops below that number, new mail flow stops and Exchange system spools out the queued data until the disk systems free up enough space to get above the threshold.

Finding the amount of memory used to reach the "high" utilization number, or the depth of the queues is a bit trickier.  There are several calculations and settings used to figure those out, and a complete description can be found in this article about Back Pressure technology in Exchange 2007.

The end result is the same.  Build an Exchange 2007 server with the wrong sized resources (disks, memory or server load), and the system will effectively halt itself to make sure that those limited resources don't crash.  So it is vital to properly size out both your expected user load, and the resources (physical and otherwise) that you're planning on running that load on.  You can change the thresholds for the Back Pressure triggers (see this link) but that's designed to be more of a temporary fix than a permanent change.  Instead, work with the publicly available sizing guides from Microsoft and other sources to make sure your hardware can handle the mail transfer without feeling the pressure.

MTC Part Two

Back at the Microsoft Technology Center here in NYC for another day of hands-on Exchange 2007 setup work.  We're presenting tomorrow on both Exchange and SharePoint protection with Double-Take Software (see disclaimer below), and today I got to finalize the environment and test failover and failback.  Of course, failover and failback worked perfectly, but I was really impressed with the new Hyper-V systems that we're using to create and run the demos.

So far they've handled all of the Exchange stuff the exact same way as physical servers, including the failover and failback processes that call for Active Directory and WMI updates along with starting and stopping Exchange services on one or both machines.  Since the system is automated, the changes happen rapidly.  On physical servers, this doesn't pose a problem, but sometimes virtual machines (sharing processors and RAM) can get bogged down.  That has not been the case here, with the Hyper-V VM's staying responsive and moving fluidly through the entire process.

Granted, you probably wouldn't put the DC/DNS Server and both the production and failover Exchange servers on the same Hyper-V host in a production environment.  But, it is nice to know that it can be done.

Hyper-V is relatively easy to use if you're familiar with the current virtualization tools on the market from a variety of vendors.  For Exchange 2007 on Server 2003, the VM's behave exactly the same as a physical box would with the same software set - as you would expect from a VM tool.  What is different is the speed at which they reboot.  Even with Exchange 2007's extended boot-up time, the average reboot took only 1-2 minutes, as compared to 3-5 minutes for physical boxes and other VM platforms.  This could just be a quirk of the environment we're building in, but that's still a fast boot up.

In terms of configuration, the GUI interface for Hyper-V allows for the standard set of server definition variables.  You select file locations for the definition files and the virtual hard disks, set the amount of RAM, processors, etc, and then boot the VM.  For these servers, we used cloned Windows 2003 Enterprise installs taken from other VM's already built, but you can boot from a virtual CD-ROM or any physical boot media the host can see attached to itself.

From there, as expected, the server behaves much like a physical box, loading the OS and rebooting as required.  Once you're up and running on a Supported OS, you can install the Hyper-V Integration Services via a virtual CD-ROM.  This enhances mouse and keyboard integration and opens up several dozen options - such as the ability to share folders between the host and the guest.

Networking options follow the standard set of VM options, including the ability to create Host-Only networks that can't see the outside world, or bridge physical adapters from the host to the guests.  This is especially helpful for Exchange, as you could create a Host-Only network between the Mailbox role servers and the Hub/Transport and CAS role servers, while giving the latter another interface to the outside world.

We're using a combination of virtual disks and physical disks, so I did not get a chance to play around with the Snapshot features - which require that all volumes be on virtual disks to function properly.  I've done snaps of Exchange 2007 servers with other tools with a good success record, and I was hoping to try it with Hyper-V too.  That'll have to wait for next time.

Oh, before I forget, though we'll be in the Grand Central Briefing Center, this MTC has an Envisioning Center as well.   It's a theater-like room that showcases all the latest (and even some upcoming) Microsoft products.  And sure enough, they have a working Surface unit that I got to play with.

You can find more info on what Surface is (and I really suggest checking it out) at this link.

Maximum Mailbox Stores

While reviewing some information, I came across an interesting fact.  While you can have up to 50 Storage Groups on a single Exchange Server 2007 Enterprise Edition box, you can still only have 50 databases total.

Some background info:  Storage Groups (SG's) are logical collections of databases in Exchange Server.  Information Stores (more often simply "Stores") are the databases themselves.  Each SG has a set of logs associated with it that cover all Stores within the group, and there can be both Public Folder Stores and Mailbox Stores within the same group.

If you intend to use any of the built-in data-replication solutions in Exchange 2007 (CCR, SCR, LCR), then you can only have one Store per Storage Group, which means you can only have 50 stores no matter what.  But if you were planning on not using these tools, and wanted to have multiple Stores per group, then that 50 store limit becomes very important to keep in mind.

Let's say that you have a very large Exchange 2007 user base, and you want to split the users into 25 SG's.  That would mean a maximum of 2 Stores per group.  Even though a SG can hold up to 5 Stores, if you tried to put 5 Stores in each of the 25 SG's, you would exceed the overall server limit of 50 Stores for the box.

Granted, this will most likely not impact the majority of users, but for Enterprise Exchange 2007 shops - and especially Exchange Hosting Services Providers - this is definitely something to be aware of.  Assigning each client or business unit their own Store could quickly fill up the number of Stores available on the server, even if you are nowhere near the total number of SG's allowed.

MTC Day One wrap-up

After the first day of working here at the Microsoft Technology Center for our upcoming presentation/demo, I can safely say there is some impressive technology to be found here.  We got a chance to play around with Hyper-V on higher-end hardware instead of the usual test machines most readers would be used to (myself included).

Hyper-V performed admirably under the load of Exchange 2007 on Server 2003 x64 Enterprise Edition.  There is definitely a "feel" that lets you know you're not just remoting into a physical machine, but nothing that is going to stop you from getting things done. 

Exchange, for its part, was exceptionally well behaved today.  Not one major hiccup during the entire install and config process.  Either I'm getting used to setting up 2007, or something in SP1 is making life easier for me.  Whichever, I'm a happy techie.

More on Thursday when I put the newly-built systems through a failover/failback test.

Hyper-V, MTC and other inside words

Double-Take is doing a tech-focused presentation in NYC for some of our current clients on Friday, which means - since Exchange will be on the menu - I am setting up some demos this week to show off a the big meet-up.  Those who are unfamiliar with Microsoft may not know about the Microsoft Technology Centers (MTC's).  MTC's are essentially mini-data-centers where Microsoft and their partners can show off new products, prove out different configurations and such.

As this is a one-day event, we're going to be using Hyper-V for the server base, which is Microsoft's new hyper-visor based virtualization system.  While the combo is not officially supported for production environments, you can install and run Exchange 2007 on Hyper-V virtual machines for testing, development, etc.  We'll be using it to setup a Source and Target server for the demos, and so far things are going well.

I'll keep everyone posted as to the overall experience path with Exchange 2007 on Hyper-V in our lab, so stay tuned!

Core is not so core for 2007

Microsoft created a shell-like Windows installation for Server 2008, called Server Core, which installs a minimum of interface components to the server, and allows for more processing power to be dedicated to the application systems.  While this would sound perfect for Exchange Server 2007, there are a few key things that will prohibit you from taking advantage of the Server Core system for your messaging servers.

First and foremost, since Exchange relies on things like PowerShell and .NET, you can't install Exchange on Server Core just yet.  Core doesn't support the installation of .NET or PowerShell (which relies on .NET), so even if you managed to get the installer to run; you'd fail all the pre-flight Exchange tests.

Also, Exchange Server is not impossible to administer without a GUI, but it is difficult to do so.  The majority of users would prefer to use a GUI to do most of the non-day-to-day stuff, and that's simply not an option where no GUI exists.  Server Core is literally a shell environment, the only thing you see on your desktop is a command line interface.

Microsoft doesn't have a clear statement on direction for running Exchange 2007 on Server Core in the future that I can find, so there is a good chance that this just won't become a supported solution.  It would have been great, since the reduced overhead of no GUI on the OS means more resources for the Exchange system itself, and since Exchange 2007 is designed to be run through PowerShell as a sole method of admin if necessary, Server Core would have been a great fit.

You can install Hyper-V to a Server Core platform (or just install the stand-alone version of Hyper-V server to the bare hardware), which would allow you to run Exchange Server 2007 in a VM.  However, with limited support for Exchange virtualization, and the fact that you'd be installing a full OS in the guest anyway, this probably won't be a helpful configuration.

My guess is that as Server Core continues to gain traction in the enterprise space, Microsoft will begin to find ways to get .NET into the platform.  It may never support the full Exchange 2007 system, but I suspect Server Core might eventually become the platform of choice for things like Hub/Transport, CAS, and Edge Services.

DST issues should be resolved now - but for how long?

In case our overseas friends were wondering why many peoples' emails were all flagged 1 hour off, it's because servers here in the US (and elsewhere) may not have been properly patched for Daylight Savings Time changes.  As odd as it might seem, many Exchange servers were not set up with the correct patches when the start and end dates for DST were changed a while back, and therefore would have the wrong time stamps until this last Sunday finally saw DST end and the problem vanish.

For those who have not yet patched your servers, please visit the Exchange Server site from Microsoft and find the right patches.  DST starts again in the Spring, and if you're still not patched up by then, no one is going to be quite so understanding =)

Remember, even if you think you have all the updates, check again anyway.  I personally have seen several emails flying around the last two weeks that were 1 hour off.  This can cause issues with mail flow, and wreaks absolute havoc on shared calendars.  The check takes only a few minutes, the results of not checking can last for months.

Also nothing to do with Exchange

For those around the world who would like to follow the elections here in the USA, this widget should show you how things are going!

(I pulled out the widget, as the election is over now, but for those interested, Barak Obama won with 364 Electoral votes to John McCain's 173)

Exchange 2007 32-bit edition?

I have fielded a lot of questions lately from clients and co-workers about using Exchange 2007 in a 32-bit environment.  So, just to clarify, here are the answers to the more frequently asked questions:

1 - There is indeed a 32-bit version of Exchange Server 2007. It is *not* supported in production, however, so you should never be using this version for any real users, even if it's only a small number (or a single user for that matter).  This version was built to allow administrators and engineers a way to test and experiment with Exchange 2007 without the x64 hardware that the production version requires.

2 - You will most likely not be able to enter your license key into the 32-bit addition.  I can't find a large amount of Microsoft-official documentation on this one, but everything on the web seems to point to the evaluation version not expiring after 120 days. You will get a nag screen every time you open the Exchange 2007 GUI, but the services will still run. Please add comments if you have more definitive information on this topic, and I will post updates.

3 - It is not meant for stress testing, just for functionality testing.  This means that you can play around with the feature sets within Exchange 2007, but since your RAM will be severely limited and - very likely - your server platform will be well below what an x64 architecture will provide, you cannot expect to see those features run as fast or efficiently as they would on a "real" 2007 install.

4 - The 32-bit version is GREAT for getting your feet wet. It requires less RAM, and whatever hardware you have sitting around that will run Windows Server 2003. That means you don't have to buy a new server, just to learn how to use Exchange 2007.  Not only that, but with lower system overhead, testing/learning on Virtual Servers becomes even easier.

5 - Both the 32-bit and 64-bit editions of Exchange 2007 can communicate with 32-bit OR 64-bit Active Directory Controllers. In short, you don't have to upgrade your DC's to x64 when you put the new Exchange 2007 servers into play.

So, go out an play with Exchange 2007 on a 32-bit box if the 64-bit hardware just isn't available for your testing/learning.  Just remember, production servers absolutely must be 64-bit, otherwise if you need to call Microsoft you'll find they are less than able to assist you - which is never a good situation to be in.