What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Net Neutrality and EAS

Mobile messaging and collaboration makes for a big, bright world in business today. What was once the domain of a single service provider (Research in Motion - RIM) has evolved into a robust set of platforms to convey email, appointments and contacts from one device to another.  Blackberries, Windows Mobile devices, iPhones, Android phones and so many other systems can communicate either directly or through a proxy to a Microsoft Exchange platform.  This unleashes the workforce and allows for your people to be where they need to be in order to work, instead of where they have to be just to talk to each other through the email system.

Net Neutrality is the idea that no matter what the network provider offers in terms of services and software, you should be able to use the devices of your choice and the platforms of your choice on those networks.  It’s a great theory, but putting it into practice is causing some issues along the way.  The FCC set forth a set of basic rules that they wanted carriers to follow, and in the greater sphere of comments, they were well received.  They recently added in two more proposed rules that directly impact cellular networks (digital broadband) and the services that run across it – including Exchange Active Sync (EAS).

You can get a rundown of the entire proposed rule set in this article, but the two that directly impact EAS most are:

5. Broadband providers cannot block or degrade lawful traffic over their networks, favor certain content or applications over others and cannot "disfavor an Internet service just because it competes with a similar service offered by that broadband provider."

On the surface, this looks like a standard anti-competitive rule.  In reality, however, many service providers in the cellular world are viciously blocking competing technologies, and their claim is that forcing neutrality will destroy their business.  EAS is a great example of this phenomenon, as not that long ago many providers didn’t allow that traffic on their mobile networks.  Mostly, this was due to the fact that they wanted to pus their own version of enterprise email synchronization (such as Sprint’s ill-fated attempt on the earlier Palm Treo devices).  Eventually, the need to allow this traffic or lose business to other devices and networks overrode the desire to use and sell their own platform, but that took a great deal of time, and lead to quite a bit of bad press and back-end attempts to circumvent the blocks.  By forcing mobile providers to allow all valid and legal traffic, the atmosphere for open communication standards will grow and more people will be able to take advantage of more technologies.

6.Broadband providers must be transparent about the service they are providing and how they are running their networks.

Proprietary networks are nothing new, but trying to create an EAS client for a phone on a network that actively blocks your ability to figure out how it sends and receives data makes this close to impossible.  Some providers have blocked all traffic they do not wish to have on their networks by simply making it very difficult – or nearly impossible – to figure out how a 3rd-party tool can possibly communicate with it.

I’m of two minds on these proposed rules.  On one side, EAS and other technologies require open, transparent communications platforms to work. Exchange can communicate with a whole world of different vendors’ mobile applications, but only if those apps can talk to the Exchange Server.  On the other side, competition drives better software and platforms.  If it wasn’t for all the things you can only do (or could only do) on an iPhone, RIM and Google would never have had the impetus to push their own platforms to new heights, and we’d still be staring at plain-text emails on black and white Blackberry devices.

It’s going to be a very loud fall season as the mobile providers and the FCC battle out these proposed rules.  The end result will have a huge effect, either good or bad, on how flexible and feasible your mobile Exchange platform plans will be. Competition is a good thing, but it cannot be forced on the market at the expense of profits.  There must be a way to balance these scales, and it will need to be found before Net Neutrality can be forged in the mobile marketplace.

 

 

Fold with me!

Folding at home is a great way to help science expand to find brand new ways to help humanity. It’s a project of Stanford University, and been around since 2000.  Long story short, you install a client software package on your PC that uses your unused CPU cycles to run protein folding equations.  This lets hundreds of thousands of computers from around the world all work together to discover how diseases work, and how to beat them!

So, what does this have to do with Exchange? Two things.

1 – I have a team on Folding @Home.  If you use team number 171744 after you install, you’ll be joining up on TalontedTweeple.

2 – For years now, virus attacks have leveraged Exchange Server to proliferate malicious software that creates a huge network of corrupted computers to act as a giant attack grid.  It’s nice to be able to use the same theory (grid computing) to do something good for the world instead.

These guys have been running the program for nine years now, and you can see on their website all of the things the research projects that use Folding @Home have accomplished.  This is a great way to let your PC work for the world when you’re not actively using it.  The software can be tweaked to contain what it is allowed to do and not do, and is very well behaved. 

Let’s face it, you’re using power every moment that your PC is running – even if you’re not using it – so why not let it do some work while you’re not around? Even if you don’t want to join my team, you can join teams for Google, IBM, or dozens of other companies and organizations.  No matter what team you join (even no team at all), everyone is working toward the same goal.

Get back to where you once belonged (Failover Cluster version)

In honor of the re-release of the Beatles stuff all over the world (games, CD’s, maybe iTunes at some point), I took the title of today’s post from their song “Get Back” on the album Let It Be (Remastered).

I am, of course, going to tie this to something in Exchange; specifically Exchange 2007 Standby Clustering. Standby clustering refers to the theory of using a replication engine (like the native CCR or a 3rd-party system like Double-Take Availability – see disclaimer below) to place a copy of the data for the Storage Groups of the production cluster onto a secondary cluster.  Once the data is replicated, you can use the /RecoverCMS commands to recreate the production Exchange Cluster Mailbox Servers (CMS’s) on that secondary cluster.

The solution set for bringing up the Storage Groups and CMS’s on another physical cluster setup in the same or another location is fairly well established.  If a single node fails on a production cluster, other nodes take over the failed Storage Groups and work resumes in a very automated fashion.  If multiple nodes, or the entire cluster, fail you use /RecoverCMS and the associated protocols to manually get everything working on another system – so long as a copy of the data exists to work from.

The problem has traditionally been best expressed by the phrase, “And then what?”

If the original cluster failed completely, the answer was simple.  Rebuild the systems with the same node names, but prepare the systems as though they would be a new /RecoverCMS target system.  However, if you have not lost the production systems, and they’re stable enough to be used again, you would still have to reinstall them without some additional help.  The most common reasons for this kind of outage are routine testing of the failover systems and extended power failures that generators and UPS systems can’t handle.

Microsoft does offer a command set to fix this particular problem, but it is not well known or publicized.  As a matter of fact, during a recent client troubleshooting session, we had a couple or techs from Microsoft on the phone (Premier Support in this case) and they were not aware of this particular method for cluster restoration.

Once you have fixed whatever went wrong, if your production cluster is still viable (and is suitably stable for continued use), you can use a command set called /ClearLocalCMS to remove the original CMS entries from the original production cluster.  Doing so is not without risks, and you should familiarize yourself with this KB article on the subject before you try it. 

/ClearLocalCMS will remove the CMS components off the original production nodes, clean up AD, and disable the virtual computer object for the original cluster CMS.  This ensures that Exchange doesn’t accidentally address the original cluster system, even after the restore process begins.  Once the CMS is cleaned, you can go about restoration of the data using the same tools as you used to get it over to the standby cluster in the first place.

To get back to your original servers, use the /RecoverCMS command in the opposite direction (from DR back to production) and then use /ClearLocalCMS commands to re-prepare your DR cluster for use in the next emergency.

Jumping between clusters is not an automated or easy process, but it does work correctly if you follow all the steps in both directions.  This set of command suites (/RecoverCMS and /ClearLocalCMS) can allow you to get back to where you once belonged, every time.

Back to Exchange stuff!

For those who missed it, the Exchange Eco-System Protection (2003, 2007 and beyond) is available as an on-demand event now.  Free with a quick registration form.

Click here to register and view the presentation.

More good stuff later this week.

Eight years out

The online world has been a part of my life since I was a teenager.  My father had been one of the gearheads who joined Compuserve before it even had a GUI, just lines of scrolling text.  When I got out on my own, I wrote columns, learned new technologies and brought my words to the world.

On September 11th, 2001; I could do no different.  So today, seven years later, as I think back about that moment in my life, I thought I’d share the words I had on that day.  Unedited, typos and misinformation still right where they were, this was the Yahoo Groups post that I put together for my weekly column, Reality Checksum, when I finally made it back home that night. It is corny, flowery, and shows the inexperience of my writing not that long ago.

It is also my tribute.

I have gotten back to my life, walked away from a lot of what I saw, but I will never forget.

 

"The whole darn world is on fire, and my favorite TV show's not on."

A Special Edition of Reality Checksum

That line above is from a song by Billy Falcon called "Wonder
Years." He sings it as a quote from his young daughter as she
watched LA burn in the riots some years ago. She couldn't understand
why the streets were erupting in flame, why people had died, why –
just why.

Today, I felt that way.

I left my apartment today to go to a business meeting. It was a
great day, my first big deal with my new company, a sales colleague
from EMC was picking me up in Astoria to drive out to Jersey City to
the meeting. We were joking in the car and going over plans for the
meeting when I glanced back over my shoulder to see my world go up in
flames.

One of the Twin Towers was burning, smoke pouring from the upper
floors as I stared in disbelief. Shortly, as we listed to the news
radio with mouths dropped open, the other tower erupted in flame as
well. We listened as the radio told us that planes had caused the
damage, just ordinary planes – the same ones you and I would fly in
to get just about anywhere these days.

Then the shocking truth hit like a bulldozer, the planes had been
hijacked, the US was under terrorist attack, and it wasn't over. The
Pentagon was hit, Camp David, and another plane was taken down before
it could reach its intended target. We couldn't believe it as we
finally reached our destination directly across the river from ground
zero. We watched as the flames grew, consuming more of the buildings
every minute, but we weren't that concerned as these buildings were
supposed to survive such things (since the Empire State building was
hit by a bomber, the Twin Towers were supposed to be able to
withstand a similar accident).

We gave our apologies and condolences to the client we had come to
see – they had many clients in the Towers and cancelled the meeting.
And then we sat in the car and tried to figure out how to get back to
Queens when Manhattan was locked down. That's when we were struck
again.

I felt a rumble, as if a subway was running beneath the car, and
looked up to see one of the Towers fall from its heavenly perch down
to earth with a crashing roar.

For a full five minutes, two guys who are paid to talk for a living
couldn't say a word.

We began to move back toward the city, trying to just get home. Cell
phone services were out, too many transmitters had been in the Towers
and now didn't function. Luckily my wireless e-mail device was
working and I found out that my loved-ones and co-workers were safe.
Just as I was getting the last of my messages out, the second Tower
joined its sister and removed an epic landmark from the New York
Skyline forever.

We pulled over and tried to get to my company's offices in Hoboken,
NJ, but that office was evacuated. Finally, after much map
consultation, we found a route home and took the long ride back to
our abodes.

Now I'm watching television, every channel screams at me about events
I witnessed first-hand. By the grace of whatever higher-power you
subscribe to I was not at ground-zero, but the shockwaves of that
explosion racked me still.

Why? What did this gain whatever group is too cowardly to announce
itself? What could this group of cold-blooded murderers ever hope to
accomplish with these violent, senseless acts? This was not a
military target, this was not a governmental target, this was a
civilian building. Yes, it represents the financial might of the
world, and many would say that capitalism is bad, but does this give
anyone the right to kill tens of thousands of innocent people?
I hope we do not go out and bomb the crap out of some small country
in retribution – that would make us as bad as them. I hope we find
the masterminds, the people who figured out how to do this but didn't
want to get on the planes themselves. I hope we hunt down everyone
responsible for this travesty and haul each of them before the World
Court. I am a peaceful man, but not a pacifist, so if America needs
a death to satisfy our vengeance, then let them be executed by the
order of that court – but first let them be used to send a message to
the entire world.

Let the trial show the world that their plight was invalid, that
their fight was unjust. Let it show everyone that these animals are
nothing more than cold-blooded murderers with no higher purpose than
to kill as many people as they can to bring attention to
their "cause". Let each of them and their "causes" be discredited on
the Global Stage, never again to be given respect, to be allowed to
grow to this dangerous level, to flow into a grievous act like this.
And when all is said and done, let us show the world what we have
shown them time and time again. No matter what you do to us, how
many you kill, how much you destroy, how far you take your "cause" on
the Global Stage; you will never kill the indomitable spirit that
burns in the heart of every American. We have survived flood, fire,
disaster, war, corruption, and even a Civil War that tore our very
heart in two – but every time we came back, stronger than before,
ready to meet any challenge put to us again and again. You have not
won the day, you faceless, nameless cowards; you have done nothing
but rallied all of America to come crashing down upon you with all
the force of our Spirit. You will be found, you will be brought to
justice, and you will find out that no one tramples the American
Will – no matter how hard they try.

Now, let us begin to heal. Let us rebuild the Towers, the Pentagon,
our lives. Mourn those we have lost to this tragedy, respect their
memories by going on and living life to the fullest every day. Talk
to your children, explain what happened as best you can to their
young minds. We have to shape the future through them, and lead them
into a world of their own American Spirit, so that they may face the
challenges their world will put to them.

This morning, I saw firsthand my world crashing down – literally.
But brighter than the fires that burn in downtown Manhattan burns
another fire. It burns in my heart, and yours, and the hearts of
every free man and woman and child in the entire world. It is the
flame born of our outrage, our pain, our morning, our shock; but it
is beyond this. The flame that burns within our hearts will fire the
forges that will rebuild not just New York and Washington DC, but
will rebuild our world to heal this rift torn by violence.

My deepest condolences to all those who lost loved ones to this
tragedy. Let us morn their passing in the very greatest possible
way – live life, live on, and in our hearts, our mind, and our deeds,
let us remember them well.

And for those who have young children who cannot understand why they
were sent home or what is happening in their world, another song may
bring some solace to your troubled parental hearts:

"Little child,
dry your crying eyes,
how can I explain the fear you feel inside?
Cause you were born into this evil world,
where man is killing man and no one knows just why.
What we have become?
Just look what we have done.
All that we destroyed you must build again."
- White Lion, "When the Children Cry"

Let us begin to build again.

 

in necessariis unitas, in dubiis libertas, in omnibus caritas
Mike Talon

September 11, 2009

New York City

CCR clustering is still clustering, and so is DAG

As more and more of my readers move to Exchange 2007 and 2010 from Exchange 2003 and earlier versions, I hear a lot about how using the new High Availability tools will finally free them from the yolk of clustering in Windows.  While both CCR and DAG are definite improvements over traditional shared-disk clustering, neither is a departure from clustering entirely.

We’ll be talking about the new HA stuff in Exchange 2010 (along with much more of course) in the webinar Double-Take Software and Microsoft are presenting tomorrow.  I’m the speaker for Double-Take, and Patrick Foley from Microsoft is going to be doing their portion. It’s September 9th at 11am, and you can still register for free by clicking here.

In the meantime, it is important to realize that both CCR (Continuous Cluster Replication) and DAG (Database Availability Groups) are offshoots of Windows Failover Clustering (WFC).  They both change the way WFC works, and by quite a lot, so you may never touch the underlying cluster technology, but it is still there.

CCR – as its name implies – works by allowing you to create a cluster during the installation of Exchange 2007.  This one is a bit easier to see as part of WFC, as you have to create a Failover Cluster first – specifically a Distributed Majority-Node File Share Witness Failover Cluster.  After that, when you install Exchange Server you can specify which server(s) will be the Active node(s) and which will be passive.  This creates the clustered Exchange resources for you, making the overall process of setting up clustering for Exchange a lot easier.  As this one has Cluster in the name, it’s easier to see the WFC roots.

DAG will permit you to create the cluster itself from Exchange 2010 command sets, eliminating the need to pre-create the Failover Cluster prior to getting the Exchange installation rolling.  While this makes the process even easier than in 2007, it still requires that you have two or more servers capable of running Distributed Failover Clustering.  This means that not every version of Windows 2008 is going to be suitable for DAG, but also means that – under the hood – you still need to know how Distributed Failover Clustering works to properly manage the DAG systems.

In both cases, the required level of understanding of clustering is greatly diminished from what was needed in Exchange 2003 and earlier versions.  Most of the guts of the cluster are controlled by Exchange itself, which is a double-edged sword.  On one side you have the fact that folks who don’t have a lot of cluster know-how can now set up HA solutions for Exchange.  On the other side, people who don’t have a lot of cluster know-how are facing troubleshooting clustered Exchange solutions they may not have realized were there.

Both solutions work great for Exchange.  While they don’t eliminate the need for 3rd-party products to help with overall HA (and I’m biased on this one, see disclaimer below), they do make mailbox server protection much more complete.  Just remember that you’re still running on a cluster, and arm yourself with the knowledge needed to keep it running smoothly.