Certificates are the mechanism that is relied on very heavily to provide authentication in federation—and in all of Office Communications Server 2007. If you have read the initial chapters on the setup and deployment of Office Communications Server, you already know that a lot of certificates are used in the server communication paths. This is not by accident—defense in depth requires that the servers re-affirm that the incoming or outgoing traffic is from trusted sources. Servers have difficulty responding to challenge/response authentication methods, similar to what is presented to a user logging on. Passwords can be compromised, as the typical password is used over and over, from service account to service account. Finally, requirements for secure communication demand strong authentication of the server and encryption of the traffic from client to client so that it is not sent in clear text and easily captured by a man in the middle attack. Though there are other security methods that could be implemented, certificates seem to be well suited for the purpose.
There are a multitude of certificate types. There are simple S/MIME certificates for signing and encrypting e-mail. Then there are very complex certificate types that are designed for a specific purpose. These certificates typically have a custom template designed for them that define exactly the fields, the key types and length (the longer the key, the harder to break the encryption), and the method of the hash or signing algorithm. Templates also define whether the certificate will carry and be able to present more than one Subject Name, known also as a Subject Alternate Name.
Subject Alternative Name (SAN) is not a feature that is new to Office Communications Server 2007, but it is used quite a bit more in this version. You're strongly encouraged to use the SAN to identify other names that the target server might be known by. It's important to fully understand what this means. A server might have many names that it is known by—for example, edge.contoso.com, access.contoso.com, sip.contoso.com, accessedge, and internaledge. The only way that the conversation between a client and server can be initiated is if the client asks to speak to a name that the server is known by. The client then presents the certificate that is associated with that name (if one exists). As a day-to-day example of what this means, consider the documents we use to identify ourselves. A driver's license or a passport identifies you, but only by a single name. If asked for a driver's license, Bob is identified as Robert on the legal document that he presents. He answers to "Robert," but he is also known as "Bob." Relating this scenario to a certificate with a SAN, the Subject Name (SN) is Robert and there is a SAN entry for Bob.
For our servers that aid in the identification and authentication process, DNS is the typical means to determine what name a server goes by. Common use of A records and PTR (pointer) records provide the most often used DNS functions—forward lookup (A record) and reverse lookup (PTR record). Of course, DNS does not have to be involved at all. If a server is isolated, a HOSTS file is the best choice for providing the name-to-IP-address resolution, and it provides the necessary resolution for certificate SN or SAN lookup. It is possible to have a disconnect between what a server is known by internally and what it is known by in the perimeter. In this case, a HOSTS file can assist in resolving the name conflict, which otherwise might invalidate a certificate.
With certificates, therefore, the most important thing is to plan and define what the names of the servers will be. All people involved in defining the server names should agree, including the network engineers, name resolution management (DNS), and the server build team. Start with an architectural session to design the environment, which will help avoid most misunderstandings. Defining the server environment, communication paths, name resolution, and certificate standards in the planning session before implementation starts—and documenting your decisions in Microsoft Office Visio for mass communication—should be considered mandatory.
Tip
If after putting your plan in place, your IM servers don't talk, it's likely to be a DNS/certificate naming conflict or mismatch. Of course, you want to check the obvious, too. Ensure that all switch ports are on and configured correctly. Check the server services for proper state of operation. Recheck your IP configuration. It happens frequently that the obvious is overlooked because so much emphasis is put on certificates and naming—simply because it's newer and therefore less understood.
To illustrate why SAN is important, consider the following: you want to name a given interface two different things based on what it will be used for by distinct groups. "Sip.contoso.com" is a good name for the external interface, but then, so is "access.contoso.com." "sip.contoso.com" is where I want internally known users to connect. "access.contoso.com" is good for my federation interface, and anything else external that wants to connect to my Access Edge Server. It simply defines purpose. And I need only one certificate to do this using the SAN field in conjunction with the Subject Name field. Also, migrations and acquisitions can be eased if the name of the service point changes as little as possible.
From the preceding example, it is evident that creating multiple names for a certificate can often be a good idea. For example, when you have a farm of servers sitting behind a load balancer with a Virtual IP (VIP), the VIP will have a unique name, and it will address each server behind it. And each server behind that load balancer might have a certificate. Each server in a farm of this type might have a copy of the certificate from the first server in the farm. But each of these servers has a unique name as well, not just the name associated with the load balancer. It is possible that you would need to refer to a server as an individual server if the load balancer fails. This is a more likely scenario on the internal network rather than external network, but it is possible nonetheless.
The specific type of certificate that Office Communications Server requires is a Transport Layer Security (TLS) certificate. This is primarily a Web Server Secure Sockets Layer (SSL) certificate, with some differences. (By and large, SSL is being replaced with TLS because it's more flexible and provides much more robust security potential. This seems to be a natural transition, as the TLS 1.0 specification is based on the SSL 3.0 specification.) Certificates of this type are of the public/private key type and require infrastructure to manage them. The subject is too broad to cover in detail here, but a public key infrastructure (PKI) or public certificates will be required. Public Certificate Authorities (CAs) that provide these types of certificates are VeriSign, Thawte, and so on. An alternative to using public certificates is to deploy your own internal PKI by using Windows Server 2003 Certificate Services.
Even if you deploy your own PKI, however, you will require at least some public certificates because federation requires public CA certificates for its external edge. The technology and process in use is related to the discussion on trust. One way that you can decide to trust someone is to have a disinterested third party vouch for them. In this case, you (Contoso) get a certificate from a public CA. You need to set up federation with Fabrikam. They get a public CA certificate. Both parties install the public CA certificates, and the high-level conversation between the two edge servers begins with a confirmation of authenticity and trust. Each of the servers attempts to authenticate the other server's certificate and the information on it. Contoso's certificate says "Contoso.com". How does Fabrikam know that this is true? Fabrikam refers to the authority (the public CA) that issued the certificate because "Contoso.com" is on the certificate as well.
When a certificate is issued, the issuing CA signs the certificate. If another entity needs to know if the certificate was really issued by the CA, all that this server needs to do is to retrieve the CA's public key (and verify the signature with the public key). Public/private key pairs are made in such a manner that there is (realistically) only one possible public key for a given private key. And the public CA "signed" (actually, used a small digest hash) the certificate with its private key. And what is the only key that can "read" the signature? The public CA's public key. Once the server has confirmed that this certificate of authenticity was actually issued by the public CA, it's safe to assume that the holder is who they say they are. It's worth mentioning at this point that one further verification is to check certificate revocation lists (CRL) at the CA—just to ensure that the certificate we are verifying has not been revoked for reasons of theft, misuse, fraud, and so on. This process continues—simultaneously—for both servers.
Once the process is done, each server has confirmed who the other is. They now can start talking and federating. However, we still need to provide encryption. TLS is a public/private key technology. But because the keys are so large, it's a waste of CPU power (because encryption is all very complex math) to use the public/private key pair—except to exchange session keys.
Session keys are a specific type of key that the two servers exchange to encrypt/decrypt the traffic between them. One server—Fabrikam for this example—creates a session key, encrypts it with the public key of the intended partner, and then sends it on. Key exchange is complete—and this was 100-percent safe, as the only entity that can open a package that was encrypted with Contoso's public key would be Contoso (the only server that would have the private key).