Chapter 8. Growing Your Domain
“What size do you want to be?” it asked.
“Oh, I’m not particular as to size,” Alice hastily replied; “only one doesn’t like changing so often, you know . . . "
“Are you content now?” said the Caterpillar.
“Well, I should like to be a little larger, sir, if you wouldn’t mind . . . "
How Many Nameservers?
We set up two nameservers in Chapter 4. Two servers are as few as you’ll ever want to run and, depending on the size of your network, you may need to run many more. It is not uncommon to run four or more nameservers, with one of them off-site. How many nameservers are enough? You’ll have to decide that based on your network. Here are some guidelines to help out:
Run at least one nameserver on each network or subnet you have. This removes routers as a point of failure. Make the most of any multihomed hosts you may have because they are (by definition) attached to more than one network.
If you have a file server and some diskless nodes, run a nameserver on the file server to serve this group of machines.
Run nameservers near, but not necessarily on, large multiuser computers. The users and their processes probably generate a lot of queries, and, as an administrator, you will work harder to keep a multiuser host up. But balance their needs against the risk of running a nameserver—a security-critical server—on a system to which lots of people have access.
Run at least one nameserver off-site. This makes your data available when your network isn’t. You might argue that it’s useless to look up an address when you can’t reach the host. Then again, the off-site nameserver may be available if your network is reachable, but your other nameservers are down. If you have a close relationship with an organization on the Internet—say another university, your ISP, or a business partner—they may be willing to run a slave for you.
Figure 8-1 presents a sample topology to show you how this might work.
Notice that if you follow our guidelines, there are still a number of places you can choose to run a nameserver. Host d, the file server for hosts a, b, c, and e, could run a nameserver. Host g, a big, multiuser host, is another good candidate. But probably the best choice is host f, the smaller host with interfaces on both networks. You’ll need to run only one nameserver, instead of two, and it will run on a closely watched host. If you want more than one nameserver on either network, you can also run one on d or g.
Where Do I Put My Nameservers?
In addition to giving you a rough idea of how many nameservers you’ll need, these criteria should help you decide where to run nameservers (e.g., on file servers and multihomed hosts). But there are other important considerations when choosing the right host.
Other factors to keep in mind are the host’s connectivity, the host’s security, the software it runs (BIND and otherwise), and maintaining the homogeneity of your nameservers:
- Connectivity
It’s important that nameservers be well connected. Running a nameserver on the fastest, most reliable host on your network won’t do you any good if the host is mired in some backwater subnet of your network behind a slow, flaky serial line. Try to find a host close to your link to the Internet (if you have one), or find a well-connected Internet host to act as a slave for your zone. On your own network, try to run nameservers near the network’s topological hubs.
It’s doubly important that your primary nameserver be well connected. The primary needs good connectivity to all the slaves that update from it, for reliable zone transfers. And, like any nameserver, it will benefit from fast, reliable networking.
- Security
Since you would undoubtedly prefer that hackers not commandeer your nameserver to assist them in attacking your own hosts or other networks across the Internet, it’s important to run your nameserver on a secure host. Don’t run a nameserver on a big, multiuser system if you can’t trust its users. If you have certain computers that are dedicated to hosting network services but don’t permit general logins, those are good candidates for running nameservers. If you have only one or a few really secure hosts, consider running the primary nameserver on one of those because its compromise would be more significant than the compromise of the slaves.
- Software
Another factor to consider in choosing a host for a nameserver is the software the host runs. Software-wise, the best candidate for a nameserver is a host running a vendor-supported version of BIND 9.2 or 9.3 and a robust implementation of TCP/IP (preferably based on 4.3 or 4.4 BSD Unix’s networking—we’re Berkeley snobs). You can compile your own 9.2 or 9.3 BIND from the sources—it’s not hard, and the latest versions are very reliable—but you’ll probably have a tough time getting your vendor to support it. If you absolutely can’t run BIND 9, you may be able to get away with running your vendor’s port of older BIND code, such as 8.2 or 8.3, which gives you the benefit of your vendor’s support, for what that’s worth.
- Homogeneity
One last thing to take into account is the homogeneity of your nameservers. As much as you might believe in operating system “standards,” hopping between different versions of Unix can be frustrating and confusing. Avoid running nameservers on lots of different platforms if you can. You can waste a lot of time porting your scripts (or ours!) from one operating system to another or looking for the location of nslookup or named.conf on three different Unixes. Moreover, different vendors’ versions of Unix tend to support different versions of BIND, which can cause all sorts of frustration. If you need the security features of BIND 9 on all your nameservers, for example, choose a platform that supports BIND 9 for all your nameservers.
Though these are really secondary considerations—it’s more important to have a nameserver on a given subnet than to have it running on the perfect host—do keep these criteria in mind when making a choice.
Capacity Planning
If you have heavily populated networks or users who do a lot of nameserver-intensive work, you may find you need more nameservers than we’ve recommended to handle the load. Likewise, our recommendations may be fine for a little while, but as people add hosts to your networks or install new nameserver-intensive programs, you may find your nameservers bogged down by queries.
Just which tasks are “nameserver-intensive”? Surfing the Web can be, as can sending electronic mail, especially to large mailing lists. Programs that make lots of remote procedure calls to different hosts can also be nameserver-intensive. Even running certain graphical user environments can tax your nameserver. X Windows-based user environments, for example, query the nameserver to check access lists (among other things).
The astute (and precocious) among you may be asking, “But how do I know when my nameservers are overloaded? What do I look for?” An excellent question!
Memory utilization is probably the most important aspect of a nameserver’s operation to monitor. named can get very large on a nameserver that is authoritative for many zones. If named’s size, plus the size of the other processes you run, exceeds the size of your host’s real memory, your host may swap furiously (“thrash”) and not get anything done. Even if your host has more than enough memory to run all its processes, large nameservers are slow to start and reload.
Another criterion you can use to measure the load on your nameserver is the load the named process places on the host’s CPU. Correctly configured nameservers don’t use much CPU time, so high CPU usage is often symptomatic of a configuration error. Programs such as top can help you characterize your nameserver’s average CPU utilization.[*]
Unfortunately, there are no universal rules when it comes to acceptable CPU utilization. We offer a rough rule of thumb, though: 5 percent average CPU utilization is probably acceptable; 10 percent is a bit high, unless the host is dedicated to providing name service.
To get an idea of what normal figures are, here’s what top might show for a relatively quiet nameserver:
last pid: 14299; load averages: 0.11, 0.12, 0.12 18:19:08 68 processes: 64 sleeping, 3 running, 1 stopped Cpu states: 11.3% usr, 0.0% nice, 15.3% sys, 73.4% idle, 0.0% intr, 0.0% ker Memory: Real: 8208K/13168K act/tot Virtual: 16432K/30736K act/tot Free: 4224K PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 89 root 1 0 2968K 2652K sleep 5:01 0.00% 0.00% named
Okay, that’s really quiet. Here’s what top shows on a busy (though not overloaded) nameserver:
load averages: 0.30, 0.46, 0.44 system: relay 16:12:20 39 processes: 38 sleeping, 1 waiting Cpu states: 4.4% user, 0.0% nice, 5.4% system, 90.2% idle, 0.0% unk5, 0.0% unk6, 0.0% unk7, 0.0% unk8 Memory: 31126K (28606K) real, 33090K (28812K) virtual, 54344K free Screen #1/ 3 PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 21910 root 1 0 2624K 2616K sleep 146:21 0.00% 1.42% /etc/named
Another statistic to look at is the number of queries the nameserver receives per minute (or second, if you have a busy nameserver). Again, there are no absolutes here: a fast processor running FreeBSD can probably handle thousands of queries per second without breaking a sweat, while an older box running an outdated version of Unix might have problems with more than a few queries a second.
To check the volume of queries your nameserver is receiving, it’s easiest to look at the nameserver’s internal statistics. You can configure the server to write the statistics at regular intervals. For example, you can configure your nameserver to dump statistics every hour (actually, that’s the default for BIND 8 servers), and compare the number of queries received between hours:
options {
statistics-interval 60;
};BIND 9 nameservers don’t support the statistics-interval substatement, but you can use rndc to tell a BIND 9 nameserver to dump statistics on the hour—for example, in crontab:
0 * * * * /usr/local/sbin/rndc stats
You should pay special attention to peak periods. For example, Monday morning is often busy because many people like to respond to mail they’ve received over the weekend first thing on Mondays.
You might also want to take a sample starting just after lunch, when people are returning to their desks and getting back to work—all at about the same time. Of course, if your organization is spread across several time zones, you’ll have to use your best judgment to determine a busy time.
Here’s a snippet from the syslog file on a BIND 8 nameserver:
Aug 1 11:00:49 toystory named[103]: NSTATS 965152849 959476930 A=8 NS=1 SOA=356966 PTR=2 TXT=32 IXFR=9 AXFR=204 Aug 1 11:00:49 toystory named[103]: XSTATS 965152849 959476930 RR=3243 RNXD=0 RFwdR=0 RDupR=0 RFail=20 RFErr=0 RErr=11 RAXFR=204 RLame=0 ROpts=0 SSysQ=3356 SAns=391191 SFwdQ=0 SDupQ=1236 SErr=0 RQ=458031 RIQ=25 RFwdQ=0 RDupQ=0 RTCP=101316 SFwdR=0 SFail=0 SFErr=0 SNaAns=34482 SNXD=0 RUQ=0 RURQ=0 RUXFR=10 RUUpd=34451 Aug 1 12:00:49 toystory named[103]: NSTATS 965156449 959476930 A=8 NS=1 SOA=357195 PTR=2 TXT=32 IXFR=9 AXFR=204 Aug 1 12:00:49 toystory named[103]: XSTATS 965156449 959476930 RR=3253 RNXD=0 RFwdR=0 RDupR=0 RFail=20 RFErr=0 RErr=11 RAXFR=204 RLame=0 ROpts=0 SSysQ=3360 SAns=391444 SFwdQ=0 SDupQ=1244 SErr=0 RQ=458332 RIQ=25 RFwdQ=0 RDupQ=0 RTCP=101388 SFwdR=0 SFail=0 SFErr=0 SNaAns=34506 SNXD=0 RUQ=0 RURQ=0 RUXFR=10 RUUpd=34475
The number of queries received is dumped in the RQ field (in bold). To calculate the number of queries received in the hour, just subtract the first RQ value from the second one: 458332 − 458031 = 301.
Even if your host is fast enough to handle the volume of queries it receives, you should make sure the DNS traffic isn’t placing undue load on your network. On most LANs, DNS traffic will be too small a proportion of the network’s bandwidth to worry about. Over slow leased lines or dial-up connections, though, DNS traffic could consume enough bandwidth to merit concern.
To get a rough estimate of the volume of DNS traffic on your LAN, multiply the number of queries received (RQ) plus the number of answers sent (SAns) in an hour by 800 bits (100 bytes, a rough average size for a DNS message), and divide by 3,600 (seconds per hour) to find the bandwidth utilized. This should give you a feeling for how much of your network’s bandwidth is being consumed by DNS traffic.[*]
To give you an idea of what’s normal, the last NSFNET traffic report (in April 1995) showed that DNS traffic constituted just over five percent of the total traffic volume (in bytes) on its backbone. The NSFNET’s figures were based on actual traffic sampling, not calculations like ours using the nameserver’s statistics.[*] If you want to get a more accurate idea of the traffic your nameserver is receiving, you can always do your own traffic sampling with a LAN protocol analyzer.
If you find that your nameservers are overworked, what then? First, it’s a good idea to make sure that your nameservers aren’t being bombarded with queries by a misbehaving program. To do that, you’ll need to find the sources of all the queries.
If you’re running a BIND 8 nameserver, you can find out which resolvers and nameservers are querying your nameserver just by dumping the server’s statistics. BIND 8 nameservers keep statistics on a host-by-host basis, which is really useful in tracking down heavy users of your nameserver. BIND 8.2 or newer nameservers don’t keep these statistics by default; to induce them to keep host-by-host statistics, use the host-statistics substatement in your options statement, like this:[†]
options {
host-statistics yes;
};For example, take these statistics:
+++ Statistics Dump +++ (829373099) Fri Apr 12 23:24:59 1996
970779 time since boot (secs)
471621 time since reset (secs)
0 Unknown query types
185108 A queries
6 NS queries
69213 PTR queries
669 MX queries
2361 ANY queries
++ Name Server Statistics ++
(Legend)
RQ RR RIQ RNXD RFwdQ
RFwdR RDupQ RDupR RFail RFErr
RErr RTCP RAXFR RLame ROpts
SSysQ SAns SFwdQ SFwdR SDupQ
SFail SFErr SErr RNotNsQ SNaAns
SNXD
(Global)
257357 20718 0 8509 19677 19939 1494 21 0 0 0 7 0 1 0
824 236196 19677 19939 7643 33 0 0 256064 49269 155030
[15.17.232.4]
8736 0 0 0 717 24 0 0 0 0 0 0 0 0 0 0 8019 0 717 0
0 0 0 8736 2141 5722
[15.17.232.5]
115 0 0 0 8 0 21 0 0 0 0 0 0 0 0 0 86 0 1 0 0 0 0 115 0 7
[15.17.232.8]
66215 0 0 0 6910 148 633 0 0 0 0 5 0 0 0 0 58671 0 6695 0
15 0 0 66215 33697 6541
[15.17.232.16]
31848 0 0 0 3593 209 74 0 0 0 0 0 0 0 0 0 28185 0 3563 0
0 0 0 31848 8695 15359
[15.17.232.20]
272 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 272 0 0 0 0 0 0 272 7 0
[15.17.232.21]
316 0 0 0 52 14 3 0 0 0 0 0 0 0 0 0 261 0 51 0 0 0 0 316 30 30
[15.17.232.24]
853 0 0 0 65 1 3 0 0 0 0 2 0 0 0 0 783 0 64 0 0 0 0 853 125 337
[15.17.232.33]
624 0 0 0 47 1 0 0 0 0 0 0 0 0 0 0 577 0 47 0 0 0 0 624 2 217
[15.17.232.94]
127640 0 0 0 1751 14 449 0 0 0 0 0 0 0 0 0 125440 0 1602 0
0 0 0 127640 106 124661
[15.17.232.95]
846 0 0 0 38 1 0 0 0 0 0 0 0 0 0 0 809 0 37 0 0 0 0 846 79 81
-- Name Server Statistics --
--- Statistics Dump --- (829373099) Fri Apr 12 23:24:59 1996After the Global entry, each host is broken out by IP address in brackets. Looking at the legend, you can see that the first field in each record is RQ, or queries received. That gives us a good reason to look at hosts 15.17.232.8, 15.17.232.16, and 15.17.232.94, which appear to be responsible for about 88 percent of our queries.
If you’re running a BIND 9 nameserver, the only way to find out which resolvers and nameservers are sending all those darned queries is to turn on nameserver debugging. (We’ll cover this in depth in Chapter 13.) All you’re really interested in is the source IP addresses of the queries your nameserver is receiving. When poring over the debugging output, look for hosts sending repeated queries, especially for the same or similar information. That may indicate a misconfigured or buggy program running on the host or a foreign nameserver pelting your nameserver with queries.
If all the queries appear to be legitimate, add a new nameserver. Don’t put the nameserver just anywhere, though; use the information from the debugging output to help you decide where it’s best to run one. If DNS traffic is gobbling up your bandwidth, it won’t help to choose a host at random and create a nameserver there. You need to consider which hosts are sending most of the queries, then figure out how to best provide them name service. Here are some hints to help you decide:
Look for queries from resolvers on hosts that share the same file server. You could run a nameserver on that file server.
Look for queries from resolvers on large, multiuser hosts. You could run a nameserver there.
Look for queries from resolvers on another subnet. Those resolvers should be configured to query a nameserver on their local subnet. If there isn’t one on that subnet, create one.
Look for queries from resolvers on the same switch. If you connect a nameserver to the switch, the traffic won’t need to traverse the rest of the network.
Look for queries from hosts connected to each other via another, lightly loaded network. You could run a nameserver on the other network.
Adding More Nameservers
When you need to create new nameservers for your zones, the simplest recourse is to add slaves. You already know how—we went over it in Chapter 4—and once you’ve set up one slave, cloning it is a piece of cake. However, you can run into trouble by adding slaves indiscriminately.
If you run a large number of slave servers for a zone, the primary nameserver can take quite a beating serving all of the slaves’ zone transfers. There are a number of remedies for this problem, as described in the sections that follow:
Make more primary master nameservers.
Direct some of the slave nameservers to load from other slave nameservers.
Create caching-only nameservers.
Create “partial-slave” nameservers.
Primary Master and Slave Servers
Creating more primaries means extra work for you because you have to keep /etc/named.conf and the zone datafiles synchronized manually. Whether this is preferable to your other alternatives is your call. You can use tools such as rdist or rsync [*] to simplify the process of distributing the files. A distfile [†] to synchronize files between primaries might be as simple as the following:
dup-primary:
# copy named.conf file to dup'd primary
/etc/named.conf -> wormhole
install ;
# copy contents of /var/named (zone data files, etc.) to dup'd primary
/var/named -> wormhole
install ;or for multiple primaries:
dup-primary:
primaries = ( wormhole carrie )
/etc/named.conf -> {$primaries}
install ;
/var/named -> {$primaries}
install ;You can even have rdist trigger your nameserver’s reload using the special option by adding lines such as:
special /var/named/* "rndc reload" ; special /etc/named.conf "rndc reload" ;
These tell rdist to execute the quoted command if any of the files change.
You can also have some of your slaves load from other slaves. Slave nameservers can load zone data from other slave nameservers instead of loading from a primary master nameserver. The slave nameserver can’t tell if it is loading from a primary or from another slave. The only requirement is that the nameserver serving the zone transfer is authoritative for the zone. There’s no trick to configuring this. Instead of specifying the IP address of the primary in the slave’s configuration file, you simply specify the IP address of another slave.
Here are the contents of the file named.conf:
// this slave updates from wormhole, another
// slave
zone "movie.edu" {
type slave;
masters { 192.249.249.1; };
file "bak.movie.edu";
};When you go to this second level of distribution, though, keep in mind that it can take up to twice as long for the data to percolate from the primary nameserver to all the slaves. Remember that the refresh interval is the period after which the slave nameservers will check to make sure that their zone data is still current. Therefore, it can take the first-level slave servers the entire length of the refresh interval before they get a new copy of the zone from the primary master server. Similarly, it can take the second-level slave servers the entire refresh interval to get a new copy of the zone from the first-level slave servers. The propagation time from the primary master server to all of the slave servers can therefore be twice the refresh interval.
One way to avoid this is to use the NOTIFY mechanism. This feature is on by default, and will trigger zone transfers soon after the zone is updated on the primary master. We’ll discuss NOTIFY in more detail in Chapter 10.
If you decide to configure your network with two (or more) tiers of slave nameservers, be careful to avoid updating loops. If we configured wormhole to update from monsters-inc and then accidentally configured monsters-inc to update from wormhole, neither would ever get data from the primary. They would merely check their out-of-date serial numbers against each other and perpetually decide that they were both up to date.
Caching-Only Servers
Creating caching-only nameservers is another alternative when you need more servers. Caching-only nameservers are nameservers not authoritative for any zones (except 0.0.127.in-addr.arpa). The name doesn’t imply that primary and slave nameservers don’t cache—they do—but rather that the only function this server performs is looking up data and caching it. As with primary and slave nameservers, a caching-only nameserver needs a root hints file and a db.127.0.0 file. The named.conf file for a caching-only server contains these lines:
options {
directory "/var/named"; // or your data directory
};
zone "0.0.127.in-addr.arpa" {
type master;
file "db.127.0.0";
};
zone "." {
type hint;
file "db.cache";
};A caching-only nameserver can look up domain names inside and outside your zone, as can primary and slave nameservers. The difference is that when a caching-only nameserver initially looks up a name within your zone, it ends up asking one of your zone’s primary master or slave nameservers for the answer. A primary or slave answers the same question out of its authoritative data. Which primary or slave does the caching-only server ask? As with nameservers outside of your zone, it finds out which nameservers serve your zone from one of the nameservers for your parent zone. Is there any way to prime a caching-only nameserver’s cache so it knows which hosts run primary and slave nameservers for your zone? No, there isn’t. You can’t use db.cache: the db.cache file is only for root nameserver hints. And actually, it’s better that your caching-only nameservers find out about your authoritative nameservers from your parent zone’s nameservers: you keep your zone’s delegation information up to date. If you hard-wired a list of authoritative nameservers on your caching-only nameservers, you might forget to update it.
A caching-only nameserver’s real value comes after it builds up its cache. Each time it queries an authoritative nameserver and receives an answer, it caches the records in the answer. Over time, the cache will grow to include the information most often requested by the resolvers querying the caching-only nameserver. And you avoid the overhead of zone transfers: a caching-only nameserver doesn’t need to do them.
Partial-Slave Servers
In between a caching-only nameserver and a slave nameserver is another variation: a nameserver that is a slave for only a few of the local zones. We call this a partial-secondary nameserver (although probably nobody else does). Suppose movie.edu had 20 /24-sized (the old Class C) networks (and a corresponding 20 in-addr.arpa zones). Instead of creating a slave server for all 21 zones (all the in-addr.arpa subdomains plus movie.edu), we can create a partial-slave server for movie.edu and only those in-addr.arpa zones the host itself is in. If the host had two network interfaces, its nameserver would be a slave for three zones: movie.edu and the two in-addr.arpa zones.
Let’s say we scare up the hardware for another nameserver. We’ll call the new host zardoz.movie.edu , with IP addresses 192.249.249.9 and 192.253.253.9. We’ll create a partial-slave nameserver on zardoz with this named.conf file:
options {
directory "/var/named";
};
zone "movie.edu" {
type slave;
masters { 192.249.249.3; };
file "bak.movie.edu";
};
zone "249.249.192.in-addr.arpa" {
type slave;
masters { 192.249.249.3; };
file "bak.192.249.249";
};
zone "253.253.192.in-addr.arpa" {
type slave;
masters { 192.249.249.3; };
file "bak.192.253.253";
};
zone "0.0.127.in-addr.arpa" {
type master;
file "db.127.0.0";
};
zone "." {
type hint;
file "db.cache";
};This server is a slave for movie.edu and only two of the 20 in-addr.arpa zones. A “full” slave would have 21 different zone statements in named.conf.
What’s so useful about a partial-slave nameserver? They’re not much work to administer because their named.conf files don’t change much. On a nameserver authoritative for all the in-addr.arpa zones, we’d need to add and delete in-addr.arpa zones as our network changed. That can be a surprising amount of work on a large network.
A partial slave can still answer most of the queries it receives, though. Most of these queries will be for data in movie.edu and the two in-addr.arpa zones. Why? Because most of the hosts querying the nameserver are on the two networks to which it’s connected, 192.249.249/24 and 192.253.253/24. And those hosts probably communicate primarily with other hosts on their own network. This generates queries for data within the in-addr.arpa zone that corresponds to the local network.
Registering Nameservers
When you get around to setting up more and more nameservers, a question may strike you—do I need to register all of my primary and slave nameservers with my parent zone? The answer is no. Only those servers you want to make available to nameservers outside your zone need to be registered with your parent. For example, if you run nine nameservers for your zone, you may choose to tell the parent zone about only four of them. Within your network, you use all nine servers. Five of those nine servers, however, are queried only by resolvers on hosts that are configured to query them (in resolv.conf, for example). Their parent zone’s nameservers don’t delegate to them, so they’ll never be queried by remote nameservers. Only the four servers registered with your parent zone are queried by other nameservers, including caching-only and partial-slave nameservers on your network. This setup is shown in Figure 8-2.
Besides being able to pick and choose which of your nameservers are hammered by outside queries, there’s a technical motivation for registering only some of your zone’s nameservers: there is a limit to how many servers will fit in a UDP-based response message. In practice, around 10 nameserver records should fit. Depending on the data (how many servers’ names are in the same domain), you can get more or fewer.[*] There’s not much point in registering more than 10 nameservers, anyway; if none of those 10 servers can be reached, it’s unlikely the destination host can be reached.
If you’ve set up a new authoritative nameserver, and you decide it should be registered, make a list of the parents of the zones for which it’s authoritative. You’ll need to contact the administrators for each parent zone. For example, let’s say we want to register the nameserver we just set up on zardoz. To get this slave registered in all the right zones, we’ll need to contact the administrators of edu and in-addr.arpa. (For help determining who runs your parent zones, turn back to Chapter 3.)
When you contact the administrators of a parent zone, be sure to follow the process they specify (if any) on their web site. If there’s no standard modification process, you’ll have to send them the domain name of the zone (or zones) for which the new nameserver is authoritative. If the new nameserver is in the new zone, you’ll also need to give them the IP address(es) of the new nameserver. In fact, if there’s no official format for submitting the information, it’s often best just to send your parent the complete list of registered nameservers for the zone, plus any addresses necessary, in zone datafile format. This avoids any potential confusion.
Since our networks were originally assigned by the InterNIC, we used the Network Modification form at http://www.arin.net/library/templates/netmod.txt to change our registration. If they hadn’t had a template for us to use, our message to the administrator of in-addr.arpa might have read something like this:
Howdy! I've just set up a new slave name server on zardoz.movie.edu for the 249.249.192.in-addr.arpa and 253.253.192.in-addr.arpa zones. Would you please add NS records for this name server to the in-addr.arpa zone? That would make our delegation information look like: 253.253.192.in-addr.arpa. 86400 IN NS toystory.movie.edu. 253.253.192.in-addr.arpa. 86400 IN NS wormhole.movie.edu. 253.253.192.in-addr.arpa. 86400 IN NS zardoz.movie.edu. 249.249.192.in-addr.arpa. 86400 IN NS toystory.movie.edu. 249.249.192.in-addr.arpa. 86400 IN NS wormhole.movie.edu. 249.249.192.in-addr.arpa. 86400 IN NS zardoz.movie.edu. Thanks! Albert LeDomaine [email protected]
Notice that we specified explicit TTLs on the NS records. That’s because our parent nameservers aren’t authoritative for those records; our nameservers are. By including them, we’re indicating our choice of a TTL for our zone’s delegation. Of course, our parent may have other ideas about what the TTL should be.
In this case, glue data—A records for each of the nameservers—isn’t necessary because the domain names of the nameservers aren’t within the in-addr.arpa zones. They’re within movie.edu, so a nameserver that was referred to toystory.movie.edu or wormhole.movie.edu can still find their addresses by following delegation to the movie.edu nameservers.
Is a partial-slave nameserver a good nameserver to register with your parent zone? Actually, it’s not ideal because it’s authoritative for only some of your in-addr.arpa zones. Administratively, it may be easier to register only servers that slave all the local zones; that way, you don’t need to keep track of which nameservers are authoritative for which zones. All your parent zones can delegate to the same set of nameservers: your primary master and your “full” slaves.
If you don’t have many nameservers, though, or if you’re good at remembering which nameservers are authoritative for which zones, go ahead and register a partial-slave.
Caching-only nameservers, on the other hand, must never be registered. A caching-only nameserver rarely has complete information for any given zone; it just has the bits and pieces of the zone that it has looked up recently. If a parent nameserver mistakenly refers a foreign nameserver to a caching-only nameserver, the foreign nameserver sends the caching-only nameserver a nonrecursive query. The caching-only nameserver might have the data cached, but then again, it might not.[*] If it doesn’t have the data, it refers the querier to the best nameservers it knows (those closest to the domain name in the query)—which may include the caching-only nameserver itself! The poor foreign nameserver may never get an answer. This kind of misconfiguration—actually, delegating a zone to any nameserver not authoritative for that zone—is known as lame delegation.
Changing TTLs
An experienced zone administrator needs to know how to set the time to live on his zone’s data to his best advantage. The TTL on a resource record, remember, is the length of time for which any nameserver can cache that record. So if the TTL for a particular resource record is 3,600 seconds and a server outside your network caches that record, it will have to remove the entry from its cache after an hour. If it needs the same data after the hour is up, it’ll have to query one of your nameservers again.
When we introduced TTLs, we emphasized that your choice of a TTL would dictate how current you would keep copies of your data, at the cost of increased load on your nameservers. A low TTL would mean that nameservers outside your network would have to get data from your nameservers often and that the data would therefore be kept current. On the other hand, your nameservers would be peppered by the nameservers’ queries.
You don’t have to choose a TTL once and for all, though. You can—and experienced administrators do—change TTLs periodically to suit your needs.
Suppose we know that one of our hosts is about to be moved to another network. This host houses the movie.edu film library, a large collection of files our site makes available to hosts on the Internet. During normal operation, outside nameservers cache the address of our host according to the default TTL set in the $TTL control statement, or for pre-BIND 8.2 nameservers, in the SOA record. (We set the default TTL for movie.edu to three hours in our sample zone datafiles.) A nameserver caching the old address record just before the change could have the wrong address for as long as three hours. A loss of connectivity for three hours is unacceptable, though. What can we do to minimize the loss of connectivity? We can lower the TTL so that outside servers cache the address record for a shorter period. By reducing the TTL, we force the outside servers to update their data more frequently, which means that any changes we make when we actually move the system will be propagated to the outside world quickly. How short can we make the TTL? Unfortunately, we can’t safely use a TTL of 0, which should mean “don’t cache this record at all.” Some older BIND version 4 nameservers can’t cope with a TTL of 0. Small TTLs, like 30 seconds, are okay, though. The easiest change is to lower the TTL in the $TTL control statement in the db.movie.edu file. If you don’t place an explicit TTL on resource records in the zone datafiles, the nameserver applies this to each resource record. If you lower the default TTL, though, the new, lower default applies to all zone data, not just the address of the host being moved. The drawback to this approach is that your nameserver will be answering a lot more queries because the querying servers will cache all the data in your zone for a shorter period. A better alternative is to put a different TTL only on the affected address record.
To add an explicit TTL on an individual resource record, place it before the IN in the class field. The TTL value is in seconds by default, but you can also specify units of m (minutes), h (hours), d (days), and w (weeks), just as you could in the $TTL control statement. Here’s an example of an explicit TTL from db.movie.edu:
cujo 1h IN A 192.253.253.5 ; explicit TTL of 1 hour
When giving out answers, a slave supplies the same TTL a primary master does; that is, if a primary gives out a TTL of one hour for a particular record, a slave will, too. The slave doesn’t decrement the TTL according to how long it has been since it loaded the zone. So, if the TTL of a single resource record is set smaller than the default, both the primary and slave nameservers give out the resource record with the same, smaller TTL. If the slave nameserver has reached the expiration time for the zone, it expires the whole zone. It will never expire an individual resource record within a zone.
So BIND allows you to put a small TTL on an individual resource record if you know that the data is going to change shortly. Thus, any nameserver caching that data caches it only for a brief time. Unfortunately, while the nameserver makes tagging records with a small TTL possible, most administrators don’t take the time to do it. When a host changes addresses, you often lose connectivity to it for a while.
More often than not, the host having its address changed is not one of the main hubs on the site, so the outage affects few people. If one of the mail hubs or a major web server or ftp archive—like the film library—is moving, though, a day’s loss of connectivity may be unacceptable. In cases like this, the administrator should plan ahead and reduce the TTL on the data to be changed.
Remember that the TTL on the affected data will need to be lowered before the change takes place. Reducing the TTL on a workstation’s address record and changing the workstation’s address simultaneously may do you little or no good; the address record may have been cached seconds before you made the change and may linger until the old TTL times out. You must also be sure to factor in the time it’ll take your slaves to load from your primary. For example, if your default TTL is 12 hours and your refresh interval is 3 hours, be sure to lower the TTLs at least 15 hours ahead of time, so that by the time you move the host, all the old, longer TTL records will have timed out. Of course, if all your slaves are using NOTIFY, the slaves shouldn’t take the full refresh interval to synch up.
Changing Other SOA Values
We briefly mentioned increasing the refresh interval as a way to offload your primary nameserver. Let’s discuss refresh in a little more detail and go over the remaining SOA values, too.
The refresh value, you’ll remember, controls how often a slave checks whether its zone data is up to date. The retry value then becomes the refresh time after the first failure to reach a master nameserver. The expire value determines how long zone data can be held before it’s discarded, when a master is unreachable. Finally, on pre-BIND 8.2 nameservers, the minimum TTL sets how long zone information may be cached. On newer nameservers, the last SOA field is the negative caching TTL.
Suppose we decide we want the slaves to pick up new information every hour instead of every three hours. We change the refresh value to one hour in each of the zone datafiles (or with the -o option to h2n). Since retry is related to refresh, we should probably reduce retry, too—to every 15 minutes or so. Typically, retry is less than refresh, but that’s not required.[*] Although lowering the refresh value speeds up the distribution of zone data, it also increases the load on the nameserver from which data is being transferred because the slaves will check more often. The added load isn’t much, though; each slave makes a single SOA query during each zone’s refresh interval to check its master’s copy of the zone. So with two slave nameservers, changing the refresh time from three hours to one hour generates only four more queries (per zone) to the master in any three-hour span.
If all your slaves run BIND 8 or 9, and you use NOTIFY, of course, refresh doesn’t mean as much. But if you have even one BIND version 4 slave, your zone data may take up to the full refresh interval to reach it.
Some versions of BIND may refresh more often than the refresh interval. All modern versions of BIND (from version 4.9 on) will wait a random number of seconds between one-half (for BIND 8) or three-quarters (BIND 9) of the refresh interval and the full refresh interval to check serial numbers.
Expiration times on the order of a week—longer if you frequently have problems reaching your updating source—are common. The expiration time should always be much larger than the retry and refresh intervals; if the expiration time is smaller than the refresh interval, your slaves will expire their data before trying to load new data. BIND 8 complains if you set an expiration time less than refresh plus retry, less than twice retry, less than seven days, or greater than six months. (BIND 9 doesn’t complain—yet.) Choosing an expiration time that meets all BIND 8’s criteria is a good idea in most situations.
If your zone’s data doesn’t change much, you might consider raising its default TTL. Default TTLs are commonly a few hours to one day. One day is about the longest value that makes sense for a TTL. If it’s longer than that, you may find yourself unable to change bad, cached data in a reasonable amount of time.
Planning for Disasters
It’s a fact of life on a network that things go wrong. Hardware fails, software has bugs, and people occasionally make mistakes. Sometimes this results in minor inconveniences, like having a few users lose connections. Sometimes the results are catastrophic and involve the loss of important data and gainful employment.
Because the Domain Name System relies so heavily on the network, it is vulnerable to network outages. Thankfully, the design of DNS takes into account the imperfection of networks: it allows for multiple, redundant nameservers; retransmission of queries; retrying zone transfers; and so on.
DNS doesn’t protect itself from every conceivable calamity, though. There are types of network failures —some of them quite common—that DNS doesn’t or can’t protect against. But with a small investment of time and money, you can minimize the threat of these problems.
Outages
Power outages, for example, are relatively common in many parts of the world. In some parts of the United States, thunderstorms or tornadoes may cause a site to lose power, or to have only intermittent power, for an extended period. Elsewhere, typhoons, volcanoes, or construction work may interrupt your electrical service. And you never know when those of you in California might lose power in a rolling blackout from a lack of electrical capacity.
If all your hosts are down, of course, you don’t need name service. Quite often, however, sites have problems when power is restored. Following our recommendations, they run their nameservers on file servers and big, multiuser machines. And when the power comes up, those machines are naturally the last to boot because all those disks need to be checked and fixed first! Which means that all the on-site hosts that are quick to boot do so without the benefit of name service.
This can cause all sorts of wonderful problems, depending on how your hosts’ startup files are written. Unix hosts often execute some variant of:
/usr/sbin/ifconfig lan0 inet `hostname` netmask 255.255.128.0 up /usr/sbin/route add default site-router 1
to bring up their network interface and add a default route.
Using hostnames in commands (`hostname` expands to the local hostname,
and site-router is the name of the local router)
is admirable because it allows administrators to change devices’ IP
addresses without changing all the startup files on-site.
Unfortunately, the route command fails without name service. The ifconfig command fails only if the local host’s name and IP address don’t appear in the host’s /etc/hosts file, so it’s a good idea to leave at least that data in each host’s /etc/hosts.
By the time the startup sequence reaches the route command, the network interface will be up, and the host will try to use name service to map the name of the router to an IP address. And since the host has no default route until the route command is executed, the only nameservers it can reach are those on the local subnet.
If the booting host can reach a working nameserver on its local subnet, it can execute the route command successfully. Quite often, however, one or more of the nameservers it can reach aren’t yet running. What happens then depends on the contents of resolv.conf.
BIND resolvers fall back to the host table only if there is just one nameserver listed in resolv.conf (or if no nameserver is listed, and the resolver defaults to using a nameserver on the local host). If only one nameserver is configured, the resolver queries it, and if the network returns an error each time the resolver sends a query, the resolver falls back to searching the host table. The errors that cause the resolver to fall back include:
Receipt of an ICMP network unreachable message
Inability to send the UDP packet (e.g., because networking is not yet running on the local host)[*]
If the host running the one configured nameserver isn’t running at all, though, the resolver doesn’t receive any errors. The nameserver is effectively a black hole. After about 75 seconds of trying, the resolver just times out and returns a null answer to the application that called it. Only if the nameserver host has actually started networking—but not yet started the nameserver—does the resolver get an error: an ICMP port unreachable message.
Overall, the single-nameserver configuration does work if you have nameservers available on each network, but not as elegantly as we might like. If the local nameserver hasn’t come up when a host on its network reboots, the route command fails.
This may seem awkward, but it’s not nearly as bad as what happens with multiple nameservers. With multiple servers listed in resolv.conf, BIND never falls back to the host table after the primary network interface has been ifconfig‘ed. The resolver simply loops through the nameservers, querying them until one answers, or the 75-plus-second timeout is reached.
This is especially problematic during bootup. If none of the configured nameservers is available, the resolver times out without returning an IP address, and adding the default route fails.
Recommendations
Our recommendation, as primitive as it sounds, is to hardcode the IP address of the default router into the startup file or an external file (many systems use /etc/defaultrouter). This ensures that your host’s networking starts correctly.
An alternative is to list just a single, reliable nameserver on your host’s local network in resolv.conf. This allows you to use the name of the default router in the startup file, as long as you make sure that the router’s name appears in /etc/hosts (in case your reliable nameserver isn’t running when the host reboots). Of course, if the host running the reliable nameserver isn’t running when your host reboots, all bets are off. You won’t fall back to /etc/hosts because there won’t be any networking running to return an error to your host.
If your vendor’s version of BIND allows configuration of the order in which services are queried or falls back from DNS to /etc/hosts if DNS doesn’t find an answer, take advantage of it! In the former case, you can configure the resolver to check /etc/hosts first, and then keep a “stub” /etc/hosts file on each host, including the default router and the local host’s name. In the latter situation, just make sure such a “stub” /etc/hosts exists; no other configuration should be necessary.
However, there’s some danger in using /etc/hosts files: unless you take care to keep the files up to date, the information in them may become stale. Maintaining stub /etc/hosts files on many hosts is a great application for rsync.
And what if your default route is added correctly but the nameservers still haven’t come up? This can affect sendmail, NFS, and a slew of other services. sendmail won’t canonicalize hostnames correctly without DNS, and your NFS mounts may fail.
The best solution to this problem is to run a nameserver on a host with uninterruptible power. If you rarely experience extended power loss, battery backup might be enough. If your outages are longer, and name service is critical to you, you should consider an uninterruptible power system (UPS) with a generator of some kind.
If you can’t afford luxuries like these, you might just try to track down the fastest-booting host around and run a nameserver on it. Hosts with filesystem journaling should boot especially quickly because they usually don’t need to check and repair their disks. Hosts with small filesystems should also boot quickly because they don’t have many disks to check.
Once you’ve located the right host, you’ll need to make sure the host’s IP address appears in the resolver configurations of all of your hosts that need full-time name service. You’ll probably want to list the backed-up host last because during normal operation, hosts should use the nameserver closest to them. Then, after a power failure, your critical applications will still have name service, albeit with a small sacrifice in performance.
Coping with Disaster
When disaster strikes, it really helps to know what to do. Knowing to duck under a sturdy table or desk during an earthquake can save you from being pinned under a toppling monitor. Knowing how to turn off your gas can save your house from conflagration.
Likewise, knowing what to do in a network disaster (or even just a minor mishap) can help you keep your network running. Living out in California, as we do, we have some first-hand experience with disaster, and some suggestions.
Long Outages (Days)
If you lose network connectivity for a long time, your nameservers may begin to have problems. If they lose connectivity to the root nameservers for an extended period, they’ll stop resolving queries outside their authoritative zone data. If the slaves can’t reach their master, sooner or later they’ll expire the zone.
In case your name service really goes haywire because of the connectivity loss, it’s a good idea to keep a site-wide or workgroup /etc/hosts around. In times of dire need, you can move resolv.conf to resolv.bak, kill the local nameserver (if there is one), and just use /etc/hosts. It’s not flashy, but it’ll get you by.
As for slaves, you can reconfigure a slave that can’t reach its master to temporarily run as a primary. Just edit named.conf and change the type substatement in the zone statement from slave to master, then delete the masters substatement. If more than one slave for the same zone is cut off, you can configure one as a primary temporarily and reconfigure the others to load from the temporary primary.
Really Long Outages (Weeks)
If an extended outage cuts you off from the Internet—say for a week or more—you may need to restore connectivity to root nameservers artificially to get things working again. Every nameserver needs to talk to a root nameserver occasionally. It’s a bit like therapy: the nameserver needs to contact a root periodically to regain its perspective on the world.
To provide root name service during a long outage, you can set up your own root nameservers, but only temporarily. Once you’re reconnected to the Internet, you must shut off your temporary root servers. The most obnoxious vermin on the Internet are nameservers that believe they’re root nameservers but don’t know anything about most top-level domains. A close second is the Internet nameserver configured to query—and report—a false set of root nameservers.
That said, and with our alibis in place, here’s what you have to do to configure your own root nameserver. First, you need to create db.root, the root zone datafile. The root zone will delegate to the highest-level zones in your isolated network. For example, if movie.edu were to be isolated from the Internet, we might create a db.root file for toystory that looks like this:
$TTL 1d
. IN SOA toystory.movie.edu. al.movie.edu. (
1 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ) ; Negative TTL
IN NS toystory.movie.edu. ; toystory is the temp. root
; Our root only knows about movie.edu and our two
; in-addr.arpa domains
movie.edu. IN NS toystory.movie.edu.
IN NS wormhole.movie.edu.
249.249.192.in-addr.arpa. IN NS toystory.movie.edu.
IN NS wormhole.movie.edu.
253.253.192.in-addr.arpa. IN NS toystory.movie.edu.
IN NS wormhole.movie.edu.
toystory.movie.edu. IN A 192.249.249.3
wormhole.movie.edu. IN A 192.249.249.1
IN A 192.253.253.1Then, we need to add the appropriate line to toystory’s named.conf file:
// Comment out hints zone
// zone . {
// type hint;
// file "db.cache";
// };
zone "." {
type master;
file "db.root";
};We then update all of our nameservers (except the new, temporary root) with a db.cache file that includes just the temporary root nameserver (it’s best to move the old root hints file aside; we’ll need it later, once connectivity is restored).
Here are the contents of the file db.cache:
. 99999999 IN NS toystory.movie.edu. toystory.movie.edu. 99999999 IN A 192.249.249.3
This process keeps movie.edu name resolution going during the outage. Once Internet connectivity is restored, we can delete the root zone statement from named.conf, uncomment the hint zone statement on toystory, then restore the original root hints files on all other nameservers.
[*] top is a very handy program, written by Bill LeFebvre, that gives you a continuous report of which processes are sucking up the most CPU time on your host. It’s included with many versions of Unix and Linux. If your operating system doesn’t include top, you can find the most recent version at http://www.unixtop.org/.
[*] For a nice package that automates the analysis of BIND’s statistics, look for Marco d’Itri’s bindgraph in the DNS Resources Directory’s tools page, http://www.dns.net/dnsrd/tools.html.
[*] We’re not sure how representative of the current state of the Internet these numbers are, because it’s extremely difficult to wheedle equivalent numbers out of the commercial backbone providers that succeeded the NSFNET.
[†] BIND 9 doesn’t support the host-statistics substatement—or keeping per-host statistics, for that matter.
[*] rsync is a remote file synchronization program that transmits only the differences between files. You can find out more about it at http://rsync.samba.org.
[†] The file rdist reads to find out which files to update.
[*] The domain names of the Internet’s root nameservers were changed because of this. All the roots were moved into the same domain, root-servers.net, to take maximum advantage of domain-name compression and to allow information about as many roots as possible to be stored in a single UDP packet.
[*] More importantly, even if the caching-only nameserver had the answer cached, it would respond non-authoritatively. The nameserver that was referred to it, expecting an authoritative answer, would ignore the response.
[*] Actually, BIND 8 servers will warn you if refresh is set to less than 10 times the retry interval.