9. Troubleshooting NTP

First things first.

Please be sure you have your NTP network properly designed.

Also make sure you have NTP properly configured. If your ntp.conf file does not have the iburst option specified in it, the odds are real good you have not properly configured NTP.

The information from 9.4. Checking ntpd 's status should be an excellent guide to how well ntpd is working for you and where to look for problems.

9. Troubleshooting NTP

9.4. Checking `ntpd` 's status

Run ntpq -p HOSTNAME, or one of the web-based utilities at 9.5. On-line Troubleshooting Utilities, to see the status of ntpd on HOSTNAME (without HOSTNAME the local host is queried). Check the official documentation for a detailed description of the ntpq utility (http://www.eecis.udel.edu/~mills/ntp/html/ntpq.html). It will report something like this:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ff05::101       .MCST.          16 u    -   64    0    0.000    0.000 4000.00
*example.site.co .PPS.            1 u  320 1024  377    1.955   -1.234   1.368

The very first column: contains the "tally code" character. Short overview:

* the source you are synchronized to (syspeer)
# source selected, distance exceeds maximum value
o the PPS(Pulse Per Second) source if your ntpd (ppspeer, only if you have a PPS capable system and refclock)
+ candidate, i.e. it is considered a good source
- outlyer, i.e. quality is not good enough
x falseticker, i.e. this one is considered to distribute bad time
blank: source discarded, failed sanity

See the Select field of the Peer status word on the NTP Event Messages and Status Words page for more information on the tally codes.

remote: the hostname or IP of the remote machine.

refid: the identification of the time source to which the remote machines is synced. May be (for example) a radio clock or another ntp server)

st: the stratum of the remote machine. 16 is "unsynchronized". 0 is the best value, that could be (for example) a radio clock or the ntp servers private caesium clock (see http://www.eecis.udel.edu/~mills/ntp/html/index.html#intro for more information about ntp in general).

t

types available:

l = local (such as a GPS, WWVB)
u = unicast (most common)
m = multicast
b = broadcast
- = netaddr

when: how many seconds since the last poll of the remote machine.

poll: the polling interval in seconds.

reach: an 8-bit left-rotating register. Any 1 bit means that a "time packet" was received. The right most bit indicate the status of the last connection with the NTP server. It is Octal number. Use calculator in progammer interface to translate from OCT to BIN: For example 377 translates to 11111111. Each 1 means a successful connection to the NTP server. If you just start a NTP service, and it connects successfully with its server, this number will change as follows (assuming connectivity is good):

00000001 = 001
00000011 = 003
00000111 = 007
00001111 = 017
00011111 = 037
00111111 = 077
01111111 = 177
11111111 = 377

delay: the time delay (in milliseconds) to communicate with the remote.

offset: the offset (in milliseconds) between our time and that of the remote.

jitter: the observed jitter (in milliseconds) of time with the remote.

9.5. On-line Troubleshooting Utilities

The following on-line troubleshooring utility is available for testing an ntpd from an "outside" IP address:

Test the time server at the IP address you are browsing from (time, peers, variables)

9.6. Check the `syslog` output

Look at the contents of your syslog output file. There is a good chance that ntpd has output some information describing any problems it has encountered. Windows users have to take a look into the application event log and look out for entries from ntpd.

9.7. Problems with RESTRICT lines

Many people have difficulties with using RESTRICT. They want to set themselves up to be as secure as possible, so they create an extremely limited default RESTRICT line in their /etc/ntp.conf file, and then they find that they can't talk to anyone.

If you're having problems with your server, in order to do proper debugging, you should turn off all RESTRICT lines in your /etc/ntp.conf file, and otherwise simplify the configuration as much as possible, so that you can make sure that the basic functions are working correctly.

Once you get the basics working, try turning back on various features, one-by-one. When turning on the RESTRICT features, make sure that you have read, understood, and followed the instructions found in AccessRestrictions.

9.7.1. Problems with _RESTRICT NOTRUST_

The behavior of NOTRUST changed between versions 4.1 and 4.2.

In 4.1 (and earlier) NOTRUST meant "Don't trust this host/subnet for time".

In 4.2 (and later) NOTRUST means "Ignore all NTP packets that are not cryptographically authenticated." This forces remote time servers to authenicate themselves to your (client) ntpd. See ConfiguringAutokey for information about configuring NTP Authentication.

Please note that most servers are not set up to do cryptographic authentication. Therefore, if you use RESTRICT NOTRUST in your configuration file, you will most likely be configuring your machine to query one or more upstream servers but then throw away any answer that they may send you. This may result in your client sending out one or more packets per second to each of your configured upstream servers, and that would be considered to be "seriously unfriendly".

Many server operators would be likely to firewall themselves off from you (and perhaps the rest of your network), to try to protect themselves against this kind of abuse.

See the page at Flawed Routers Flood University of Wisconsin Internet Time Server to get an idea of how bad this can be, when a vendor mis-configures commodity-grade hardware and causes all their devices in the field to start bombarding time servers with a packet every second. See http://people.freebsd.org/~phk/dlink/ for a more recent example.

Do NOT use RESTRICT NOTRUST unless you know what it means and you know how to use it properly!!!

9.8. Check the NTP port

The first thing to do is to make sure that UDP port 123 is open on all firewalls between you and the remote time servers that you wish to synchronize to. See 9.5. On-line Troubleshooting Utilities for browser-based tests.

When trying to debug problems using ntpdate and ntpq, note that these utilities may use unprivileged high-numbered ports, while ntpd requires full bidirectional access to the privileged UDP port 123. So, ntpdate -u may work, but ntpd may not. Or ntpq may work, but ntpd may not. OpenNTPD also uses high-numbered source ports so if it is able to synchronize but ntpd is not, it is very probable that the incoming UDP port 123 is blocked.

If you're going to run ntpd, you need to fix your network/firewall/NAT so that ntpd can have full unrestricted access to UDP port 123 in both directions. However, this may not be allowed by your firewall administrators.

If this is not possible, you may need to run ntpd on the firewall itself, so that it can have full unrestricted access to UDP port 123 in both directions, and then have it serve time to your internal clients. However, this may also be disallowed.

If that's not possible, your only other option may be to buy the necessary hardware to connect to one or more of your own computers and run your own Stratum 1 time server (typically $200-300 for the radio or GPS receiver hardware, plus the computer to connect it to), or buy a pre-packaged Stratum 1 time server (frequently $1000-2000 or more). With your own Stratum 1 time server, you can sync your internal clients to it, it will get its signal via a radio signal from WWV/WWVB/DCF77/CHU/etc... (depending on where you live) or maybe a GPS or CDMA radio signal, and no packets will be required to cross your firewall on UDP port 123.

Only your management and your firewall administrators will be able to tell you which options are feasible.

9.9. ntp.conf and dhcp

If your /etc/ntp.conf is being automatically overwritten, this may be due to DHCP. Either run your dhcpd (dhcp server) with the dhcpd.conf option "option ntp-servers <your ntp server>;", or run your dhcpcd (dhcp client) with the -N arg to prevent ntp.conf from being rewritten at all.

9.10. synchronizing ntp with a server running w32time

In the base release of Windows server 2003 running w32time a hotfix is required otherwise ntp cannot reach (and therefore not sync with) that server.

This hotfix is available from Microsoft on request only, see http://support.microsoft.com/?kbid=830092. The hotfix is included in SP1 and later service packs.