Updating the REFID format
Related Topics: RefidFormat
, NTP-BUG 278
, NTP-BUG 505
The current format for the Refid is either:
.ABCD. for a reference clock
- Something that looks like an IPv4 address
There are a few problems with this:
- The IPv4 format takes up a lot of room
- We now support IPv6 and that address is lots longer
- There is no real reason to use the refid for address information
It has been proposed that we change the non-refclock format to something "more compressed", like hex or base64.
Instead, I'm leaning toward Dan Mahoney's suggestion in comment #9 of NTP-BUG 278
, which is an IPv4 format consisting of 255 followed by 3 bytes from the hash.
The following initial octets are potentially available:
- 127 (at S2+)
- 240-247 (011110xxx)
- 248-251 (0111110xx)
- 252-253 (01111110x)
- 254 (011111110)
- 255 (011111111)
but see https://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml
In addition to the original plan, the following S2+ REFIDs will be used.
255: IPv6 refid
The IPv6 hash is 4 bytes of the hash, layed out as
We will also use
to indicate an IPv6 hash.
Related Topics: NTP-BUG 278
254: Leap Smear
If the server is offering a smeared leap-second, then while the server is effecting a leap smear it will send packets with a REFID of the form
is a 2:22 signed integer:fraction that indicates the current offset to UTC. Put another way, this is the amount of leap smear in the offered timestamp.
files in the distribution for more information, along with NTP-BUG 2855
253: Suggested REFID nonce
server can offer a "Suggested REFID" to its clients.
The format of this REFID is
are effectively random numbers.
This is a mechanism to prevent outsiders from easily identifying the IPv4 address that
While we'd have the most bits available to us using 240-247, it's not clear that we need that many bits for the RefID because when using the RefID for loop detection we're only worried about immediate neighbors. In this case, the question is what are the odds that the top 3 bytes of the MD5 hash of two IPv6 addresses will match? If this is a problem, adding 3 more bits will reduce the chance of collision by a factor of 8. Is that a significant reduction?
And if there is a collision and it's a problem, the solution may be to manually assign a RefID. In this case, one could assign a RefID using 254. as the initial octet.
Some other issues that have been brought up by DannyMayer
, in regard to recent events:
- Also needs to deal with the endian problem (not all platforms represent their bits and bytes in the same order).
- The internal binary representation of the refid is 4 bytes, but 16 byte IPv6 addresses won't fit.
- The spec calls for using the top 32 bits of the MD5 hash
- If we do this, we might need to use the 4to6 representation of the IPv4 address, effectively converting that into an IPv6 address, which then gets treated the same way.
- I would also like to voice my support for the use of base64 for the presentation of the refid to the user, since this is more compact
- 4 bytes = 8 nibbles = 8 hex characters
- 4 bytes = 32 bits = 5.33333 base64 characters
- Either is more compact than our current IPv4 presentation, but IMO the more space we can save, the better.
- I'm not sure what to do about representations for refclocks.
- 12 Jul 2004
We may not be able to use 4to6 if the target is a platform where somebody said "--disable-ipv6" (or whatever).
be useful to ID the "source" for the refid:
- is a refclock
- is a Unicast client
- is a Broadcast client
- is a manycast/multicast client (is it OK to lump these?)
- 13 Jul 2004
With regards to the 4to6 translation, I think we can do that regardless of whether or not the machine supports IPv6 or not. I think it boils down to just tacking on a certain string to the front of the IPv4 address, which might even be all zeros. We might need our own routine to calculate that, so that we're not dependant on the routines in the IP stack, but I think that should be do-able.
Otherwise, I think Harlan has some excellent suggestions here for the characters which might be displayed in front and/or behind the refid, to help identify refclocks and client types.
- 13 Jul 2004
Based on some discussions that we've had on both hackers as well as chat it looks like the best initial cut would be to create a different option at least for now, that I have tentatively called xpeers (-x on the command line) which will put the address where the refid currently is when it's not one of the standard items like .GPS. MCAST, or other clocks.
Because of the limited space on a line that would allow for the longer IPv6 addresses this means making the line longer than 80 characters (hence the -x). However, at least a full IPv6 address will appear. The refid field will not show up in this display. This will be experimental for now (hence the -x).
A second issue has been raised here that the refid is not enough for loop prevention since you could be contact the same host over different IP addresses. It is not clear how we will solve that issue.
- 16 Jul 2004
Speaking only for myself, I think we need a refid that is guaranteed unique for a given executing ntpd image, regardless of which interface you're coming in over or what hostnames that machine may be known by.
You could potentially have multiple copies of ntpd running on a single machine, each listening to different interfaces. I think each of these processes should have a unique refid to distinguish it from the others.
How we achieve these things, is a question I do not know. But I think that this is the only way we're going to be able to solve the loop prevention issue.
- 19 Jul 2004
You cannot have more than one ntpd server running on any system as this would cause the clock to be changed by multiple processes and cause clock instability. You cannot run ntpd and some other server that might change the clock.
- 21 Jul 2004
(Should we be discussing this in a ...Dev topic instead?)
The REFID is used by stratum 1 servers to identify their source of time, and at lower strata to prevent loops.
In the "old days" the lower strata servers used their IP address. This is no longer sufficient or appropriate, as:
- machines often have multiple addresses
- the increasing use of IPv6 means we don't have a convenient 4-byte number; the draft SNTP spec says the top 4 bytes of the MD5 hash of the IPv6 address should be used.
- 01 Nov 2004
This has just come up again in a question from Microsoft. We need to revisit this again, particularly in light of the discussions on auditability. If we don't have a unique refid per system (as opposed to interface) we can't ensure that we are preventing looping. This should be discussed in the IETF WG now that we have one.
- 16 Jun 2005
For the base64 representation we could use
(and I'm not attached to
). The current representation takes up 15 characters, and with base64 we should be able to get that down to 6 characters (if I did the math correctly), 7 if we prepend a character to tell us how we are talking to that system (which would save us another character as we would not be needing the
I'm pretty sure we use network byte order to move these things around.
- 17 Jun 2005 --- To summarize (based on recent email and conversation with Dave), there are 2 issues with reference ids: their purpose and their interpretation.
A primary purpose of the refid is loop-detection/avoidance. Given two machines, A and B, the refid is used so that if A tries to get time from B it can tell if B is already sync'd to A.
As for the interpretation of the refid, there are 3 cases, all based on the stratum.
- S0: the refid contains a KOD code
- S1: the refid contains the refclock type
- S2+: the refid contains a "token" that IDs the server we are syncing with
In the past, the S2+ refid was the IPv4 address of the server. This is sometimes useful, as it makes it easy to see who a remote server is talking to.
It is arguably Bad to display the md5 hash remnant of an IPv6 address as a network address.
The problem as I see it is that the receiving side does not know if the S2+ refid is IPv4 or not, and without this knowledge cannot properly decode the refid.
Additionally, a refclock can now advertise itself at S2+.
- 17 Jun 2005
This issue is really bigger than the mere format of the Ref-ID because IPv4 has led to a situation where people use the
column in the ntpq peer billboard as a "next stratum up" display. Although some of you make take the position that "the Ref-ID was never intended to be used that way so tough luck", the fact is we should not capriciously break existing functionality as it has evolved.
As for the Ref-ID itsself:
- A Ref-ID which is a hash / number / token / whatever which is opaque to the user will lead to complaints because it will make quick diagnosis more difficult. An additional
ntpq command which displays the talley code, remote host, hostname of that remote hosts sys_peer, and a few more columns (such as stratum, when, poll, reach), would do much to offset the use of an opaque Ref-ID.
ntpd should have a GUID (*G*lobally *U*nique *ID*entifier), so that it is uniquely identifyable regardless of how many IP-addresses it is bound to, that serves as its Ref-ID. These GUIDs should be cryptographically strong enough to reasonably reduce the chance of GUID collision.
-- 12 May 2008
Looking at the IANA IPv4 Special-Purpose Address Registry
, it seems like all of 240.0.0.0/4 (0xF0.0.0.0) is reserved "for future use". So if there is a move to give special meaning to the bytes of the existing four-byte Refid field (and we're not worried about that range someday getting used for some incompatible purpose), it might be possible to keep 28 bits of the md5 hash (if the extra seemed useful to avoid clashes), or to use the second nibble of the first byte to announce some other information (e.g. S2+ refclocks). (Note that that page doesn't list 255.0.0.0 as reserved separately from the 240.0.0.0/4 reservation; only 255.255.255.255/32 has a separate mention.)
(For completeness, two other network ranges listed on that page that could conceivably be used for special-case Refid values are 0.0.0.0/8 and 127.0.0.0/8 . There are a handful of other ranges that look like they would never be used as the IP number of an actual server, but those ranges are small enough that it doesn't seem like they'd be useful for the special cases we're talking about here.)
Certainly it would also be nice to figure out some (backward-compatible) way to send a longer Refid (or a new field with the upstream information, separate from the Refid) to avoid these problems in future versions of NTP.
Possible Refid changes were covered in a thread on the ntp hackers mailing list in May 2015 (with the subject line of NTP Development Snapshot 4.3.33 Released
). Ideas discussed in this thread include:
- In the short run, using a reserved value (e.g. 255) for the first byte within in the existing 4-byte Refid field (followed by 3 bytes of the hash) seems the best way to avoid confusion between IPv6 hashes and "real" IPv4 address.
- There was a proposal to try to handle specially the particular case where a server has two published IP numbers, one IPv4 and one IPv6, in which case the generated Refid for that server be based on a combination of both of those addresses. The idea would be to avoid one common category of failures of loop-detection, where a single server is accessible over both IPv4 and IPv6 but otherwise has a simple connectivity profile. Implementing such special handling might involve reserving a couple bits at the top of the 3-byte field to flag which sort of hashing/combining was used to generate that particular Refid.
- In the long run, to serve the loop-detection function, there will need to be a change to the protocol so that a particular ntpd server comes up with its own unique ID which it then publishes to all its clients (who then propagate it to their own clients), rather than the current situation where each server comes up with its own Refid for upstream servers. Probably this could be only 10 or 20 bits in size and still work for loop detection (and thus could even get encoded within the current four-byte Refid field for backwards compatibility). However, it might also be worth having this value encode information about the refclock used by a particular S1 server, so that client ntpds can try to ensure a diversity of upstream refclocks (but doing so would require more bits).
- With that change in place, a separate extension to the protocol would be needed so that clients could continue to learn the upstream servers' current sync sources' IP info. This would replace that aspect of the current Refid for IPv4 sources, and also support obtaining the full IP address in the IPv6 case, or perhaps could send the hostname instead of IP number. (However, some of these cases might involve too much data to fit in a mode 6 response packet.)
Also, it's worth noting that the recently implemented LeapSmearForNTPv4
makes use of a special Refid format, flagged by a 254 in the first byte.