GNUAutoGenConversion < Dev

r58 - 2013-08-07 - 09:18:08 - HarlanStennYou are here: NTP >
Dev Web > DevelopmentIssues > GNUAutoGenConversion
NTP users are strongly urged to take immediate action to ensure that their NTP daemons are not susceptible to being used in distributed denial-of-service (DDoS) attacks. Please also take this opportunity to defeat denial-of-service attacks by implementing Ingress and Egress filtering through BCP38.

ntp-4.2.8p15 was released on 23 June 2020. It addresses 1 medium-severity security issue in ntpd, and provides 13 non-security bugfixes over 4.2.8p13.

Please see the NTP Security Notice for vulnerability and mitigation details.
Are you using Autokey in production? If so, please contact Harlan - he's got some questions for you.
---+!! GNU <nop>AutoGen Conversion

Starting with 4.2.4 we are using [[https://www.gnu.org/software/autogen/][GNU !AutoGen]] to provide consistent options processing, easier enum/flag/descriptor maintenance, and eventually documentation (including html, man, and info pages, as appropriate) for several programs and files.

__Related Topics:__ GNUAutoGenImprovements, GenLocInfo

%TOC%

---++ Conversion Progress Status

The conversion status is as follows:

| *Item* | *Directory* | * =.def= * | * =.man= * | * =.mdoc= * | * =.texi= * | * =.html= * | *Reconcile with %BR% !FreeBSD* | *Reconcile with %BR% DLM's html pages* | *Man Section* | *Notes* |
| =.ntprc= | ? | Not started | | | | | | N/A | 5 | |
| =ntp.audio= | ? | Not started | | | | | | N/A | 5 | |
| =ntp.conf= | =ntpd= | %Y% | %ICON{wip}% | %Y% | %ICON{wip}% | %ICON{wip}% | | | 5 | What about the simulator? |
| =ntp.keys= | =ntpd= | %Y% | %ICON{wip}% | %Y% | %ICON{wip}% | %ICON{wip}% | | | 5 | |
| =ntp-keygen= | =util= | %Y% | | %Y% | %ICON{wip}% | %ICON{wip}% | | | =loc= | |
| =ntp-wait= | =util= | %Y% | | %Y% | %ICON{wip}% | | N/A | | =loc= | |
| =ntpd= | =ntpd= | %Y% | | %Y% | %ICON{wip}% | | | | =loc= | |
| =ntpdsim= | =ntpd= | %Y% | | | | | | | =loc= | Remember =ntp.conf= |
| =ntpdate= (old) | =ntpdate= | Will be deprecated | | | | | | | =loc= | |
| =ntpdate= (new) | ? | Not started | | | | | | | =loc= | |
| =ntpdc= | =ntpdc= | %Y% | | %Y% | | | | | =loc= | |
| =ntpq= | =ntpq= | %Y% | | %Y% | %ICON{wip}% | | | | =loc= | |
| =ntpsnmpd= | =ntpsnmpd= | %Y% | | %ICON{wip}% | %ICON{wip}% | | | | =loc= | |
| =ntptime= | =util= | Not started | | | | | | | =loc= | |
| =ntptrace= | =scripts= | Not started | | | | | | | =loc= | |
| =sntp= | =sntp= | %Y% | %ICON{wip}% | %Y% | %ICON{wip}% | %ICON{wip}% | | | =loc= | |
| =tickadj= | =util= | Not started | | | | | | | =loc= | |
| =timetrim= | =util= | Not started | | | | | | | =loc= | |
| =/dev= refclock entries | ? | Not started | | | | | | | 4 | |
| refclock drivers | =html/drivers= | Not started | | | | | | | ? | |

There are other items in the =scripts/= directory that are not in the above table. If you think more items should be there, please add them.

Note that we can use DLM's =.html= pages as the starting point for the =.texi= pages.

We can use !FreeBSD's =.mdoc= pages as the starting point for the info in our =.def= pages.

In the =.def= pages:
   * in the =flag= stanzas:
      * the =descrip= tags may contain *no* formatting tags
      * the =doc= tags may contain =.texi= tags
   * the =doc-section= stanzas may contain tags - by default these are =.texi= tags and some of these stanzas are only used for =.texi= and =.info= files.  Some are used for manual pages, and for those we specify =ds-format = 'mdoc';= and use =.mdoc= tags.

---++ Making sure it works

Start with DLM's =.html= page for a given document and also the corresponding man page from !FreeBSD.

Add appropriate content to the =.def= file for the manual pages,
and to the =foo.texi= file for the =.html= page we'll produce.

   * Look at the resulting =.man=, =.mdoc=, =invoke-XXX.texi= (if generated), and =.html= files for things like unprocessed or incorrectly processed tags and punctuation not "cuddling" properly.
      * Should we be producing (and installing) =.info= files?
   * Compare the *formatted* ouput  of the =.mdoc=, =.man=, and =.html= files to the formatted output from DLM and !FreeBSD.
   * If you have =texi2pdf=, run it on the =.texi= file we produce, work on any errors that are found, and look at the resulting =.pdf= file.

Note that DLM's pages often contain "Millspeak" and sometimes typos.
Sometimes his pages contain broken sentences or formatting problems.

We should leave the "Millspeak" alone.  We should fix the typos, any broken sentences, and improve formatting.

---++ =autogen= issues
   * =ntp.conf= (?) has some problem that causes autogen to abort.

---++ =mdoc= tag fonts and formatting

To see exactly what sort of fonts and formatting are needed, here is a link to a PDF file for the formatted =mdoc= man page:

   * [[%ATTACHURL%/groff_mdoc.pdf][groff_mdoc.pdf]]: PDF of the formatted mdoc man page

---++ =mdoc2man= needs

| *Item* | *State* | *From* |
| =.Pa= | | =sntp= |
| =.Li= | | =sntp= |

---++ =mdoc2texi= needs

| *Item* | *State* | *From* | *Notes* |
| =.Oc= | | =invoke-ntp.conf.texi= | |
| =.Oo= | | =invoke-ntp.conf.texi= | |
| =.Sh= | %ICON{wip}% | =ntp.conf= | What to do about the node and menu? |
| =.Ss= | %ICON{wip}% | =ntp.conf= | What to do about the node and menu? |

---++ Current Usage Notes
We are using =autogen-5.18.x= as of July 2013.

We are using =autogen-5.16.2= as of September 2012.

We are using =autogen-5.11.9= as of May 2011.

This version seems to have useful support for =mdoc= tags, and since that is the canonical form we are using that tag format internally now.
=sntp= has been (perhaps fully) converted.

This version also supports useful license content information.
See https://autogen.sourceforge.net/doc/autogen_104.html#SCM-license_002dfull for more information.

We were using =autogen-5.11.6= as of Feb 2011.

This release of =autogen= removes some of Kapila's scripts, and Bruce indicates they can be put back in when they are better tested.

Additionally, =agman-mdoc-cmd.tpl=, =agman5.tpl= and =agman8.tpl= have been removed.
While =agman1.tpl= remains, the following is now the preferred template and method of generating  manual pages.

<verbatim>
autogen -DMAN_SECTION=n -Tagman-cmd opts.def
</verbatim>

where =n= is 1, 5, or 8.

---++ Old Notes
---+++ Capabilities we still need

   * We need !AutoGen support for man pages in other sections (4, 5, 8, at least).
   * Supporting more formatting directives
   * Supporting tables/definitions
   * Supporting configuration file formats (syntax, etc.)

It might be really nice to allow folks to format the documentation section in any of a number of formats. We could then use the post-processing stuff described below to get the data into a canonical form and perform the conversion to the desired output format from there.

---+++ !AutoGen version in the generated files

As of 5.8.6, the !AutoGen templates do not include the version of !AutoGen that was used to produce the output files.

Here are some reasons why Harlan wants the version of !AutoGen to be in the generated files:

   * The rest of the GNU =auto*= tools do it.
   * It makes it obvious which version of !AutoGen was used to generate a file. This is useful for: 
      * detecting the need to possibly rebuild if a developer used a significantly different version from what we are using on the tarball generation machine
      * seeing how different versions of !AutoGen behave

Bruce suggests adding:

<verbatim>
 [= (. autogen-version) =]
</verbatim>

to the templates. Note, this is the *templates*, not the =.def= files.

Harlan is concerned about upgrade issues and would rather not change the master template files.

---

=make= can do things like:

<verbatim>
 FOO?= bar
</verbatim>

and something similar to this would be useful to allow us to override the stock header format and get the value of =autogen-version= in there.

It is also possible that if we had pre- and post- triggers we could use one of them to change the header template after it was set to get the =autogen-version= into the generated file.

There is also a similar issue here as with the expansion of, for example, =prog-name= in the documentation (described below). In this case we have some information in the header we want formatted and it will need to be in a C-style comment for =.[ch]= files and prefixed with =./"= for the man pages (and something else for the =.texi= files).

-- Users.HarlanStenn - 26 Sep 2006

Note also, in the =.def= file you may:
<verbatim>
export = `v=\`autogen --version |sed 's/.*Ver. *//'\`
    echo '#define AUTOGEN_BUILD_VERSION "'${v}'"'
`;
</verbatim>

and with that you will have a fairly obvious AG version string in your generated .h file. It _still_ does not explain why you would want to do this. An upgraded !AutoGen is not going to alter the output from the option processing templates, unless the templates are updated or a generator bug is fixed. The latter is not likely and the former is tested with:
<verbatim>
#define AO_TEMPLATE_VERSION 118784
#if (AO_TEMPLATE_VERSION < OPTIONS_MINIMUM_VERSION) \
 || (AO_TEMPLATE_VERSION > OPTIONS_STRUCT_VERSION)
# error option template version mismatches autoopts/options.h header
  Choke Me.
#endif
</verbatim>

So, it is not clear to me the need for having the !AutoGen version in your header. Nevertheless, you can do it without changing the current templates.

-- %MAINWEB%.BruceKorb

---

---+++ Expanding =prog-name= in the documentation

The various bits of documentation will often refer to the =prog-name=, and it would be really nice if:

   * we did not have to type the program name each time
   * the program name was in the proper font/style for each output type 
      * =.Nm= for man pages
      * =@command{...}= for .texi stuff (this is a relatively new =.texi= directive)

This is similar to the above issue of customizing the header format of generated files.

__Related Topics:__ GSoC2008doc
---+++ Pre- and Post- processing

There are a variety of different strings that are set in a =.def= file that get used in various template generation mechanisms. These are documented in the "Information Attributes" section of =autogen.info= (Section 7.5.4 in the one I saw).

Harlan thinks it would be Very Useful to be able to use optional pre- and/or post- triggers when processing =.def= and =.tpl= files. OK, perhaps only post-processing would be useful.

The point is it might be really useful to use some sort of cookies in the strings in the =.def= files that are post-processed to expand them (and possibly re-flow the text, for example in the =--help= output).

If we are going to allow an optional post-processing of a string it can happen in at least 2 places:

   * right after the string is used 
      * this is probably lame, but it would allow for per-string processing directives
   * after the output file is created we do the post-processing.

Perhaps the following choices are sufficient. If there is a =foo.def= file:

   * =foo.def.TEMPLATE.post= is post-processing directives for =TEMPLATE= (curdir)
   * =foo.def.post= is a post-processing file (curdir)
   * =def.TEMPLATE.post= is a post-processing file for =TEMPLATE=
   * =def.post= is a general post-processing file

Where should the last 2 cases "live"? Should they have curdir-scope or curdir+subdir scope? What about a =-I= sort of thing?