[MS9] Schema and documentation generation from source code

[grobmeier]

Abstract: This PR completes the STF milestone 9 by implementing the infrastructure to auto-generate 1) the Log4j plugin documentation (to be deployed as a part of the Log4j website) and 2) the XSD (for assisting the Log4j runtime configuration, i.e., log4j2.xml, in user applications).

The problem description

Log4j runtime is a composition of 300+ plugins, which is analogous to beans in Java EE and Spring applications. Developers document these plugins in the website. Users assemble them in the log4j2.xml file describing their Log4j runtime configurations.

Currently, several plugins are either not documented (e.g. LOG4J2-3485), or their documentation doesn't match the behaviour of the source code. This not only causes users to misconfigure their applications or not be aware of all options available to them, but can also pose security implications due to unintended (mis)configurations. Next to lacking documentation, the log4j-config.xsd file users use to get assisted while assembling their log4j2.xml configurations is severely outdated. It doesn't cover a very big part of the available plugin ecosystem.

Deliverable 1: Log4j Docgen

Log4j Docgen (Documentation Generator) project aims to attack this problem by generating 1) the documentation, 2) plugin configuration options, and 3) XSD from the Java source code of the Log4j project itself. Effectively, this enables the following improvements:

• Avoid the need to maintain documentation in multiple places – For instance, Log4j JSON Template Layout is composed of several plugins with sufficient Javadoc explaining their behaviour. Next to this we also maintain a big json-template-layout.adoc page for the website, again, documenting each component. Last, but not least, log4j-config.xsd also needs to be updated (which is not!) to document all these whistles and bells too. By generating documentation from the source code, developers can precisely capture the behaviour of their components only in the very code itself that they change. As a result, there will be one and only place of the truth: the source code itself.
• Accurately capture the component configuration – It is pretty common that source code changes are not reflected to documentation and XSD. By generating the documentation from the source code, this discrepancy gets removed.

Currently being released Log4j Tools 0.8.0 bundles several components to implement this end-to-end operation. Refer to the Log4j Tools 0.8.0 website to read about all these new exciting Log4j Docgen features. Implementation details of the new log4j-docgen, log4j-docgen-maven-plugin, and log4j-docgen-asciidoc-extension can be viewed in this diff.

Deliverable 2: Log4j plugin documentation and XSD

Of course, it would not be complete to announce this work as complete without demonstrating its integration into the Log4j project itself. The 2.x-docgen and 3.x-docgen branches of the Log4j repository contain the changes that integrate Log4j Docgen into Log4j 2 and 3, respectively, to generate plugin reference manual to be built along with the website and to generate the plugin XSD (aka. log4j-config.xsd) capturing the type hierarchy of the available set of plugins. (The branch will be merged after the Log4j Tools 0.8.0 release scheduled for 2024-03-21.)

Deliverable 3: Centralizing XML namespace

Log4j project publishes four major XSDs:

1. Log4j Changelog (packaged with Log4j Tools)
2. Log4j 2
3. Log4j 3

Yet there does not exist a central place where users can access to all available versions of all these XSDs with sufficient descriptions. Inspired by the battle-tested principles practiced in Java EE: XML Schemas for Java EE Deployment Descriptors, we have compiled https://logging.apache.org/xml/ns/ for this purpose.

Implementation details

The bulk of the work is accessible in two locations:

1. Log4j Tools 0.8.0 release (the changes incorporated)
2. Add auto-generated plugin reference to the 2.x site #2404
3. Add auto-generated plugin reference to the 3.x site #2407

Other supporting ground work is implemented in following tasks:

• Generate reference documentation and XSD from metadata #1955
• Extract configuration metadata from source code #1956
• Fix the Javadoc comments of all our plugins and generate a "Reference" section in our documentation.
• Check the generated schema with xmllint (cf. Fix regex error in Log4j-config.xsd #2357)

[jvz]

I think I've proposed something along these lines in the past, but it seemed like too much work at the time. However, I love this idea in general. It would be great if we could extract JavaDoc metadata from plugin declarations to help form the documentation. Otherwise, we have a lot of duplicated information about plugins.

Might also be useful to include metadata that specifies some sample source files to inline for examples.

[ppkarwasz]

The plan is to delete all *.md, *.adoc and other files that describe our components' configuration and put everything into Javadoc.

The manual will be split into a Configuration Reference (cf Tomcat's reference) and a User Guide (cf. Tomcat's user guide). The latter will only explain how the system works, how to use the different configuration formats (with examples) and will not be encumbered with lots of technical details.

The Configuration Reference can even collect documentation from third-party plugin developers, if they embed metadata into their jars (or publish on Maven).

[vy]

Note that XmlSchemaTest needs to be removed and its functionality should ideally be replaced with a xml-maven-plugin validation that can take place at the Maven validation phase.