Zstd compression support

[anuragagarwal561994]

At present log4j2 provides certain compression algorithms like

Gzip
Lz4
Etc.

Zstd in recent times have become popular and we have tried compressing our logs in production with zstd by running a fork of log4j2 and it does provide close to 20-30% saving with respect to gzip.

It depends on the file size actually. So not applicable for all use cases.

But log4j2 library today does not provide an extensible way to add a compression algorithm.

Do you think, addition of zstd to the list of compressions will be a good addition?

[vy]

I really liked both ideas!

1. Making compression extensible (I actually have created LOG4J2-3023 for something similar in the past)
2. Adding zstd

Are you interested in submitting a PR? If so, I can assist you with the design.

[garydgregory]

FYI, Apache Commons Compress supports zstd and many other formats.

[anuragagarwal561994]

@vy I am not sure if I will be able to contribute to in near term.

I can share the code we have been using for this purpose in our organisation. May be one doesn't have to do anything and just integrate that code itself, but I think when going open source some standards needs to be maintained.

Have been a bit occupied recently with work, still let me check if I get time over the weekend to look into this.

[vy]

@anuragagarwal561994, could you at least share the outline of what you have at work? Is it a custom appender? A layout?

[anuragagarwal561994]

So it is mostly built like Gzip only, just that instead of using GzipOurstream we are using ZstdOutputStream

We are using zstd-jni as a dependency here.

[anuragagarwal561994]

@vy the MR is actually ready I have integrated it using commons compress itself and the tests were passing in the first go.

[vy]

Thanks for the contribution @anuragagarwal561994. As I indicated in my response earlier, I was in the hope of a contribution to first make the compression extensible and then implement zstd by means of a simple plugin. (Hence, was my offer for assistance...) I am not much fond of these kinds of specialized solutions similar to the one you implemented in #1514, adding zstd only for the rolling appender. (For the records, I have also raised this issue in the mailing list.)

If you would like to collaborate on a more general solution, please let me know.

[anuragagarwal561994]

@vy I would have loved to but because of time constraints at work, I am not able to give time to contribute to open source at present. I do understand that this is a specialised solution, but since it has been done before for other compression algorithms, I think it can be a good way to add the functionality right now.

Designing a generic solution would take time including design, reviews and tests which I guess can also be taken care of as a separate step. What do you think, may be a good first issue can be opened in the repository for the contributors to contribute. What do you say?

[rgoers]

I will note that the choice of what compression to use is driven by the FileExtension enum. This could easily be replaced by Plugins where each plugin specifies the file extensions it supports however, it would be a little less efficient because most of the extensions use the compress action but construct it with the compression type as a parameter. Instead, we would probably have to declare each instance as a separate plugin class.

[ppkarwasz]

I would model this feature after ConfigurationFactory. Each CompressActionFactory would declare the supported compression methods through a getSupportedTypes() method and each factory would have a priority value.

Currently we would only need two CompressActionFactory plugins:

• one that only depends on java.base for *.zip and *.gz,
• one that depends on commons-compress and find out the available compression methods dynamically using CompressStreamFactory#findAvailableCompressorOutputStreamProviders. We could move this factory to a separate log4j-commons-compress artifact.

[rgoers]

@ppkarwasz Yes, that sounds like a good approach.