Skip to content

Support for ZSTD Compression #1514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions log4j-core-test/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,11 @@
<artifactId>xz</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ public static Collection<Object[]> data() {
{"log4j-rolling-bzip2.xml", ".bz2", false},
{"log4j-rolling-deflate-lazy.xml", ".deflate", true},
{"log4j-rolling-deflate.xml", ".deflate", false},
{"log4j-rolling-zstd-lazy.xml", ".zst", true},
{"log4j-rolling-zstd.xml", ".zst", false},
{"log4j-rolling-pack200-lazy.xml", ".pack200", true},
{"log4j-rolling-pack200.xml", ".pack200", false},
{"log4j-rolling-xz-lazy.xml", ".xz", true},
Expand Down
59 changes: 59 additions & 0 deletions log4j-core-test/src/test/resources/log4j-rolling-zstd-lazy.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to you under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<Configuration status="WARN" name="XMLConfigTest">
<Properties>
<Property name="filename">target/rolling1/rollingtest.log</Property>
</Properties>
<ThresholdFilter level="debug"/>

<Appenders>
<Console name="STDOUT">
<PatternLayout pattern="%m%n"/>
</Console>
<RollingFile name="RollingFile" fileName="${filename}"
filePattern="target/rolling1/test1-$${date:MM-dd-yyyy}-%i.log.zst"
createOnDemand="true">
<PatternLayout>
<Pattern>%d %p %C{1.} [%t] %m%n</Pattern>
</PatternLayout>
<SizeBasedTriggeringPolicy size="500" />
<DefaultRolloverStrategy compressionLevel="9" />
</RollingFile>
<List name="List">
<ThresholdFilter level="error"/>
</List>
</Appenders>

<Loggers>
<Logger name="org.apache.logging.log4j.test1" level="debug" additivity="false">
<ThreadContextMapFilter>
<KeyValuePair key="test" value="123"/>
</ThreadContextMapFilter>
<AppenderRef ref="STDOUT"/>
</Logger>>

<Logger name="org.apache.logging.log4j.core.appender.rolling" level="debug" additivity="false">
<AppenderRef ref="RollingFile"/>
</Logger>>

<Root level="error">
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>

</Configuration>
58 changes: 58 additions & 0 deletions log4j-core-test/src/test/resources/log4j-rolling-zstd.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to you under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<Configuration status="WARN" name="XMLConfigTest">
<Properties>
<Property name="filename">target/rolling1/rollingtest.log</Property>
</Properties>
<ThresholdFilter level="debug"/>

<Appenders>
<Console name="STDOUT">
<PatternLayout pattern="%m%n"/>
</Console>
<RollingFile name="RollingFile" fileName="${filename}"
filePattern="target/rolling1/test1-$${date:MM-dd-yyyy}-%i.log.zst">
<PatternLayout>
<Pattern>%d %p %C{1.} [%t] %m%n</Pattern>
</PatternLayout>
<SizeBasedTriggeringPolicy size="500" />
<DefaultRolloverStrategy compressionLevel="9" />
</RollingFile>
<List name="List">
<ThresholdFilter level="error"/>
</List>
</Appenders>

<Loggers>
<Logger name="org.apache.logging.log4j.test1" level="debug" additivity="false">
<ThreadContextMapFilter>
<KeyValuePair key="test" value="123"/>
</ThreadContextMapFilter>
<AppenderRef ref="STDOUT"/>
</Logger>>

<Logger name="org.apache.logging.log4j.core.appender.rolling" level="debug" additivity="false">
<AppenderRef ref="RollingFile"/>
</Logger>>

<Root level="error">
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>

</Configuration>
Original file line number Diff line number Diff line change
Expand Up @@ -46,33 +46,41 @@ Action createCompressAction(final String renameTo, final String compressedName,
@Override
Action createCompressAction(final String renameTo, final String compressedName, final boolean deleteSource,
final int compressionLevel) {
// One of "gz", "bzip2", "xz", "pack200", or "deflate".
// One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
return new CommonsCompressAction("bzip2", source(renameTo), target(compressedName), deleteSource);
}
},
DEFLATE(".deflate") {
@Override
Action createCompressAction(final String renameTo, final String compressedName, final boolean deleteSource,
final int compressionLevel) {
// One of "gz", "bzip2", "xz", "pack200", or "deflate".
// One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
return new CommonsCompressAction("deflate", source(renameTo), target(compressedName), deleteSource);
}
},
PACK200(".pack200") {
@Override
Action createCompressAction(final String renameTo, final String compressedName, final boolean deleteSource,
final int compressionLevel) {
// One of "gz", "bzip2", "xz", "pack200", or "deflate".
// One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
return new CommonsCompressAction("pack200", source(renameTo), target(compressedName), deleteSource);
}
},
XZ(".xz") {
@Override
Action createCompressAction(final String renameTo, final String compressedName, final boolean deleteSource,
final int compressionLevel) {
// One of "gz", "bzip2", "xz", "pack200", or "deflate".
// One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
return new CommonsCompressAction("xz", source(renameTo), target(compressedName), deleteSource);
}
},
ZSTD(".zst") {
@Override
Action createCompressAction(final String renameTo, final String compressedName, final boolean deleteSource,
final int compressionLevel) {
// One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
return new CommonsCompressAction("zstd", source(renameTo), target(compressedName), deleteSource);
}
};

public static FileExtension lookup(final String fileExtension) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public final class CommonsCompressAction extends AbstractAction {
private static final int BUF_SIZE = 8192;

/**
* Compressor name. One of "gz", "bzip2", "xz", "pack200" or "deflate".
* Compressor name. One of "gz", "bzip2", "xz", "zstd", "pack200" or "deflate".
*/
private final String name;

Expand All @@ -58,7 +58,7 @@ public final class CommonsCompressAction extends AbstractAction {
/**
* Creates new instance of Bzip2CompressAction.
*
* @param name the compressor name. One of "gz", "bzip2", "xz", "pack200", or "deflate".
* @param name the compressor name. One of "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
* @param source file to compress, may not be null.
* @param destination compressed file, may not be null.
* @param deleteSource if true, attempt to delete file on completion. Failure to delete does not cause an exception
Expand Down Expand Up @@ -88,7 +88,7 @@ public boolean execute() throws IOException {
/**
* Compresses a file.
*
* @param name the compressor name, i.e. "gz", "bzip2", "xz", "pack200", or "deflate".
* @param name the compressor name, i.e. "gz", "bzip2", "xz", "zstd", "pack200", or "deflate".
* @param source file to compress, may not be null.
* @param destination compressed file, may not be null.
* @param deleteSource if true, attempt to delete file on completion. Failure to delete does not cause an exception
Expand Down
7 changes: 7 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,7 @@
<woodstox.version>6.5.0</woodstox.version>
<xmlunit.version>2.9.1</xmlunit.version>
<xz.version>1.9</xz.version>
<zstd.version>1.5.5-4</zstd.version>

</properties>

Expand Down Expand Up @@ -1301,6 +1302,12 @@
<version>${xz.version}</version>
</dependency>

<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>${zstd.version}</version>
</dependency>

</dependencies>
</dependencyManagement>

Expand Down
10 changes: 7 additions & 3 deletions src/site/xdoc/manual/appenders.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3322,10 +3322,14 @@ public class JpaLogEntity extends AbstractLogEventWrapperEntity {
is present it will be replaced with the current date and time values. If the pattern contains an integer
it will be incremented on each rollover. If the pattern contains both a date/time and integer
in the pattern the integer will be incremented until the result of the date/time pattern changes. If
the file pattern ends with ".gz", ".zip", ".bz2", ".deflate", ".pack200", or ".xz" the resulting archive
the file pattern ends with ".gz", ".zip", ".bz2", ".deflate", ".pack200", "zstd" or ".xz" the resulting archive
will be compressed using the compression scheme that matches the suffix. The formats bzip2, Deflate,
Pack200 and XZ require <a href="https://commons.apache.org/proper/commons-compress/">Apache Commons Compress</a>.
In addition, XZ requires <a href="https://tukaani.org/xz/java.html">XZ for Java</a>.
Pack200, ZSTD and XZ require <a href="https://commons.apache.org/proper/commons-compress/">Apache Commons Compress</a>.
In addition:
<ul>
<li>XZ requires <a href="https://tukaani.org/xz/java.html">XZ for Java</a>.</li>
<li>ZSTD requires <a href="https://github.com/luben/zstd-jni">JNI binding for Zstd</a>.</li>
</ul>
The pattern may also contain lookup references that can be resolved at runtime such as is shown in the example
below.
</p>
Expand Down
6 changes: 5 additions & 1 deletion src/site/xdoc/runtime-dependencies.xml
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,11 @@
<tr>
<td>Bzip2, Deflate, Pack200, and XZ compression on rollover</td>
<td><a href="https://commons.apache.org/proper/commons-compress/">Apache Commons Compress</a>.
In addition, XZ requires <a href="https://tukaani.org/xz/java.html">XZ for Java</a>.
In addition:
<ul>
<li>XZ requires <a href="https://tukaani.org/xz/java.html">XZ for Java</a>.</li>
<li>ZSTD requires <a href="https://github.com/luben/zstd-jni">JNI binding for Zstd</a>.</li>
</ul>
</td>
</tr>
<tr>
Expand Down