Skip to content

Commit d0ef338

Browse files
authored
vdk-structlog: add default logging format values (#3055)
## Why? For users who do not wish to configure each structlog option separately, introduce presets for different cases. ## What? Introduce CLOUD and LOCAL presets. The CLOUD preset hardcodes the following options ```sh export VDK_STRUCTLOG_FORMAT=console export VDK_STRUCTLOG_CONSOLE_LOG_PATTERN="%(asctime)s [VDK] %(vdk_job_name)s [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%(lineno)-4.4s %(funcName)-16.16s[id:%(attempt_id)s]- %(message)s" ``` LOCAL just uses the defaults set in the configuration code Refactor structlog plugin to eliminate repeating code Remove syslog config tests Fix bug with vdk fields crashing logging for custom format Add tests for vdk fields with custom format ## How was this tested? Functional tests CI ## What kind of change is this? Feature/non-breaking Signed-off-by: Dilyan Marinov <mdilyan@vmware.com>
1 parent 7c993c9 commit d0ef338

File tree

6 files changed

+406
-293
lines changed

6 files changed

+406
-293
lines changed

projects/vdk-plugins/vdk-structlog/README.md

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,18 @@ pip install vdk-structlog
2121

2222
(`vdk config-help` is a useful command to browse all config options of your installation of vdk)
2323

24-
| Name | Description | Example Value | Possible Values |
25-
|----------------------------|-------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
26-
| logging_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
27-
| logging_format | Configure the logging output format. Available formats: json, console | "console" | "console", "json", "ltsv" |
28-
| custom_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
29-
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
30-
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
31-
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |
32-
| syslog_protocol | Protocol used to emit logs | "UDP" | "TCP", "UDP" |
33-
| syslog_enabled | Enable/disable syslog | "True" | "True", "False" |
24+
| Name | Description | Example Value | Possible Values |
25+
|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
26+
| use_structlog | Use the structlog logging config instead of using the one in vdk-core | "True" | "True", "False" |
27+
| structlog_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
28+
| structlog_format | Configure the logging output format. Available formats: json, console, ltsv | "console" | "console", "json", "ltsv" |
29+
| structlog_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. Note: For config.ini, %-signs should be escaped by doubling, e.g. %(asctime)s should become %%(asctime)s | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
30+
| structlog_config_preset | Choose a configuration preset. Any config options set together with the preset will override the preset options. Available presets: LOCAL, CLOUD. | "CLOUD" | "console", "json", "ltsv" |
31+
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
32+
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
33+
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |
34+
| syslog_protocol | Protocol used to emit logs | "UDP" | "TCP", "UDP" |
35+
| syslog_enabled | Enable/disable syslog | "True" | "True", "False" |
3436

3537
### Example: Configure Custom Console Format
3638

projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/constants.py

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,17 @@
33
import logging
44
import socket
55

6+
STRUCTLOG_USE_STRUCTLOG = "use_structlog"
67
STRUCTLOG_LOGGING_METADATA_KEY = "structlog_metadata"
78
STRUCTLOG_LOGGING_FORMAT_KEY = "structlog_format"
89
STRUCTLOG_CONSOLE_LOG_PATTERN = "structlog_console_custom_format"
10+
STRUCTLOG_CONFIG_PRESET = "structlog_config_preset"
911

1012
STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES = ["console", "json", "ltsv"]
1113
STRUCTLOG_LOGGING_FORMAT_DEFAULT = "console"
1214

1315
STRUCTLOG_LOGGING_METADATA_JOB = {
16+
"attempt_id": "%(attempt_id)s",
1417
"vdk_job_name": "%(vdk_job_name)s",
1518
"vdk_step_name": "%(vdk_step_name)s",
1619
"vdk_step_type": "%(vdk_step_type)s",
@@ -44,6 +47,7 @@
4447
}
4548

4649
CONSOLE_STRUCTLOG_LOGGING_METADATA_JOB = {
50+
"attempt_id": "[id:%(attempt_id)s]",
4751
"vdk_job_name": "%(vdk_job_name)s",
4852
"vdk_step_name": "%(vdk_step_name)s",
4953
"vdk_step_type": "%(vdk_step_type)s",
@@ -66,10 +70,10 @@
6670
RECORD_DEFAULT_FIELDS = set(vars(logging.LogRecord("", "", "", "", "", "", "")))
6771

6872
# Syslog constants
69-
SYSLOG_HOST_KEY = "SYSLOG_HOST"
70-
SYSLOG_PORT_KEY = "SYSLOG_PORT"
71-
SYSLOG_PROTOCOL_KEY = "SYSLOG_PROTOCOL"
72-
SYSLOG_ENABLED_KEY = "SYSLOG_ENABLED"
73+
SYSLOG_HOST_KEY = "syslog_host"
74+
SYSLOG_PORT_KEY = "syslog_port"
75+
SYSLOG_PROTOCOL_KEY = "syslog_protocol"
76+
SYSLOG_ENABLED_KEY = "syslog_enabled"
7377

7478
# Default values for Syslog
7579
DEFAULT_SYSLOG_HOST = "localhost"
@@ -78,3 +82,8 @@
7882
DEFAULT_SYSLOG_ENABLED = False
7983

8084
SYSLOG_PROTOCOLS = {"UDP": socket.SOCK_DGRAM, "TCP": socket.SOCK_STREAM}
85+
86+
DETAILED_LOGGING_FORMAT = (
87+
"%(asctime)s [VDK] %(vdk_job_name)s [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%("
88+
"lineno)-4.4s %(funcName)-16.16s[id:%(attempt_id)s]- %(message)s"
89+
)

0 commit comments

Comments
 (0)