Release 1.184.0

See release notes.

Author: cjdsellers

.github/workflows/docs.yml

@@ -2,7 +2,7 @@ name: docs
 
 on:
   push:
-    branches: [nightly]
+    branches: [master, nightly]
 
 jobs:
   docs:

.github/workflows/nightly-merge.yml

@@ -13,19 +13,18 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v4
         with:
-          fetch-depth: 0  # Fetch all history for all branches and tags
+          token: ${{ secrets.NIGHTLY_TOKEN }}
+          fetch-depth: 0  # Fetch all history for all tags and branches
 
-      # Temporary config before nautilus-bot account
-      - name: Configure Git
-        run: |
-          git config --local user.email "[email protected]"
-          git config --local user.name "Chris Sellers"
-
-      - name: Merge develop into nightly
-        run: |
-          git checkout nightly
-          git merge --no-ff origin/develop -m "Automated merge of develop into nightly"
-
-      - name: Push changes
-        run: |
-          git push origin nightly
+      - name: Nightly merge
+        uses: robotology/[email protected]
+        with:
+          stable_branch: 'develop'  # Branch to merge from
+          development_branch: 'nightly'  # Branch to merge to
+          allow_ff: false
+          user_name: 'Chris Sellers'
+          user_email: '[email protected]'
+          push_token: 'NIGHTLY_TOKEN'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NIGHTLY_TOKEN: ${{ secrets.NIGHTLY_TOKEN }}

.gitignore

@@ -54,6 +54,7 @@ venv*/
 PERF.JSON
 output.json
 *dask-worker-space*
+.cursorignore
 
 docs/**/*.ipynb
 examples/backtest/notebooks/catalog

.pre-commit-config.yaml

@@ -36,7 +36,7 @@ repos:
       - id: codespell
         description: Checks for common misspellings.
         types_or: [python, cython, rst, markdown]
-        args: ["-L", "ot,zar"]
+        args: ["-L", "crate,ot,zar"]
 
   ##############################################################################
   #  Rust formatting and linting
@@ -82,7 +82,7 @@ repos:
         exclude: "docs/_pygments/monokai.py"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.1.12
+    rev: v0.1.14
     hooks:
       - id: ruff
         args: ["--fix"]

Makefile

@@ -52,6 +52,10 @@ pre-commit:
 ruff:
 	ruff check . --fix
 
+.PHONY: outdated
+outdated:
+	(cd nautilus_core && cargo outdated && poetry show --outdated)
+
 .PHONY: update cargo-update
 update: cargo-update
 	poetry update

RELEASES.md

@@ -1,3 +1,30 @@
+# NautilusTrader 1.184.0 Beta
+
+Released on 22nd January 2024 (UTC).
+
+### Enhancements
+- Added `LogLevel.OFF` (matches the Rust `tracing` log levels)
+- Added `init_logging` function with sensible defaults to initialize the Rust implemented logging system
+- Updated Binance Futures enum members for `BinanceFuturesContractType` and `BinanceFuturesPositionUpdateReason`
+- Improved log header using the `sysinfo` crate (adds swap space metrics and a PID identifier)
+- Removed Python dependency on `psutil`
+
+### Breaking Changes
+- Removed `clock` parameter from `Logger` (no dependency on `Clock` anymore)
+- Renamed `LoggerAdapter` to `Logger` (and removed old `Logger` class)
+- Renamed `Logger` `component_name` parameter to `name` (matches Python built-in `logging` API)
+- Renamed `OptionKind` `kind` parameter and property to `option_kind` (better clarity)
+- Renamed `OptionsContract` Arrow schema field `kind` to `option_kind`
+- Changed `level_file` log level to `OFF` (file logging is off by default)
+
+### Fixes
+- Fixed memory leak for catalog queries (#1430), thanks @twitu
+- Fixed `DataEngine` order book snapshot timer names (could not parse instrument IDs with hyphens), thanks for reporting @x-zho14 and @dimitar-petrov
+- Fixed `LoggingConfig` parsing of `WARNING` log level (was not being recognized), thanks for reporting @davidsblom
+- Fixed Binance Futures `QuoteTick` parsing to capture event time for `ts_event`, thanks for reporting @x-zho14
+
+---
+
 # NautilusTrader 1.183.0 Beta
 
 Released on 12th January 2024 (UTC).
@@ -7,7 +34,7 @@ Released on 12th January 2024 (UTC).
 - Added `InstrumentClass.BOND`
 - Added `MessageBusConfig` `use_trader_prefix` and `use_trader_id` options (provides more control over stream names)
 - Added `CacheConfig.drop_instruments_on_reset` (default true to retain current behavior)
-- Implemented core logging interface via the `log` library, thanks @twitu
+- Implemented core logging interface via the `log` crate, thanks @twitu
 - Implemented global atomic clock in Rust (improves performance and ensures properly monotonic timestamps in real-time), thanks @twitu
 - Improved Interactive Brokers adapter raising docker `RuntimeError` only when needed (not when using TWS), thanks @rsmb7z
 - Upgraded core HTTP client to latest `hyper` and `reqwest`, thanks @ayush-sb
@@ -42,7 +69,7 @@ Released on 12th January 2024 (UTC).
 - Fixed `BinanceSpotInstrumentProvider` fee loading key error for partial instruments load, thanks for reporting @doublier1
 - Fixed Binance API key configuration parsing for testnet (was falling through to non-testnet env vars)
 - Fixed TWAP execution algorithm scheduled size handling when first order should be for the entire size, thanks for reporting @pcgm-team
-- Added `BinanceErrorCode.SERVER_BUSY` (-1008). Also added to the retry error codes.
+- Added `BinanceErrorCode.SERVER_BUSY` (-1008), also added to the retry error codes
 - Added `BinanceOrderStatus.EXPIRED_IN_MATCH` which is when an order was canceled by the exchange due self-trade prevention (STP), thanks for reporting @doublier1
 
 ---
@@ -207,7 +234,7 @@ This will be the final release with support for Python 3.9.
 - Moved `manage_gtd_expiry` from `Strategy.submit_order(...)` and `Strategy.submit_order_list(...)` to `StrategyConfig` (simpler and allows re-activiting any GTD timers on start)
 
 ### Fixes
-- Fixed `LimitIfTouchedOrder.create` (exec_algorithm_params were not being passed in)
+- Fixed `LimitIfTouchedOrder.create` (`exec_algorithm_params` were not being passed in)
 - Fixed `OrderEmulator` start-up processing of OTO contingent orders (when position from parent is open)
 - Fixed `SandboxExecutionClientConfig` `kw_only=True` to allow importing without initializing
 - Fixed `OrderBook` pickling (did not include all attributes), thanks @limx0
@@ -282,7 +309,7 @@ this change.
 Released on 31st July 2023 (UTC).
 
 ### Enhancements
-- Implemented string interning with the [ustr](https://github.com/anderslanglands/ustr) library, thanks @twitu
+- Implemented string interning with the [ustr](https://github.com/anderslanglands/ustr) crate, thanks @twitu
 - Added `SyntheticInstrument` capability, including dynamic derivation formulas
 - Added `Order.commissions()` convenience method (also added to state snapshot dictionaries)
 - Added `Cache` position and order state snapshots (configure via `CacheConfig`)

build.py

@@ -154,15 +154,23 @@ def _build_extensions() -> list[Extension]:
         extra_link_args += [
             "AdvAPI32.Lib",
             "bcrypt.lib",
+            "Crypt32.lib",
+            "Iphlpapi.lib",
             "Kernel32.lib",
+            "ncrypt.lib",
+            "Netapi32.lib",
             "ntdll.lib",
+            "Ole32.lib",
+            "OleAut32.lib",
+            "Pdh.lib",
+            "PowrProf.lib",
+            "Psapi.lib",
+            "schannel.lib",
+            "secur32.lib",
+            "Shell32.lib",
             "User32.Lib",
             "UserEnv.Lib",
             "WS2_32.Lib",
-            "Crypt32.lib",
-            "secur32.lib",
-            "schannel.lib",
-            "ncrypt.lib",
         ]
 
     print("Creating C extension modules...")

docs/concepts/adapters.md

@@ -28,7 +28,6 @@ from nautilus_trader.adapters.binance.common.enums import BinanceAccountType
 from nautilus_trader.adapters.binance.factories import get_cached_binance_http_client
 from nautilus_trader.adapters.binance.futures.providers import BinanceFuturesInstrumentProvider
 from nautilus_trader.common.clock import LiveClock
-from nautilus_trader.common.logging import Logger
 
 
 clock = LiveClock()
@@ -37,7 +36,6 @@ account_type = BinanceAccountType.USDT_FUTURE
 client = get_cached_binance_http_client(
     loop=asyncio.get_event_loop(),
     clock=clock,
-    logger=Logger(clock=clock),
     account_type=account_type,
     key=os.getenv("BINANCE_FUTURES_TESTNET_API_KEY"),
     secret=os.getenv("BINANCE_FUTURES_TESTNET_API_SECRET"),
@@ -47,7 +45,6 @@ await client.connect()
 
 provider = BinanceFuturesInstrumentProvider(
     client=client,
-    logger=Logger(clock=clock),
     account_type=BinanceAccountType.USDT_FUTURE,
 )
 

docs/concepts/advanced/synthetic_instruments.md

@@ -21,8 +21,8 @@ Note that the venue for a synthetic instrument is always designated as `'SYNTH'`
 ## Formula
 A synthetic instrument is composed of a combination of two or more component instruments (which
 can include instruments from multiple venues), as well as a "derivation formula". 
-Utilizing the dynamic expression engine powered by the [evalexpr](https://github.com/ISibboI/evalexpr) 
-Rust library, the platform can evaluate the formula to calculate the latest synthetic price tick 
+Utilizing the dynamic expression engine powered by the [evalexpr](https://github.com/ISibboI/evalexpr)
+Rust crate, the platform can evaluate the formula to calculate the latest synthetic price tick
 from the incoming component instrument prices.
 
 See the `evalexpr` documentation for a full description of available features, operators and precedence.

docs/concepts/data.md

@@ -115,7 +115,7 @@ We have chosen Parquet as the storage format for the following reasons:
 - It does not require any separate running components (for example a database)
 - It is quick and simple to get up and running with
 
-The Arrow schemas used for the Parquet format are either single sourced in the core `persistence` Rust library, or available
+The Arrow schemas used for the Parquet format are either single sourced in the core `persistence` Rust crate, or available
 from the `/serialization/arrow/schema.py` module.
 
 ```{note}

docs/concepts/instruments.md

@@ -37,10 +37,7 @@ Exchange specific instruments can be discovered from live exchange data using an
 from nautilus_trader.adapters.binance.spot.providers import BinanceSpotInstrumentProvider
 from nautilus_trader.model.identifiers import InstrumentId
 
-provider = BinanceSpotInstrumentProvider(
-    client=binance_http_client,
-    logger=live_logger,
-)
+provider = BinanceSpotInstrumentProvider(client=binance_http_client)
 await self.provider.load_all_async()
 
 btcusdt = InstrumentId.from_str("BTCUSDT.BINANCE")

docs/concepts/logging.md

@@ -1,12 +1,10 @@
 # Logging
 
-The platform provides logging for both backtesting and live trading using a high-performance logger implemented in Rust.
-The logger operates in a separate thread and uses a multi-producer single consumer (MPSC) channel to receive log messages.
-This design ensures that the main thread is not blocked by log string formatting or file I/O operations.
+The platform provides logging for both backtesting and live trading using a high-performance logging system implemented in Rust
+with a standardized facade from the `log` crate.
 
-```{note}
-The latest stable Rust MPSC channel is used, which is now based on the `crossbeam` implementation.
-```
+The core logger operates in a separate thread and uses a multi-producer single consumer (MPSC) channel to receive log messages.
+This design ensures that the main thread is not blocked by log string formatting or file I/O operations.
 
 There are two configurable writers for logging:
 - stdout/stderr writer
@@ -19,11 +17,12 @@ Infrastructure such as [vector](https://github.com/vectordotdev/vector) can be c
 Logging can be configured by importing the `LoggingConfig` object.
 By default, log events with an 'INFO' `LogLevel` and higher are written to stdout/stderr.
 
-Log level (`LogLevel`) values include:
-- 'DEBUG' or 'DBG'
-- 'INFO' or 'INF'
-- 'WARNING' or 'WRN'
-- 'ERROR' or 'ERR'
+Log level (`LogLevel`) values include (and generally match Rusts `tracing` level filters):
+- `OFF`
+- `DEBUG`
+- `INFO`
+- `WARNING` or `WARN`
+- `ERROR`
 
 ```{note}
 See the `LoggingConfig` [API Reference](../api_reference/config.md#LoggingConfig) for further details.
@@ -33,8 +32,12 @@ Logging can be configured in the following ways:
 - Minimum `LogLevel` for stdout/stderr
 - Minimum `LogLevel` for log files
 - Automatic log file naming and daily rotation, or custom log file name
+- Directory for writing log files
 - Plain text or JSON log file formatting
+- Filtering of individual components by log level
+- ANSI colors in log lines
 - Bypass logging completely
+- Print Rust config to stdout at initialization
 
 ### Standard output logging
 Log messages are written to the console via stdout/stderr writers. The minimum log level can be configured using the `log_level` parameter.
@@ -95,3 +98,25 @@ these color codes may not be appropriate as they can appear as raw text.
 To accommodate for such scenarios, the `LoggingConfig.log_colors` option can be set to `false`.
 Disabling `log_colors` will prevent the addition of ANSI color codes to the log messages, ensuring
 compatibility across different environments where color rendering is not supported.
+
+### Using a Logger directly
+
+It's possible to use `Logger` objects directly, and these can be initialized anywhere (very similar to the Python built-in `logging` API).
+
+If you ***aren't*** using an object which already initializes a `NautilusKernel` (and logging) such as `BacktestEngine` or `TradingNode`, 
+then you can initialize a logging in the following way:
+```python
+from nautilus_trader.common.logging import init_logging
+from nautilus_trader.common.logging import Logger
+
+init_logging()
+logger = Logger("MyLogger")
+```
+
+```{note}
+See the `init_logging` [API Reference](../api_reference/common.md#init_logging) for further details.
+```
+
+```{warning}
+Only one logging system can be initialized per process with an `init_logging` call.
+```

docs/integrations/ib.md

@@ -1,12 +1,14 @@
 # Interactive Brokers
 
-Interactive Brokers (IB) is a trading platform that allows trading in a wide range of financial instruments, including stocks, options, futures, currencies, bonds, funds, and cryptocurrencies. NautilusTrader offers an adapter to integrate with IB using their [Trader Workstation (TWS) API](https://interactivebrokers.github.io/tws-api/index.html) through their Python library, [ibapi](https://github.com/nautechsystems/ibapi).
+Interactive Brokers (IB) is a trading platform that allows trading across a wide range of financial instruments, including stocks, options, futures, currencies, bonds, funds, and cryptocurrencies. NautilusTrader offers an adapter to integrate with IB using their [Trader Workstation (TWS) API](https://interactivebrokers.github.io/tws-api/index.html) through their Python library, [ibapi](https://github.com/nautechsystems/ibapi).
 
 The TWS API serves as an interface to IB's standalone trading applications: TWS and IB Gateway. Both can be downloaded from the IB website. If you haven't installed TWS or IB Gateway yet, refer to the [Initial Setup](https://interactivebrokers.github.io/tws-api/initial_setup.html) guide. In NautilusTrader, you'll establish a connection to one of these applications via the `InteractiveBrokersClient`.
 
 Alternatively, you can start with a [dockerized version](https://github.com/gnzsnz/ib-gateway-docker) of the IB Gateway, particularly useful when deploying trading strategies on a hosted cloud platform. This requires having [Docker](https://www.docker.com/) installed on your machine, along with the [docker](https://pypi.org/project/docker/) Python package, which NautilusTrader conveniently includes as an extra package.
 
-**Note**: The standalone TWS and IB Gateway applications necessitate manual input of username, password, and trading mode (live or paper) at startup. The dockerized version of the IB Gateway handles these steps programmatically.
+```{note}
+The standalone TWS and IB Gateway applications necessitate manual input of username, password, and trading mode (live or paper) at startup. The dockerized version of the IB Gateway handles these steps programmatically.
+```
 
 ## Installation
 
@@ -22,7 +24,9 @@ For installation via poetry, use:
 poetry add "nautilus_trader[ib,docker]"
 ```
 
-**Note**: Because IB does not provide wheels for `ibapi`, NautilusTrader [repackages]( https://pypi.org/project/nautilus-ibapi/) it for release on PyPI.
+```{note}
+Because IB does not provide wheels for `ibapi`, NautilusTrader [repackages]( https://pypi.org/project/nautilus-ibapi/) it for release on PyPI.
+```
 
 
 ## Getting Started
@@ -65,6 +69,22 @@ The adapter includes several major components:
 - `InteractiveBrokersDataClient`: Connects to the Gateway for streaming market data.
 - `InteractiveBrokersExecutionClient`: Handles account information and executes trades.
 
+## The Interactive Brokers Client
+
+The `InteractiveBrokersClient` serves as the central component of the IB adapter, overseeing a range of critical functions. These include establishing and maintaining connections, handling API errors, executing trades, and gathering various types of data such as market data, contract/instrument data, and account details.
+
+To ensure efficient management of these diverse responsibilities, the `InteractiveBrokersClient` is divided into several specialized mixin classes. This modular approach enhances manageability and clarity. The key subcomponents are:
+- `InteractiveBrokersClientConnectionMixin`: This class is dedicated to managing the connection with TWS/Gateway.
+- `InteractiveBrokersClientErrorMixin`: It focuses on addressing all encountered errors and warnings.
+- `InteractiveBrokersClientAccountMixin`: Responsible for handling requests related to account information and positions.
+- `InteractiveBrokersClientContractMixin`: Handles retrieving contracts (instruments) data
+- `InteractiveBrokersClientMarketDataMixin`: Handles market data requests, subscriptions and data processing
+- `InteractiveBrokersClientOrderMixin`: Oversees all aspects of order placement and management.
+
+```{tip}
+To troubleshoot TWS API incoming message issues, consider starting at the `InteractiveBrokersClient._process_message` method, which acts as the primary gateway for processing all messages received from the API.
+```
+
 ## Instruments & Contracts
 
 In IB, a NautilusTrader `Instrument` is equivalent to a [Contract](https://interactivebrokers.github.io/tws-api/contracts.html). Contracts can be either a [basic contract](https://interactivebrokers.github.io/tws-api/classIBApi_1_1Contract.html) or a more [detailed](https://interactivebrokers.github.io/tws-api/classIBApi_1_1ContractDetails.html) version (ContractDetails). The adapter models these using `IBContract` and `IBContractDetails` classes. The latter includes critical data like order types and trading hours, which are absent in the basic contract. As a result, `IBContractDetails` can be converted to an `Instrument` while `IBContract` cannot.

examples/backtest/crypto_ema_cross_with_binance_provider.py

@@ -26,7 +26,6 @@
 from nautilus_trader.backtest.engine import BacktestEngine
 from nautilus_trader.backtest.engine import BacktestEngineConfig
 from nautilus_trader.common.clock import LiveClock
-from nautilus_trader.common.logging import Logger
 from nautilus_trader.config import InstrumentProviderConfig
 from nautilus_trader.config import LoggingConfig
 from nautilus_trader.examples.strategies.ema_cross_trailing_stop import EMACrossTrailingStop
@@ -47,19 +46,16 @@ async def create_provider():
     Create a provider to load all instrument data from live exchange.
     """
     clock = LiveClock()
-    log = Logger(clock=clock)
 
     client = get_cached_binance_http_client(
         clock=clock,
-        logger=log,
         account_type=BinanceAccountType.USDT_FUTURE,
         is_testnet=True,
     )
 
     binance_provider = BinanceFuturesInstrumentProvider(
         client=client,
         clock=clock,
-        logger=log,
         config=InstrumentProviderConfig(load_all=True, log_warnings=False),
     )