Description
The Nautilus value types Price, Money, and Quantity are currently backed by 64-bit integer raw values, with a maximum precision of 9 decimals enforced through validation (see table below). This design provides memory efficiency and high performance, using simple fixed-point arithmetic to represent actual values. (Integers are more CPU-native than string or decimal types, and are highly efficient for comparisons.)
Limitations of the current 64-bit specification:
- 9 decimals of precision cannot fully represent all fractional units for certain digital assets (cryptocurrencies).
- The allowable minimum and maximum value range is limiting for some use cases.
- For
Quantity, larger time frame bar volumes cannot be adequately represented.
To address these issues, increasing the raw integer width from 64 bits to 128 bits will significantly expand the precision and allowable range.
Current specification (64-bit integer backing):
| Type | Raw backing | Max precision | Min value | Max value |
|---|---|---|---|---|
| Price | i64 |
9 | -9,223,372,036 | 9,223,372,036 |
| Money | i64 |
9 | -9,223,372,036 | 9,223,372,036 |
| Quantity | u64 |
9 | 0 | 18,446,744,073 |
Proposed specification (128-bit integer backing):
| Type | Raw backing | Max precision | Min value | Max value |
|---|---|---|---|---|
| Price | i128 |
16 | -17,014,118,346,046 | 17,014,118,346,046 |
| Money | i128 |
16 | -17,014,118,346,046 | 17,014,118,346,046 |
| Quantity | u128 |
16 | 0 | 34,028,236,692,093 |
Pros of 128-bit backing:
- 16 decimals of precision can represent practically all fractional units, making it suitable for most cryptocurrencies.
- Increased headroom resolves limitations in representing large bar volumes and extreme price values.
Cons of 128-bit backing:
- Increased in-memory footprint (nearly doubling for some types such as
QuoteTick). - While Parquet efficiently compresses data, there will still be a modest increase in storage size.
- Breaking change to specification and API requiring catalogs to be re-written to upgrade to the latest version.
Other solutions, such as introducing an additional scaling factor field and using arithmetic for more flexible precisions, were considered. However, these approaches significantly increased complexity and the likelihood of bugs, as raw values are directly accessed in many parts of the codebase.
Mitigations
For the in-memory cache, the number of 128-bit fields across all objects is relatively small, and for data is further constrained by deques with a defined maxlen.
For the in-memory footprint of data during large backtests, this is mostly alleviated by streaming from the Parquet data catalog where only a limited amount of data is held in memory at a time. Users who are using the BacktestEngine directly will be most affected.
The breaking change to catalogs is a temporary inconvenience. Users can decide whether to upgrade to nautilus_trader versions with 128-bit backing or choose the timing of their upgrade to align with their operational needs. Alternatively, they can compile from source with the existing 64-bit backing.
Implementation
To provide flexibility, @twitu is implementing a solution where a high-precision feature flag will control whether these values are backed by 64-bit or 128-bit integers. This allows users to:
- Retain the current 64-bit specification for reduced memory usage.
- Opt-in to 128-bit backing to overcome precision and range limitations.
Final comments
While the increased precision and range may be unnecessary or even excessive for traditional financial assets, the limitations are far more severe for crypto users. Traditional users may appreciate the lower memory footprint as a "nice to have," but crypto users are constrained by the 9-decimal precision cap and a more restricted range.
It's expected that high-precision will quickly become the default, as the advantages outweigh the trade-offs for most users.
We're open to feedback and suggestions on the above.