Skip to content

Fix large difference between Data enum variants #2315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 13, 2025
Merged

Conversation

twitu
Copy link
Collaborator

@twitu twitu commented Feb 12, 2025

Pull Request

The data enum has the orderbook depth variant which is significantly larger than other variants bloating memory footprint of all other variants. Boxing orderbook depth minimizes its footprint with minimal changes.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How has this change been tested?

    #[rstest]
    pub fn test_load_deltas_memory_usage() {
        use nautilus_model::data::Data;
        use procfs::page_size;
        use procfs::process::Process;

        let page_size = page_size();
        let me = Process::myself().unwrap();
        let setup_mem = me.stat().unwrap().rss * page_size / 1024;

        let before = me.stat().unwrap().rss * page_size / 1024 - setup_mem;

        let filepath = "/home/twitu/Code/nautilus_trader/tests/test_data/large/tardis_deribit_incremental_book_L2_2020-04-01_BTC-PERPETUAL.csv.gz";
        let instrument_id = InstrumentId::from("BTC-PERPETUAL.DERIBIT");
        let deltas = load_deltas(filepath, 1, 0, Some(instrument_id), Some(1_000_000)).unwrap();

        // TODO: Wrapping as enum explodes the memory
        let deltas: Vec<Data> = deltas.into_iter().map(Data::Delta).collect();

        assert_eq!(deltas.len(), 1_000_000);

        let after = me.stat().unwrap().rss * page_size / 1024 - setup_mem;
        println!("Memory allocated: {} KB", after - before);
    }

Before the change this test shows 10x increase in delta size.
1M deltas -> 103 MB
1M deltas (wrapped with Data) -> 1 GB

After the change
1M deltas -> 103 MB
1M deltas (wrapped with Data boxed Orderbook depth) -> 160 MB

@cjdsellers cjdsellers merged commit 9ccee8c into develop Feb 13, 2025
12 checks passed
@cjdsellers cjdsellers deleted the tardis-mem-issue branch February 13, 2025 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants