Skip to content

Add benchmarks for transformer primitives and json serialization #5957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Dec 19, 2022

Conversation

tanujkhattar
Copy link
Collaborator

This PR adds representative benchmarks for transformer primitives and circuit serialization, both of which are important use cases as we start dealing with larger circuits.

@kjsatz Can you please take a look at the benchmarks and see if it captures our discussion? Would be also good to get your perspective on what should be the target timings these benchmarks should ideally run in? (cc @maffoo as well)

Tl;Dr is that the performance right now is too slow and we'd have to make sure we address the performance bottlenecks here so that these benchmarks can run significantly faster.

$> asv run -b serialization.SerializeLargeExpandedCircuits -b transformers.transformer_primitives*
· Creating environments
· Discovering benchmarks
· Running 7 total benchmarks (1 commits * 1 environments * 7 benchmarks)
[  0.00%] · For Cirq commit 8ea41a8a <master>:
[  0.00%] ·· Benchmarking virtualenv-py3.8
[  7.14%] ··· Running (serialization.SerializeLargeExpandedCircuits.time_json_serialization--).
[ 14.29%] ··· Running (serialization.SerializeLargeExpandedCircuits.time_json_serialization_gzip--)
[ 28.57%] ··· Running (transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_moments--).
[ 35.71%] ··· Running (transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_apply_tag--).
[ 42.86%] ··· Running (transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_to_optree--).
[ 50.00%] ··· Running (transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_to_optree_and_unroll--).
[ 57.14%] ··· serialization.SerializeLargeExpandedCircuits.time_json_serialization                                                                                                                        ok
[ 57.14%] ··· ============ ============ ============ ============
              --                        num_moments
              ------------ --------------------------------------
               num_qubits      100          1000         4000
              ============ ============ ============ ============
                  100        403±70ms    3.66±0.3s    14.1±0.2s
                  500       1.82±0.06s   18.6±0.3s    1.22±0.01m
                  1000      3.50±0.08s   36.2±0.06s   2.61±0.04m
              ============ ============ ============ ============

[ 64.29%] ··· serialization.SerializeLargeExpandedCircuits.time_json_serialization_gzip                                                                                                                   ok
[ 64.29%] ··· ============ =========== =========== ============
              --                       num_moments
              ------------ ------------------------------------
               num_qubits      100         1000        4000
              ============ =========== =========== ============
                  100        420±30ms   4.12±0.1s    18.8±1s
                  500       2.09±0.3s    22.4±1s    1.54±0.07m
                  1000      4.34±0.6s    45.5±2s    3.49±0.4m
              ============ =========== =========== ============

[ 71.43%] ··· serialization.SerializeLargeExpandedCircuits.track_json_serialization_gzip_size                                                                                                             ok
[ 71.43%] ··· ============ ========= ========= ========
              --                   num_moments
              ------------ ----------------------------
               num_qubits     100       1000     4000
              ============ ========= ========= ========
                  100        "31KB"   "309KB"   "1MB"
                  500       "144KB"    "1MB"    "5MB"
                  1000      "278KB"    "2MB"    "10MB"
              ============ ========= ========= ========

[ 78.57%] ··· transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_moments                                                                                                                ok
[ 78.57%] ··· ============ =========== =========== ===========
              --                       num_moments
              ------------ -----------------------------------
               num_qubits      100         1000        4000
              ============ =========== =========== ===========
                  100        147±40ms   1.15±0.7s    4.49±1s
                  500       797±300ms    8.78±1s     33.6±6s
                  1000      1.46±0.4s    16.2±3s    1.15±0.2m
              ============ =========== =========== ===========

[ 85.71%] ··· transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_apply_tag                                                                                                   ok
[ 85.71%] ··· ============ =========== =========== ===========
              --                       num_moments
              ------------ -----------------------------------
               num_qubits      100         1000        4000
              ============ =========== =========== ===========
                  100        274±40ms   1.77±0.6s   9.47±0.2s
                  500       1.18±0.2s    15.6±2s     45.6±3s
                  1000      1.98±0.4s    24.4±2s    1.65±0.2m
              ============ =========== =========== ===========

[ 92.86%] ··· transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_to_optree                                                                                                   ok
[ 92.86%] ··· ============ =========== =========== ===========
              --                       num_moments
              ------------ -----------------------------------
               num_qubits      100         1000        4000
              ============ =========== =========== ===========
                  100        426±60ms   3.15±0.7s    22.9±4s
                  500       1.66±0.5s    23.1±2s    1.65±0.2m
                  1000      4.32±0.3s    44.2±5s    3.17±0.3m
              ============ =========== =========== ===========

[100.00%] ··· transformers.transformer_primitives.MapLargeExpandedCircuit.time_map_operations_to_optree_and_unroll
                                                                                2/9 failed
[100.00%] ··· ============ =========== ============ ===========
              --                       num_moments
              ------------ ------------------------------------
               num_qubits      100         1000         4000
              ============ =========== ============ ===========
                  100       1.43±0.3s    11.6±1s     1.10±0.2m
                  500       4.37±0.4s    56.4±4s       failed
                  1000      11.9±0.4s   1.80±0.06m     failed
              ============ =========== ============ ===========

@tanujkhattar tanujkhattar requested a review from a team as a code owner December 9, 2022 00:26
@tanujkhattar tanujkhattar requested a review from dabacon December 9, 2022 00:26
@CirqBot CirqBot added the size: M 50< lines changed <250 label Dec 9, 2022
@kjsatz
Copy link
Collaborator

kjsatz commented Dec 12, 2022

Thanks @tanujkhattar. From a quick look, the test topics look reasonable. For the scaling, I love the num_qubits in (100, 500, 1000). From the "M3 era QEC" perspective, the relevant num_moments is almost surely 100 (and not 4000) if that helps contextualize, although the scaling on several of those with num_moments is somewhat alarming.

As for how long these should take, I don't really know, but 100 ms is a nice ballpark to work in. I imagine there are some clear limiting bottlenecks you'll discover, and it's a matter of how much low-hanging fruit there might be.

Copy link
Collaborator

@pavoljuhas pavoljuhas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a few small changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/performance size: M 50< lines changed <250
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants