feat: no longer recompute vk's in CIVC proofs (#13590)

Apologies for the big PR. Lots of the stuff in here ended up being
chicken and egg with wanting to do refactoring to make this process
smoother/help debug tricky issues, and wanting to see that those
refactorings actually make sense by the end.

We no longer compute VKs on the fly in CIVC. This saves ~25% of
computation. This is done throughout by consolidating IVC inputs into a
single ivc-inputs.msgpack structure which supports passing bytecode,
witness & vk information to the bb backend. Now attaches a name for each
function, as well.

Major features:
- IVC inputs passed thru native and wasm are always passed a single
file/buffer. This is encoded using msgpack and capture bytecode,
witness, vk, and function name (which is now printed, but only properly
passed by native) For native, the bincode and witnesses are gzipped, for
WASM they are uncompressed. For actions such as gates or write_vk, the
IVC inputs are used with same structure but witness and vk data can be
blank.

This has a bunch of implications, such as having to break away from the
rigid API base class in bb cli (which overall doesn't feel worthwhile
anyway as CIVC is fundamentally different than UH), having to string vk
info along, etc.

Other features:

Debuggability:

- Correct README.md instructions on WASM stack traces (give up on
getting line numbers working :/)
- clangd now properly shows all errors in a C++ file you're browsing,
instead of only showing you the first error.

Cleanup

- small cleanup to acir tests, but still not testing new ivc flow there.
Lightest weight test is ivc-integration in yarn-project
- Get rid of --input_type in bb cli for CIVC. now implied always to be
what was previously runtime_stack. Simplifies usages, other modes were
unused.
- more ignored linting in the clangd file. Maybe one day we can enforce
the remaining as errors.
- Clean up msgpack usage. Msgpack headers were leaking everywhere and it
is a chunky library.
- Consolidate with using msgpackr as our only typescript messagepack
library

Benches

- use wasmtime helper in bb bootstrap. deduplicate code in bench. bench
now honours NATIVE_PRESET, and if you do
```
export NATIVE_PRESET=op-count-time
./bootstrap.sh
./bootstrap.sh bench
```
you will get op count timings for our native ivc benches.

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: thunkar <[email protected]>
Co-authored-by: maramihali <[email protected]>
Co-authored-by: ledwards2225 <[email protected]>

Author: 5 people

barretenberg/README.md

@@ -477,7 +477,7 @@ To do a single benchmark you can do e.g.
 `IVC_BENCH=amm-add-liquidity ./bootstrap.sh bench_ivc origin/master~3`
 
 If one doesn't provide the commit, it generates these IVC inputs on the fly (depends on yarn-project having been bootstrapped).
-To use these inputs manually, just abort after input download and run ClientIVC proving with --input runtime_stack on those inputs (stored in `yarn-project/end-to-end/example-app-ivc-inputs-out`).
+To use these inputs manually, just abort after input download and run ClientIVC proving on those inputs (stored in `yarn-project/end-to-end/example-app-ivc-inputs-out`).
 
 #### Using Tracy to Profile Memory/CPU/Gate Counts
 
@@ -532,7 +532,7 @@ By default, the barretenberg.wasm.gz that is used by bb.js (aka barretenberg/ts)
 One can get stack traces working from WASM by running root level ./bootstrap.sh (or otherwise building what you need) and then doing:
 ```
 cmake --build --preset wasm-threads --target barretenberg-debug.wasm.gz
-mv build-wasm-threads/barretenberg-debug.wasm.gz build-wasm-threads/barretenberg.wasm.gz
+cp build-wasm-threads/bin/barretenberg-debug.wasm.gz ../ts/dest/node/barretenberg_wasm/barretenberg-threads.wasm.gz
 ```
 
 This will mean that any yarn-project or barretenberg/ts tests that run will get stack traces with function names.
@@ -541,5 +541,5 @@ To get more detailed information use the following (NOTE: takes >10 minutes!):
 ```
 cmake --preset wasm-threads-dbg
 cmake --build --preset wasm-threads-dbg --target barretenberg-debug.wasm.gz
-mv build-wasm-threads-dbg/barretenberg-debug.wasm.gz build-wasm-threads/barretenberg.wasm.gz
+cp build-wasm-threads-dbg/bin/barretenberg-debug.wasm.gz ../ts/dest/node/barretenberg_wasm/barretenberg-threads.wasm.gz
 ```

barretenberg/acir_tests/bootstrap.sh

@@ -215,9 +215,6 @@ function run_benchmark {
 
 # TODO(https://github.com/AztecProtocol/barretenberg/issues/1254): More complete testing, including failure tests
 function bench {
-  # TODO(https://github.com/AztecProtocol/barretenberg/issues/1265) fix acir benchmarking
-  # LOG_FILE=bench-acir.jsonl ./bench_acir_tests.sh
-
   export HARDWARE_CONCURRENCY=16
 
   rm -rf bench-out && mkdir -p bench-out

barretenberg/acir_tests/flows/prove_then_verify.sh

@@ -6,23 +6,6 @@ BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH ${VERBOSE:+-v}"
 [ "${RECURSIVE}" = "true" ] && FLAGS+=" --recursive"
 
-# TODO: Use this when client ivc support write_vk. Currently it keeps its own flow.
-# case ${SYS:-} in
-#   "")
-#     prove_cmd=prove
-#     verify_cmd=verify
-#     ;;
-#   "client_ivc")
-#     prove_cmd=prove
-#     verify_cmd=verify
-#     flags+=" --scheme client_ivc --input_type ${INPUT_TYPE:-single_circuit}"
-#     ;;
-#   *)
-#     prove_cmd=prove_$SYS
-#     verify_cmd=verify_$SYS
-#     ;;
-# esac
-
 # Test we can perform the proof/verify flow.
 # This ensures we test independent pk construction through real/garbage witness data paths.
 # We use process substitution pipes to avoid temporary files, which need cleanup, and can collide with parallelism.

barretenberg/acir_tests/flows/prove_then_verify_client_ivc.sh

@@ -1,17 +1,8 @@
 #!/usr/bin/env bash
 # Create intermediate state in a directory. Uses a temp dir to ensure parallel safe and cleans up on exit.
+# TODO this is unused
 set -eux
 
-INFLAG=${INPUT_TYPE:---input_type runtime_stack}
-
-if [ "$INFLAG" = "--input_type runtime_stack" ]; then
-  BFLAG=target/acir.msgpack
-  WFLAG=target/witness.msgpack
-else
-  BFLAG=target/program.json
-  WFLAG=target/witness.gz
-fi
-
 CRS_PATH=${CRS_PATH:-$HOME/.bb-crs}
 BIN=$(realpath ${BIN:-../cpp/build/bin/bb})
 
@@ -23,6 +14,6 @@ trap "rm -rf $outdir" EXIT
 flags="--scheme client_ivc -c $CRS_PATH ${VERBOSE:+-v}"
 
 parallel ::: \
-  "$BIN prove $flags -b $BFLAG -w $WFLAG $INFLAG --output_format proof -o $outdir" \
-  "$BIN write_vk $flags -b $BFLAG $INFLAG --verifier_type ivc -o $outdir"
+  "$BIN prove $flags -i target/ivc-inputs.msgpack $INFLAG --output_format proof -o $outdir" \
+  "$BIN write_vk $flags -i target/ivc-inputs.msgpack $INFLAG --verifier_type ivc -o $outdir"
 $BIN verify $flags -p $outdir/proof -k $outdir/vk

barretenberg/acir_tests/flows/sol_honk.sh

@@ -4,7 +4,7 @@ set -eu
 VFLAG=${VERBOSE:+-v}
 BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH $VFLAG --scheme ultra_honk"
-PROVE_FLAGS="$FLAGS $BFLAG --oracle_hash keccak --output_format bytes_and_fields --write_vk --input_type single_circuit"
+PROVE_FLAGS="$FLAGS $BFLAG --oracle_hash keccak --output_format bytes_and_fields --write_vk"
 VERIFY_FLAGS="$FLAGS --oracle_hash keccak"
 
 outdir=$(mktemp -d)

barretenberg/cpp/.clangd

@@ -1,5 +1,7 @@
 CompileFlags: # Tweak the parse settings
-  Remove: -fconstexpr-ops-limit=* -Wfatal-errors
+  Remove:
+    - "-fconstexpr-ops-limit=*"
+    - "-Wfatal-errors"
 ---
 # Applies all barretenberg source files
 If:
@@ -51,6 +53,8 @@ Diagnostics:
       - readability-container-data-pointer
       # Many hits; potential for false positives.
       - cppcoreguidelines-pro-type-member-init
+      # We do use this, so no point banning it. The name is a warning enough.
+      - cppcoreguidelines-pro-type-reinterpret-cast
       # As cryptographers, we often think of bools as 0/1 values; would bloat code in some places.
       - modernize-use-bool-literals
       # Triggers on every TYPED_TEST
@@ -71,6 +75,12 @@ Diagnostics:
       - modernize-use-auto
       # Not worth rewriting. Redundant types are often clearer.
       - modernize-return-braced-init-list
+      # Not worth rewriting. Also some gotchas with mac not accepting emplaces of C++20 default constructors.
+      - modernize-use-emplace
+      # We only really use macros where a constexpr would NOT be able to replace its functionality.
+      - cppcoreguidelines-macro-usage
+      # Opinionated, we sometimes just want to use a default-derived constructor.
+      - modernize-use-designated-initializers
 
 --- # this divider is necessary
 # Disable some checks for Google Test/Bench

barretenberg/cpp/CMakePresets.json

@@ -471,7 +471,7 @@
     },
     {
       "name": "wasm-threads-assert",
-      "displayName": "Build for WASM with multithreading and  and asserts",
+      "displayName": "Build for WASM with multithreading and asserts",
       "binaryDir": "build-wasm-threads-assert",
       "description": "Build with wasi-sdk with asserts",
       "inherits": "wasm-threads",

barretenberg/cpp/bootstrap.sh

@@ -220,10 +220,9 @@ function bench {
       --benchmark_filter="construct_proof_ultrahonk_power_of_2/20$"
   }
   function ultra_honk_wasm {
-    wasmtime run --env HARDWARE_CONCURRENCY --env IGNITION_CRS_PATH --env GRUMPKIN_CRS_PATH -Wthreads=y -Sthreads=y --dir=. \
-      ./build-wasm-threads/bin/ultra_honk_bench \
-        --benchmark_out=./bench-out/ultra_honk_wasm.json \
-        --benchmark_filter="construct_proof_ultrahonk_power_of_2/20$"
+    scripts/wasmtime.sh ./build-wasm-threads/bin/ultra_honk_bench \
+      --benchmark_out=./bench-out/ultra_honk_wasm.json \
+      --benchmark_filter="construct_proof_ultrahonk_power_of_2/20$"
   }
 
   # Client IVC
@@ -243,10 +242,9 @@ function bench {
       --benchmark_filter="ClientIVCBench/Full/6$"
   }
   function client_ivc_wasm {
-    wasmtime run --env HARDWARE_CONCURRENCY --env IGNITION_CRS_PATH --env GRUMPKIN_CRS_PATH -Wthreads=y -Sthreads=y --dir=. \
-      ./build-wasm-threads/bin/client_ivc_bench \
-        --benchmark_out=./bench-out/client_ivc_wasm.json \
-        --benchmark_filter="ClientIVCBench/Full/6$"
+    scripts/wasmtime.sh ./build-wasm-threads/bin/client_ivc_bench \
+      --benchmark_out=./bench-out/client_ivc_wasm.json \
+      --benchmark_filter="ClientIVCBench/Full/6$"
   }
 
   function run_benchmark {
@@ -300,7 +298,7 @@ case "$cmd" in
     # Takes an optional master commit to download them from. Otherwise, downloads from latest master commit.
     git fetch origin master
 
-    build the benchmarked benches
+    # build the benchmarked benches
     parallel --line-buffered --tag -v denoise ::: \
       "build_preset $native_preset --target bb_cli_bench" \
       "build_preset wasm-threads --target bb_cli_bench"

barretenberg/cpp/scripts/ci_benchmark_ivc_flows.sh

@@ -4,21 +4,23 @@
 source $(git rev-parse --show-toplevel)/ci3/source
 
 if [[ $# -ne 2 ]]; then
-  echo "Usage: $0 <bench_input_folder> <output_folder>"
+  echo "Usage: $0 <bench_input_folder> <benchmark_output>"
   exit 1
 fi
-export input_folder="$1"
-output_folder="$2"
-
 cd ..
+export input_folder="$1"
+benchmark_output="$2"
 
 echo_header "bb ivc flow bench"
 
 export HARDWARE_CONCURRENCY=16
 export IGNITION_CRS_PATH=./srs_db/ignition
 export GRUMPKIN_CRS_PATH=./srs_db/grumpkin
+export native_preset=${NATIVE_PRESET:-clang16-assert}
+export native_build_dir=$(scripts/cmake/preset-build-dir $native_preset)
 
-rm -rf bench-out/ivc && mkdir -p bench-out/ivc
+mkdir -p bench-out
+rm -rf bench-out/ivc-*
 
 function verify_ivc_flow {
   local flow="$1"
@@ -28,10 +30,10 @@ function verify_ivc_flow {
   # TODO(AD): Checking which one would be good, but there isn't too much that can go wrong here.
   set +e
   echo_stderr "Private verify."
-  ./build/bin/bb verify --scheme client_ivc -p "$proof" -k ../../yarn-project/bb-prover/artifacts/private-civc-vk 1>&2
+  "./$native_build_dir/bin/bb" verify --scheme client_ivc -p "$proof" -k ../../yarn-project/bb-prover/artifacts/private-civc-vk 1>&2
   local private_result=$?
   echo_stderr "Private verify: $private_result."
-  ./build/bin/bb verify --scheme client_ivc -p "$proof" -k ../../yarn-project/bb-prover/artifacts/public-civc-vk 1>&2
+  "./$native_build_dir/bin/bb" verify --scheme client_ivc -p "$proof" -k ../../yarn-project/bb-prover/artifacts/public-civc-vk 1>&2
   local public_result=$?
   echo_stderr "Public verify: $public_result."
   if [[ $private_result -eq $public_result ]]; then
@@ -44,86 +46,71 @@ function verify_ivc_flow {
   fi
 }
 
-function client_ivc_flow_native {
-  set -eu
-  local flow=$1
-  local flow_folder="$input_folder/$flow"
-  local start=$(date +%s%N)
-  mkdir -p "bench-out/$flow-proof-files"
-  export MEMUSAGE_OUT="bench-out/$flow-proof-files/peak-memory-native-mb.txt"
-
-  function bb_cli_bench_native {
-    export MAIN_ARGS="$*"
-    memusage ./build/bin/bb_cli_bench \
-        --benchmark_out=bench-out/$flow-proof-files/op-counts.json \
-        --benchmark_out_format=json || {
-      echo "bb_cli_bench failed with args: $*"
+function run_bb_cli_bench {
+  local runtime="$1"
+  local output="$2"
+  local args="$3"
+  export MAIN_ARGS="$args"
+
+  if [[ "$runtime" == "native" ]]; then
+    memusage "./$native_build_dir/bin/bb_cli_bench" \
+      --benchmark_out=$output/op-counts.json \
+      --benchmark_out_format=json || {
+      echo "bb_cli_bench native failed with args: $args"
       exit 1
     }
-  }
-
-  bb_cli_bench_native prove -o "bench-out/$flow-proof-files" -b "$flow_folder/acir.msgpack" -w "$flow_folder/witnesses.msgpack" --scheme client_ivc --input_type runtime_stack
-  local end=$(date +%s%N)
-  local elapsed_ns=$(( end - start ))
-  local elapsed_ms=$(( elapsed_ns / 1000000 ))
-  local memory_taken_mb=$(cat "$MEMUSAGE_OUT")
-  echo "$flow (native) has proven in $((elapsed_ms / 1000))s and peak memory of ${memory_taken_mb}MB."
-  dump_fail "verify_ivc_flow $flow bench-out/$flow-proof-files/proof"
-  echo "$flow (native) has verified."
-  cat > "./bench-out/ivc/$flow-ivc-native.json" <<EOF
-  {
-    "benchmarks": [
-    {
-      "name": "$flow-ivc-proof",
-      "time_unit": "ms",
-      "real_time": ${elapsed_ms}
-    },
-    {
-      "name": "$flow-ivc-proof-memory",
-      "time_unit": "MB",
-      "real_time": ${elapsed_ms}
+  else # wasm
+    export WASMTIME_ALLOWED_DIRS="--dir=$flow_folder --dir=$output"
+    # TODO support wasm op count time preset
+    memusage scripts/wasmtime.sh $WASMTIME_ALLOWED_DIRS ./build-wasm-threads/bin/bb_cli_bench \
+      --benchmark_out=$output/op-counts.json \
+      --benchmark_out_format=json || {
+      echo "bb_cli_bench wasm failed with args: $args"
+      exit 1
     }
-    ]
-  }
-EOF
+  fi
 }
 
-function client_ivc_flow_wasm {
+function client_ivc_flow {
   set -eu
-  local flow=$1
+  local runtime="$1"
+  local flow="$2"
   local flow_folder="$input_folder/$flow"
   local start=$(date +%s%N)
-  mkdir -p "bench-out/$flow-proof-files"
-  export MEMUSAGE_OUT="bench-out/$flow-proof-files/peak-memory-wasm-mb.txt"
 
-  function bb_cli_bench_wasm {
-    export MAIN_ARGS="$*"
-    export WASMTIME_ALLOWED_DIRS="--dir=$HOME/.bb-crs --dir="$flow_folder" --dir=bench-out/$flow-proof-files"
-    memusage scripts/wasmtime.sh $WASMTIME_ALLOWED_DIRS ./build-wasm-threads/bin/bb_cli_bench \
-        --benchmark_out=bench-out/$flow-proof-files/op-counts.json \
-        --benchmark_out_format=json || {
-      echo "bb_cli_bench failed with args: $*"
-      exit 1
-    }
-  }
-  bb_cli_bench_wasm prove -o "bench-out/$flow-proof-files" -b "$flow_folder/acir.msgpack" -w "$flow_folder/witnesses.msgpack" --scheme client_ivc --input_type runtime_stack
+  local output="bench-out/ivc-$flow-$runtime"
+  rm -rf "$output"
+  mkdir -p "$output"
+  export MEMUSAGE_OUT="$output/peak-memory-$runtime-mb.txt"
+
+  run_bb_cli_bench "$runtime" "$output" "prove -o $output --ivc_inputs_path $flow_folder/ivc-inputs.msgpack --scheme client_ivc"
+
+  if [ -f "$output/op-counts.json" ]; then
+    scripts/google-bench/summarize-op-counts "$output/op-counts.json"
+  fi
+
   local end=$(date +%s%N)
   local elapsed_ns=$(( end - start ))
   local elapsed_ms=$(( elapsed_ns / 1000000 ))
   local memory_taken_mb=$(cat "$MEMUSAGE_OUT")
-  echo "$flow (WASM) has proven in ${elapsed_ms}ms and peak memory of ${memory_taken_mb}MB."
-  dump_fail "verify_ivc_flow $flow bench-out/$flow-proof-files/proof"
-  echo "$flow (WASM) has verified."
-  cat > "./bench-out/ivc/$flow-ivc-wasm.json" <<EOF
+
+  echo "$flow ($runtime) has proven in $((elapsed_ms / 1000))s and peak memory of ${memory_taken_mb}MB."
+  dump_fail "verify_ivc_flow $flow $output/proof"
+  echo "$flow ($runtime) has verified."
+
+  local runtime_suffix=""
+  [[ "$runtime" == "wasm" ]] && runtime_suffix="-wasm"
+
+  cat > "$output/benchmarks.json" <<EOF
   {
     "benchmarks": [
     {
-      "name": "$flow-ivc-proof-wasm",
+      "name": "$flow-ivc-proof$runtime_suffix",
       "time_unit": "ms",
       "real_time": ${elapsed_ms}
     },
     {
-      "name": "$flow-ivc-proof-wasm-memory",
+      "name": "$flow-ivc-proof$runtime_suffix-memory",
       "time_unit": "MB",
       "real_time": ${memory_taken_mb}
     }
@@ -140,7 +127,7 @@ function run_benchmark {
   taskset -c $start_core-$end_core bash -c "$2"
 }
 
-export -f verify_ivc_flow client_ivc_flow_native client_ivc_flow_wasm run_benchmark
+export -f verify_ivc_flow client_ivc_flow run_bb_cli_bench run_benchmark
 
 # TODO this does not work with smaller core counts - we will soon have a benchmark-wide mechanism for this.
 num_cpus=$(get_num_cpus)
@@ -149,15 +136,16 @@ jobs=$((num_cpus / HARDWARE_CONCURRENCY))
 # Split up the flows into chunks to run in parallel - otherwise we run out of CPUs to pin.
 if [ -n "${IVC_BENCH:-}" ]; then
   # If IVC_BENCH is set, run only that benchmark.
-  run_benchmark 1 "client_ivc_flow_native $IVC_BENCH"
-  run_benchmark 1 "client_ivc_flow_wasm $IVC_BENCH"
+  run_benchmark 1 "client_ivc_flow native $IVC_BENCH"
+  run_benchmark 1 "client_ivc_flow wasm $IVC_BENCH"
 else
-  parallel -v --line-buffer --tag --jobs "$jobs" run_benchmark {#} '"client_ivc_flow_native {}"' ::: $(ls "$input_folder")
-  parallel -v --line-buffer --tag --jobs "$jobs" run_benchmark {#} '"client_ivc_flow_wasm {}"' ::: $(ls "$input_folder")
+  for runtime in native wasm; do
+    parallel -v --line-buffer --tag --jobs "$jobs" run_benchmark {#} '"client_ivc_flow '$runtime' {}"' ::: $(ls "$input_folder")
+  done
 fi
 
-mkdir -p "$output_folder"
+mkdir -p "$benchmark_output"
 
 ../scripts/combine_benchmarks.py \
-  ./bench-out/ivc/*.json \
-  > $output_folder/ivc-bench.json
+  ./bench-out/*/benchmarks.json \
+  > $benchmark_output/ivc-bench.json