Skip to content

optimize using kth algorithm#9599

Merged
macfarla merged 5 commits intobesu-eth:mainfrom
cuiweixie:kth
Jan 9, 2026
Merged

optimize using kth algorithm#9599
macfarla merged 5 commits intobesu-eth:mainfrom
cuiweixie:kth

Conversation

@cuiweixie
Copy link
Copy Markdown
Contributor

@cuiweixie cuiweixie commented Jan 4, 2026

PR description

Fixed Issue(s)

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • spotless: ./gradlew spotlessApply
  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests
  • hive tests: Engine or other RPCs modified?

Copilot AI review requested due to automatic review settings January 4, 2026 11:58
@cuiweixie
Copy link
Copy Markdown
Contributor Author

cuiweixie commented Jan 4, 2026

benchmark code:

/*
 * Copyright contributors to Besu.
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
 * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
 * specific language governing permissions and limitations under the License.
 *
 * SPDX-License-Identifier: Apache-2.0
 */
package org.hyperledger.besu.ethereum.api.query;

import org.hyperledger.besu.datatypes.Wei;
import org.hyperledger.besu.util.OrderStatistics;

import java.util.Arrays;
import java.util.SplittableRandom;
import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;

/**
 * A smaller parameter set than {@link GasPriceSelectionBenchmark} for quicker local runs.
 *
 * <p>Run: {@code ./gradlew :ethereum:api:jmh -Pincludes=GasPriceSelectionBenchmarkQuick}
 */
@State(Scope.Thread)
@Warmup(iterations = 2, time = 1, timeUnit = TimeUnit.SECONDS)
@OutputTimeUnit(value = TimeUnit.NANOSECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@BenchmarkMode(Mode.AverageTime)
public class GasPriceSelectionBenchmarkQuick {

  public enum Distribution {
    RANDOM,
    DUPLICATES
  }

  @Param({"1024", "32768"})
  public int size;

  @Param({"0.6", "0.9"})
  public double fraction;

  @Param({"RANDOM", "DUPLICATES"})
  public Distribution distribution;

  private Wei[] base;
  private int k;

  @Setup(Level.Trial)
  public void setup() {
    k = Math.min(size - 1, (int) (size * fraction));

    final long[] data = new long[size];
    final SplittableRandom rnd = new SplittableRandom(1L);

    if (distribution == Distribution.DUPLICATES) {
      for (int i = 0; i < size; i++) {
        data[i] = rnd.nextLong(0, 128);
      }
    } else {
      for (int i = 0; i < size; i++) {
        data[i] = rnd.nextLong(0, Long.MAX_VALUE);
      }
    }

    // Shuffle to avoid being already sorted.
    shuffleInPlace(data, rnd.split());

    base = new Wei[size];
    for (int i = 0; i < size; i++) {
      base[i] = Wei.of(data[i]);
    }
  }

  @Benchmark
  public void sortThenPick(final Blackhole bh) {
    final Wei[] copy = base.clone();
    Arrays.sort(copy);
    bh.consume(copy[k]);
  }

  @Benchmark
  public void kthPick(final Blackhole bh) {
    final Wei[] copy = base.clone();
    bh.consume(OrderStatistics.selectKthInPlace(copy, k));
  }

  private static void shuffleInPlace(final long[] a, final SplittableRandom rnd) {
    for (int i = a.length - 1; i > 0; i--) {
      final int j = rnd.nextInt(i + 1);
      final long tmp = a[i];
      a[i] = a[j];
      a[j] = tmp;
    }
  }
}

result in:

sort vs kth

distribution fraction size sortThenPick(ns) kthPick(ns) 提升倍数(sort/kth)
RANDOM 0.6 1024 357,908.897 81,626.937 4.38x
RANDOM 0.6 32768 19,627,828.974 3,140,656.417 6.25x
RANDOM 0.9 1024 356,960.112 118,311.400 3.02x
RANDOM 0.9 32768 19,800,297.082 1,946,121.699 10.17x
DUPLICATES 0.6 1024 447,849.719 107,948.064 4.15x
DUPLICATES 0.6 32768 18,513,831.108 3,847,787.384 4.81x
DUPLICATES 0.9 1024 446,585.869 74,765.710 5.97x
DUPLICATES 0.9 32768 18,416,303.801 3,932,379.648 4.68x

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@cuiweixie cuiweixie requested a review from Copilot January 5, 2026 14:10
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@cuiweixie cuiweixie requested a review from Copilot January 6, 2026 00:26
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@cuiweixie
Copy link
Copy Markdown
Contributor Author

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

it do not show why it failed.can someone help me?

@cuiweixie cuiweixie requested a review from Copilot January 6, 2026 05:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: cuiweixie <cuiweixie@gmail.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 6, 2026

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

it do not show why it failed.can someone help me?

You can ignore the copilot error. It's due to restricted permissions of GHA in besu repo.

@cuiweixie can you explain rationale for this PR?

@cuiweixie
Copy link
Copy Markdown
Contributor Author

cuiweixie commented Jan 6, 2026

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

it do not show why it failed.can someone help me?

You can ignore the copilot error. It's due to restricted permissions of GHA in besu repo.

@cuiweixie can you explain rationale for this PR?

Hello, this pr use kth alogrithm to select the kth element such as gas price list. here is the benchmark data:

sort vs kth

distribution fraction size sortThenPick(ns) kthPick(ns) 提升倍数(sort/kth)
RANDOM 0.6 1024 357,908.897 81,626.937 4.38x
RANDOM 0.6 32768 19,627,828.974 3,140,656.417 6.25x
RANDOM 0.9 1024 356,960.112 118,311.400 3.02x
RANDOM 0.9 32768 19,800,297.082 1,946,121.699 10.17x
DUPLICATES 0.6 1024 447,849.719 107,948.064 4.15x
DUPLICATES 0.6 32768 18,513,831.108 3,847,787.384 4.81x
DUPLICATES 0.9 1024 446,585.869 74,765.710 5.97x
DUPLICATES 0.9 32768 18,416,303.801 3,932,379.648 4.68x

I think i might have some perf improvement.

@cuiweixie
Copy link
Copy Markdown
Contributor Author

@macfarla Any chance to get merge?

@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 7, 2026

Hello, this pr use kth alogrithm to select the kth element such as gas price list. here is the benchmark data:

what RPCs will show the performance improvement?

@cuiweixie
Copy link
Copy Markdown
Contributor Author

Hello, this pr use kth alogrithm to select the kth element such as gas price list. here is the benchmark data:

what RPCs will show the performance improvement?

It might have minor improve, but I don't know much about how to bench this rpc.

@macfarla macfarla self-assigned this Jan 7, 2026
@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 7, 2026

These RPCs should benefit:

  1. eth_gasPrice - Returns current gas price estimate
    - Calls blockchainQueries.gasPrice() → uses optimized kth selection
    - Impact: 9-30x faster gas price calculations
  2. eth_maxPriorityFeePerGas - Returns max priority fee estimate
    - Calls blockchainQueries.gasPriorityFee() → uses optimized kth selection
    - Impact: Similar performance improvement for EIP-1559 fee estimation

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 7, 2026

similar change in neth NethermindEth/nethermind#10100
also trying to add it to the go standard lib
golang/go#77055

@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 8, 2026

I ran hive tests, no changes

@cuiweixie
Copy link
Copy Markdown
Contributor Author

I ran hive tests, no changes

how many element in kth select array?

@macfarla
Copy link
Copy Markdown
Contributor

macfarla commented Jan 8, 2026

I ran hive tests, no changes

how many element in kth select array?

I ran rpc-compat tests which test a number of RPCs. I don't think there's any that test this code actively so more of a sanity check I guess. The rpc-compat test results are here https://hive.ethpandaops.io/#/group/generic

@macfarla macfarla enabled auto-merge (squash) January 9, 2026 00:03
@macfarla macfarla merged commit 4062386 into besu-eth:main Jan 9, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants