Retro Funding Season 7 (S7) Data and Algorithms

Quickstart

Clone the repo:

gh repo clone ethereum-optimism/Retro-Funding

Navigate to the repo:

cd Retro-Funding

Install the dependencies:

poetry install

Activate the virtual environment:

poetry env activate

Then, you can execute the commands below to test the simulation pipeline. For example:

python eval-algos/S7/utils/process_onchain_builders.py --measurement-period M4 --model onchain__goldilocks

Simulation Pipeline

For each measurement period (e.g. M1, M2, etc.), the maintainers of this repository fetch a snapshot of the data, review the model weights, and run the scoring algorithm.

Directory Structure

The project directory is organized as follows:

Retro-Funding/
├── docs/
├── eval-algos/
│   └── S7/
│       ├── models/           # Core model implementations
│       ├── utils/            # Utility functions and scripts
└── results/
    └── S7/
        ├── M1/               # Measurement period
        |   ├── data/         # Input data
        |   ├── weights/      # Model configuration files
        |   └── outputs/      # Output results
        └── other_measurement_periods/

Data Snapshots

We add raw data snapshots in the data directory associated with each measurement period. The following files are included for each measurement period:

• onchain__project_metadata.csv: Metadata about each project that applied to the Onchain Builders round.
• onchain__metrics_by_project.csv: Chain-level, month-by-month metrics about each project in the Onchain Builders round.
• devtooling__project_metadata.csv: Metadata about each project that applied to the Devtooling round.
• devtooling__onchain_metadata.csv: Metadata about each onchain project that was linked to a devtooling project.
• devtooling__dependency_graph.csv: A graph of the dependencies between devtooling and onchain projects.
• devtooling__developer_graph.csv: A graph of the developers who have committed code to onchain projects and engaged with devtooling projects.
• devtooling__raw_metrics: A summary of the raw (graph-level) metrics for each devtooling project, in JSON format.

Note that the project_id for each project is generated by OSO and is not the same as the project_id used by OP Atlas. To get the OP Atlas identifier for a project, query the project_name field in the project_metadata tables. The display_name is the name of the project as it will appear on OP Atlas's website.

Important

The reason for using snapshots instead of live data is because the underlying mappings between projects and their artifacts (i.e., contracts, repos, packages, etc.) are continuously changing. Therefore, each snapshot represents the state of which projects are enrolled in the round, which artifacts are associated with those projects, and which metrics are publicly available about those artifacts.

You can also query Retro Funding data directly from OSO by following the guides here.

There are test datasets as well for testing the simulation pipeline.

Fetching Data

To fetch the latest data for a specific measurement period, you need to:

1. Add your OSO API key to a .env file in the root directory:

   OSO_API_KEY=your_api_key_here
   

2. Run the fetch_data.py script with the desired measurement period. For example:

   python eval-algos/S7/utils/fetch_data.py --measurement-period M4

This will download the data from OSO and save it to the results/S7/M4/data/ directory. The script will fetch all the necessary CSV files for both the onchain builders and devtooling rounds.

Weights

The weights for each model are stored in the results/S7/<measurement_period>/weights directory.

Be sure to use the right data snapshot and simulation period for your simulation.

data_snapshot:
  data_dir: 'results/S7/M4/data/'
  projects_file: 'onchain__project_metadata.csv'
  metrics_file: 'onchain__metrics_by_project.csv'

simulation:
  periods:
    previous: 'Apr 2025'
    current: 'May 2025'

You can also modify the weights for each model as you see fit. Note that new metrics may be added throughout the season. For example, the onchain__metrics_by_project.csv in M2 includes new metrics related to Worldchain activity.

  metrics:
    amortized_contract_invocations_monthly: 0.25
    gas_fees_monthly: 0.25
    average_tvl_monthly: 0.25
    active_farcaster_users_monthly: 0.25
    
    qualified_addresses_monthly: 0.0
    active_addresses_monthly: 0.0
    contract_invocations_monthly: 0.0
  
  # Variant weights must sum to 1.0
  metric_variants:
    adoption: 0.20    # Current period value
    growth: 0.20      # Change from previous period
    retention: 0.60   # Minimum of current and previous

Allocation Settings

Each model configuration file includes allocation settings that determine how funding is distributed:

allocation:
  budget: 1333333.3333333333  # In OP tokens
  min_amount_per_project: 200 # In OP tokens
  max_share_per_project: 0.05 # Percentage of the total budget

These settings control:

• The total budget available for allocation
• The minimum amount each project can receive
• The maximum share of the budget any single project can receive

Run the Pipeline

To run the simulation with a specific model and measurement period:

python eval-algos/S7/utils/process_onchain_builders.py --measurement-period M4 --model onchain__goldilocks
python eval-algos/S7/utils/process_devtools.py --measurement-period M4 --model devtooling__arcturus

This will:

1. Load the model configuration from results/S7/M4/weights/<model>.yaml
2. Process the data according to the model's algorithm
3. Save the results to results/S7/M4/outputs/<model>_rewards.csv

Consolidating and Serializing Results

After running the simulation pipeline, you can consolidate and serialize the results using the following utilities:

Consolidating Rewards

The consolidate_rewards.py script combines all rewards files from a measurement period into a single CSV file:

python eval-algos/S7/utils/consolidate_rewards.py --measurement-period M4

This will:

1. Find all rewards CSV files in the results/S7/M4/outputs/ directory
2. Standardize the format (ensuring consistent column names)
3. Add round_id ('7' for devtooling, '8' for onchain) and filename columns
4. Save the consolidated data to results/S7/M4/outputs/M4_consolidated_rewards.csv

Serializing Results

The serialize.py script creates JSON files that combine metrics and rewards data:

python eval-algos/S7/utils/serialize.py --measurement-period M4

This will:

1. Use the consolidated rewards file (or create it if it doesn't exist)
2. Create devtooling__results.json with devtooling metrics merged with round_id '7' rewards
3. Create onchain__results.json with onchain metrics merged with round_id '8' rewards

These JSON files are used for displaying results on OP Atlas and for further analysis.

Model Details

Onchain Builders

The Onchain Builders model analyzes and scores active Superchain projects based on their onchain activity. It processes raw metrics through a multi-step pipeline to produce normalized, weighted scores that reflect project performance across different chains and time periods.

Pipeline Steps

1. Filter and Pivot Raw Metrics

• Takes raw metrics data with non-zero weights for specified measurement periods
• Pivots data by chain and metric to create a structured view
• Groups by project_id, project_name, display_name, and chain

2. Sum and Weight by Chain

• Applies chain-specific weights (e.g., different weights for OP Mainnet vs other chains)
• Sums weighted metrics across all chains for each project
• Preserves project metadata in the aggregation

3. Calculate Metric Variants

For each metric, computes three variants:

• Adoption: Current period value
• Growth: Positive difference between current and previous period values
• Retention: Minimum value between current and previous periods

4. Normalize Metric Variants

• Applies min-max normalization to each metric variant
• Scales values to [0,1] range while preserving null values (e.g., TVL for non-DeFi projects)
• Optionally caps values at a specified percentile (e.g., 97th percentile)
• Uses fallback center value (0.5) when range is zero

5. Apply Weights

• Multiplies each normalized metric variant by:
  • Its metric-specific weight
  • Its variant-specific weight (adoption/growth/retention)

6. Aggregate Final Scores

• Combines weighted variants using power mean (p=2)
• Normalizes final scores to sum to 1.0

7. Prepare Results

• Flattens multi-level columns for readability
• Merges intermediate results for transparency
• Sorts projects by final weighted score

Devtooling OpenRank

The Devtooling model evaluates open-source developer tools by analyzing their relationships with onchain projects through package dependencies and developer contributions. It uses an EigenTrust-based algorithm released by OpenRank to propagate value through the graph.

Pipeline Steps

1. Build Unweighted Graph

Constructs a directed graph with three types of edges:

• Package Dependencies: Onchain projects → Devtooling projects
• Commit Events: Onchain projects → Developers
• GitHub Engagement: Developers → Devtooling projects

Removes duplicate edges when an onchain project is also a devtooling project.

2. Compute Onchain Project Pretrust

• Uses economic metrics from onchain projects
• Applies log transformation and min-max scaling
• Combines metrics using configured weights
• Normalizes scores to sum to 1.0

3. Compute Devtooling Project Pretrust

• Uses GitHub metrics (num packages, stars, forks, etc.)
• Applies log transformation and min-max scaling
• Combines metrics using configured weights
• Normalizes scores to sum to 1.0

4. Compute Developer Reputation

• Developers are pre-filtered to focus on active developers committing code to onchain project repos in Rust, Solidity, TypeScript, and Vyper
• Distributes onchain project pretrust to developers based on commit activity

5. Weight Edges

Applies weights based on:

• Link type (package dependency, commit, GitHub engagement)
• Event type (NPM, CARGO, COMMIT_CODE)
• Time decay for non-static relationships
• More recent interactions contribute more weight in most cases

6. Apply EigenTrust

• Combines pretrust scores from all sources
• Runs EigenTrust propagation on weighted graph
• Computes final trust scores for each node

7. Rank and Evaluate Projects

• Ranks devtooling projects by final EigenTrust scores
• Applies eligibility criteria:
  • Minimum package dependency count
  • Minimum developer link count
• Normalizes scores among eligible projects

8. Serialize Value Flow

• Uses iterative proportional fitting (IPF)
• Creates detailed value flow attribution
• Ensures contribution sums match:
  • Per devtool: Sum equals its overall score
  • Per onchain project: Sum equals its pretrust