fix(api): reaggregate resource inventory and attack surface after muting findings#10843
Merged
AdriiiPRodri merged 8 commits intoApr 27, 2026
Merged
Conversation
Contributor
|
✅ All necessary |
Contributor
|
✅ Conflict Markers Resolved All conflict markers have been successfully resolved in this pull request. |
AdriiiPRodri
added a commit
that referenced
this pull request
Apr 22, 2026
Contributor
🔒 Container Security ScanImage: 📊 Vulnerability Summary
4 package(s) affected
|
AdriiiPRodri
added a commit
that referenced
this pull request
Apr 22, 2026
c9c28ee to
ed54d23
Compare
…ing findings Extend `reaggregate_all_finding_group_summaries_task` (already chained after `mute_historical_findings_task`) so that, for each latest scan per `(provider, day)`, the same per-scan aggregation pipeline that scan completion runs is re-executed against `ScanGroupSummary`, `ScanCategorySummary` and `AttackSurfaceOverview` too. Without this, `/overviews/resource-groups` (resource inventory), `/overviews/categories` and `/overviews/attack-surfaces` kept pre-mute totals until the next scan. Make the three non-idempotent aggregators safe to re-run: - `aggregate_attack_surface`, `backfill_scan_resource_group_summaries` and `backfill_scan_category_summaries` now delete the scan's existing rows before `bulk_create`. A plain INSERT or silent `ignore_conflicts=True`/`already backfilled` short-circuit would either violate the unique constraint (aborting the Celery chain and skipping downstream aggregators) or silently no-op, leaving the pre-aggregated tables stale after the mute. Tests cover the new dispatch fan-out and the updated backfill behaviour.
ed54d23 to
14321fd
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #10843 +/- ##
==========================================
+ Coverage 93.64% 93.65% +0.01%
==========================================
Files 230 230
Lines 32987 33058 +71
==========================================
+ Hits 30890 30961 +71
Misses 2097 2097
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
josema-xyz
reviewed
Apr 22, 2026
…iew queue Instead of DELETE+INSERT, `aggregate_findings`, `aggregate_attack_surface`, `backfill_scan_resource_group_summaries` and `backfill_scan_category_summaries` now write via `bulk_create(update_conflicts=True, ...)`. The DELETE+INSERT pattern was vulnerable to a race between two concurrent writers on the same scan (e.g. scan completion overlapping with post-mute reaggregation): the second task's INSERT would hit the `unique_*_per_scan` constraint after the first committed and abort the Celery chain, leaving downstream aggregators unexecuted. Upsert is atomic in PostgreSQL and safe for the pre-aggregated tables here because the underlying aggregators seed every (combination-key) unconditionally, so muting never drops a key and no row is left orphan. Also move `backfill_scan_resource_group_summaries_task` and `backfill_scan_category_summaries_task` from the `backfill` queue to `overview`, matching their sibling per-scan aggregators (`perform_scan_summary_task`, `aggregate_daily_severity_task`, `aggregate_finding_group_summaries_task`, `aggregate_attack_surface_task`). These wrappers had no dispatchers outside the post-mute reaggregation chain, so the queue rename is safe and removes the `.set(queue=...)` dispatch-site hack.
…ign Celery task names These two aggregators were misnamed as "backfill" but are, in practice, the primary per-scan aggregators for `ScanGroupSummary` and `ScanCategorySummary` -- same role as `aggregate_findings`, `aggregate_daily_severity`, `aggregate_finding_group_summaries` and `aggregate_attack_surface`. Rename to reflect that: - Python: `backfill_scan_category_summaries` -> `aggregate_scan_category_summaries`; `backfill_scan_resource_group_summaries` -> `aggregate_scan_resource_group_summaries`; and the matching `*_task` wrappers. - Celery task names: `backfill-scan-category-summaries` -> `scan-category-summaries`; `backfill-scan-resource-group-summaries` -> `scan-resource-group-summaries`. These slot in with the sibling `scan-summary`, `scan-daily-severity`, `scan-finding-group-summaries`, `scan-attack-surface-overviews` naming. Safe because the old task wrappers had no dispatchers outside the post-mute reaggregation chain introduced in this PR -- no queued tasks with the old name exist during deploy. Updates all imports/callers in `tasks.py`, `conftest.py`, `test_backfill.py` and `test_tasks.py`.
josema-xyz
reviewed
Apr 27, 2026
- Assert ScanSummary fail/muted move when a finding is muted mid-rerun - Assert ScanCategorySummary counters drop to zero after mute - Assert ScanGroupSummary counters drop to zero after mute - Reference compliance fixture explicitly to satisfy vulture
josema-xyz
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Follow-up to #10827, which made Overview and Finding Groups reflect newly-muted findings by extending
reaggregate_all_finding_group_summaries_task. While validating that fix we saw that several other endpoints backed by their own pre-aggregated tables were still stale after muting:/overviews/resource-groups(resource inventory widget)/overviews/categories/overviews/attack-surfaces/overviews/findings-severity/timeseries(severity timeline)Both the reaggregation dispatch and the underlying aggregators were affected.
Description
Two-part fix:
Dispatch: the post-mute
reaggregate-all-finding-group-summariestask now also re-runs, for each latest scan per(provider, day):backfill_scan_resource_group_summaries_task->ScanGroupSummarybackfill_scan_category_summaries_task->ScanCategorySummaryaggregate_attack_surface_task->AttackSurfaceOverviewServices watchlist and severity timeseries were already covered, since they read from
ScanSummaryandDailySeveritySummarywhich the pipeline was recomputing.Idempotency: four aggregators used plain
bulk_create(or aalready backfilledshort-circuit plus silentignore_conflicts=True). The first post-mute run tripped theunique_*constraints, aborting the Celery chain and leaving the dependent aggregators unexecuted (observable as a PostgreSQLduplicate key value violates unique constraint "unique_scan_summary"during the first reaggregation attempt). They now delete the scan's existing rows beforebulk_createso the write is atomic and re-runnable, and dropped combinations no longer linger in the summary table:aggregate_findings(scan.py)aggregate_attack_surface(scan.py)backfill_scan_resource_group_summaries(backfill.py)backfill_scan_category_summaries(backfill.py)Other pre-aggregated tables (
ResourceScanSummary,ComplianceOverviewSummary,ProviderComplianceScore) do not depend onFinding.mutedand are intentionally left out of the reaggregation fan-out.Steps to review
resource_groupsvalues are populated)./overviews/resource-groups,/overviews/categories,/overviews/attack-surfacesand/overviews/findings-severity/timeseries. Note the totals.POST /mute-ruleson a subset of those finding IDs.overviewqueue to drain (docker compose logs worker | grep -E "reaggregate|scan-(summary|daily-severity|finding-group-summaries|resource-group-summaries|category-summaries|attack-surface)").duplicate key value violates unique constrainterrors appear in the worker log during step 4.Automated:
Checklist
Community Checklist
API
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.