Repository navigation
fix(storage): dedupe edge endpoint ids before AGE batch lookup - #522
Open
pcbeingused333 wants to merge 1 commit into
Open
pcbeingused333 wants to merge 1 commit into
pcbeingused333 wants to merge 1 commit into
Conversation
flush_graph_mutations collected the source and target of every edge upsert into one list. A node shared by k edges appeared k times, and the get_edges_for_nodes_batch join returned an edge between two such nodes k_src * k_tgt times, inflating memory on dense documents. Sort and dedup the ids first. Refs raphaelmansuy#511 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Author
|
Hi, friendly ping on this one when you have a moment. Happy to adjust anything or rebase if needed. Thanks for maintaining the project! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AgeGraphProjectionApplier::flush_graph_mutationsbuilds the list of endpoint ids forget_edges_for_nodes_batchby flat-mapping the source and target of every pending edge upsert. A node used bykedges therefore appearsktimes in the list. On the AGE path the query unnests that array and joins"EDGE"to it on both ends, so an edge between two repeated nodes comes backk_src * k_tgttimes: on an entity-dense document the lookup result (and memory) grows with the square of the repetition instead of with the number of edges, which is the first half of #511.The fix moves the collection into
edge_upsert_endpoint_ids, which sorts and dedups the ids before the lookup, the same thingresolve_edge_deletesalready does for the delete path. Behaviour for distinct ids is unchanged.This only addresses the duplicate-id amplification. The missing retry ceiling in the projection worker (second half of the issue) is not touched here.
Refs #511
Type of change
Checklist
Additional context
New
edge_upsert_endpoint_testsinprojection/appliers.rs: edgesA-B,B-A,A-CandA-Bwith another relation must yield["A", "B", "C"], plus the empty case. Without the sort/dedup the first test gets 8 ids back and fails; with it both pass (cargo test -p edgequake-storage --lib --features postgres edge_upsert_endpoint).rustfmt --checkis clean and the crate builds with no new warnings.This change was prepared with AI assistance (Claude Code).