PYTHON-5253 Automated Spec Test Sync #2409

sleepyStick · 2025-06-25T21:51:27Z

Generates at most one weekly PR on Monday's at 9am pst (assuming the evergreen hosts are in utc time...).

At a high level, I've written a script that simply loops through pymongo's existing spec tests and calls the existing bash resync-spec.sh to re-sync all the specs. Then if changes were made, a PR would be created. If all specs are up to date, then no PR will be made.

This PR will be created by mongodb-drivers-pr-bot with the changes. If an error occurred during the syncing of a spec, the PR description will display the name of the spec along with stderr from the bash resync-spec.sh <spec> command.

An example of a generated PR #2407

…thon-driver into PYTHON-5253

sleepyStick

Added some comments that I think might help the reviewer :)

sleepyStick · 2025-06-26T16:44:33Z

.evergreen/resync-specs.sh

-cd $SPECS/source
-make
-cd -
+if ! [ -n "${CI:-}" ]


CI is always defined in an EVG run and we only want to run these things in a local checkout.

sleepyStick · 2025-06-26T16:46:13Z

.evergreen/scripts/create-pr.sh

+#!/usr/bin/env bash
+
+tools="../drivers-evergreen-tools"
+git clone https://github.com/mongodb-labs/drivers-evergreen-tools.git $tools


cloning drivers evergreen tools because there are some scripts (mainly ./secrets-export.sh and get-access-token.sh that are there and i'd rather not copy it over)

sleepyStick · 2025-06-26T16:49:24Z

.evergreen/scripts/resync-all-specs.py

+        and len(list(os.scandir(pathlib.Path(entry.path) / "tests"))) > 1
+    }
+    test_set = {entry.name.replace("-", "_") for entry in os.scandir(directory) if entry.is_dir()}
+    known_mappings = {


I think there's a slight difference in how we name the directories vs how the spec names them sometimes? (Not 100% sure if these are correct though, so please let me know if these need to be changed!)

Yes we often have different names than the spec repo, that's expected.

sleepyStick · 2025-06-26T16:58:48Z

.evergreen/scripts/resync-all-specs.py

+        if spec.name in ["asynchronous"]:
+            continue
+        process = subprocess.run(
+            ["bash", "./.evergreen/resync-specs.sh", spec.name],  # noqa: S603, S607


Note:

S603 `subprocess` call: check for execution of untrusted input S607 Starting a process with a partial executable path

S603: I think spec.name is safe because they're only acquired from our directories within test
S607: I think bash is sufficient for executable (they want the whole path)

sleepyStick · 2025-06-26T17:01:43Z

.evergreen/scripts/resync-all-specs.py

+            errored[spec.name] = process.stderr
+
+    process = subprocess.run(
+        ["git diff --name-only | awk -F'/' '{print $2}' | sort | uniq"],  # noqa: S607


Note:

S607 Starting a process with a partial executable path

I tried to separate the string into ["git", "diff --name-only | awk -F'/' '{print $2}' | sort | uniq"] but that caused the command to fail? So I kept it as a whole string..

Each space-separated word has to be a separate arg, keeping it as one here is probably more readable.

sleepyStick · 2025-06-26T17:03:06Z

.evergreen/specs.patch

@@ -0,0 +1,1327 @@
+diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json


This is a file of known spec test differences that we don't want to be committed. If any one of these is incorrect, or I'm missing some, lmk!

We should add a section to CONTRIBUTING.md detailing the why and how of modifying this file. It should also contain the policy we follow summarized by Shane in the Slack thread.

Good point. Just added a blurb, but lmk if the language is unclear or anything like that. (I feel like my technical writing skills are lacking sometimes)

NoahStapp

Great work! Some minor comments.

NoahStapp · 2025-06-26T18:54:10Z

.evergreen/scripts/resync-all-specs.py

+            ["bash", "./.evergreen/resync-specs.sh", spec.name],  # noqa: S603, S607
+            capture_output=True,
+            text=True,
+            check=False,


What's the motivation behind check=False here and then checking the returncode explicitly instead of using check=True?

err no motivation (I learned about check when ruff yelled at me for it so) Changed it to check=True inside of a try catch!

NoahStapp · 2025-06-26T18:54:47Z

.evergreen/scripts/resync-all-specs.py

+            errored[spec.name] = process.stderr
+
+    process = subprocess.run(
+        ["git diff --name-only | awk -F'/' '{print $2}' | sort | uniq"],  # noqa: S607


Each space-separated word has to be a separate arg, keeping it as one here is probably more readable.

NoahStapp · 2025-06-26T18:56:02Z

.evergreen/scripts/resync-all-specs.py

+        and len(list(os.scandir(pathlib.Path(entry.path) / "tests"))) > 1
+    }
+    test_set = {entry.name.replace("-", "_") for entry in os.scandir(directory) if entry.is_dir()}
+    known_mappings = {


Yes we often have different names than the spec repo, that's expected.

NoahStapp · 2025-06-27T17:58:21Z

.evergreen/specs.patch

@@ -0,0 +1,1327 @@
+diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json


We should add a section to CONTRIBUTING.md detailing the why and how of modifying this file. It should also contain the policy we follow summarized by Shane in the Slack thread.

NoahStapp

Some wording updates to more clearly communicate the structure and purpose of our new patch files.

NoahStapp · 2025-06-27T19:41:25Z

CONTRIBUTING.md

@@ -432,6 +432,32 @@ update in PyMongo. This is primarily helpful if you are implementing a
 new feature in PyMongo that has spec tests already implemented, or if
 you are attempting to validate new spec tests in PyMongo.

+### Automated Specification Test Resyncing


This section should focus more on the process and what this automated syncing does, rather than the literal sequence of scripts and operations it performs.

CONTRIBUTING.md

NoahStapp · 2025-06-27T19:50:12Z

CONTRIBUTING.md

+a new ticket to unskip these.
+
+#### Adding to a patch file
+Assuming the changes are committed somewhere, to add to any of the


It's a little unclear what this is saying. Can you provide a small example of what this should look like?

Co-authored-by: Noah Stapp <[email protected]>

sleepyStick and others added 19 commits May 7, 2025 10:14

WIP

fe1b07d

trying to get it to work locally?

f822f31

wip

6909518

still a wip

73139a4

WIP

b3a2b29

trying to get it to work locally?

49685ae

wip

076ed93

still a wip

07ea1f6

add patch file, use comment bot, still a wip

f4b3ddc

Merge branch 'PYTHON-5253' of https://github.com/sleepyStick/mongo-py…

547dd76

…thon-driver into PYTHON-5253

Merge branch 'mongodb:master' into PYTHON-5253

a97d5f4

fun fact, using a python script is actually *way* easier

925aeb5

remove comment

67b995a

remove comment pt2

53e8945

9am pst is 4pm utc i think?

9f6f727

remove unused vars and cleanup debugging prints

3144383

oops accidentally deleted this line

2bea24b

fix resync commit msg

4bef7ca

add / fix comments

8aaf791

sleepyStick commented Jun 26, 2025

View reviewed changes

sleepyStick marked this pull request as ready for review June 26, 2025 17:04

NoahStapp requested changes Jun 26, 2025

View reviewed changes

sleepyStick and others added 2 commits June 26, 2025 13:56

check=True, catch error instead

413b463

Merge branch 'master' into PYTHON-5253

75ddd14

sleepyStick requested a review from NoahStapp June 27, 2025 16:09

NoahStapp requested changes Jun 27, 2025

View reviewed changes

multiple patch files

1a3d87f

sleepyStick requested a review from a team as a code owner June 27, 2025 19:00

update contributing to explain patch files

4f3cb61

sleepyStick requested a review from NoahStapp June 27, 2025 19:27

NoahStapp requested changes Jun 27, 2025

View reviewed changes

sleepyStick and others added 2 commits June 27, 2025 12:53

Update CONTRIBUTING.md

ec50aff

Co-authored-by: Noah Stapp <[email protected]>

explain what syncing is, add example, and modify spacing

579c194

sleepyStick requested a review from NoahStapp June 27, 2025 20:08

		@@ -0,0 +1,1327 @@
		diff --git a/test/load_balancer/cursors.json b/test/load_balancer/cursors.json

PYTHON-5253 Automated Spec Test Sync #2409

Are you sure you want to change the base?

PYTHON-5253 Automated Spec Test Sync #2409

Uh oh!

Conversation

sleepyStick commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sleepyStick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sleepyStick Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NoahStapp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NoahStapp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sleepyStick commented Jun 25, 2025 •

edited

Loading

sleepyStick Jun 26, 2025 •

edited

Loading