-
Notifications
You must be signed in to change notification settings - Fork 120
👔 Reset the cached pipeline state in the profile as well #1067
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Before this change, the pipeline reset script would change the `get_pipeline_state_db()` but not the cached pipeline start and end in the profile. Since we now use the profile to decide whether to run the pipeline, this led to issues where the pipeline for a user would not re-run even after the pipeline was reset. e-mission/e-mission-docs#1105 (comment) I wonder if we should use only the pipeline state to decide whether or not to run the pipeline, instead of the profile. It seems like restructuring to use a single source of truth would be good. Testing done: Ran it to reset our test user back to `2024-07-30`; verified that the value in the profile was reset
When we reset entries, we attempt to print the result of the database operations. However, in recent versions of pymongo, the results are objects, so we end up with results like this. ``` DEBUG:root:this is not a dry run, result of updating trip_segmentation stage in reset_pipeline_state = <pymongo.results.UpdateResult object at 0x7f1bb9b43850> DEBUG:root:this is not a dry run, result of updating all other stages in reset_pipeline_state = <pymongo.results.UpdateResult object at 0x7f1bb9b43af0> DEBUG:root:this is not a dry run, result of updating the profile in reset_pipeline_state = <pymongo.results.UpdateResult object at 0x7f1bb9b43460> ``` We change all the logs to print the `raw_result` instead, https://pymongo.readthedocs.io/en/stable/api/pymongo/results.html#pymongo.results.UpdateResult and add a missing log from 31239b7 (👔 Reset the cached pipeline state in the profile as well)
@JGreenlee for visibility Copilot suggests something like
Please make sure to use your best judgement in integrating it with the rest of the codebase; find the current test suite for pipeline resets; make sure that the checks are actually checking the new functionality here, etc. |
Added test in #1068 |
So that we can verify the results if we need to
Since it can happen if the trip has been labeled before it is processed (aka in draft mode)
Also made some additional logging changes to clean up this branch before I am done |
When we first started e-mission-common, @JGreenlee hosted it in his own account as an initial location. We then moved it to the "proper" location in the e-mission org, and `JGreenlee/e-mission-common` redirected to `e-mission/e-mission-common` for backwards compatibility. But then @JGreenlee created a new fork to submit e-mission/e-mission-common#15 so the redirect stopped. And the new fork did not have any releases, so we couldn't pull the library any more. Fortunately, this was caught by our automated tests :) All of this played out as expected, but it does provide a strong hint/motivation to switch the URL over, which I am doing here. Testing done: Before this, the additional logging commits in the PR (pushed after the failure) failed tests. After this, they passed.
introduces two new test cases in `TestPipelineReset.py` to ensure that the `pipeline_range` in user profiles is correctly reset during pipeline operations. The first test checks that the `end_ts` is updated after resetting the pipeline to a specific timestamp, while the second test verifies that both `start_ts` and `end_ts` are set to `None` when resetting to the start. These tests address previous issues where the profile was not updated correctly after pipeline resets
Before this change, the pipeline reset script would change the
get_pipeline_state_db()
but not the cached pipeline start and end in the profile.Since we now use the profile to decide whether to run the pipeline, this led to issues where the pipeline for a user would not re-run even after the pipeline was reset.
e-mission/e-mission-docs#1105 (comment)
I wonder if we should use only the pipeline state to decide whether or not to run the pipeline, instead of the profile. It seems like restructuring to use a single source of truth would be good.
Testing done:
Ran it to reset our test user back to
2024-07-30
; verified that the value in the profile was resetVerified that the pipeline was not skipped on the run after the reset