Clean up the TRB_label_assist code

[shankari]

This will:

• help us modify it and make it better if we want to experiment with different algorithms, including trajectory matching
• allow us to integrate into our production system
• make sure that it is reproducible before the related paper gets published

The code is in: https://github.com/e-mission/e-mission-eval-private-data/tree/master/TRB_label_assist
It works, but it requires us to use a custom fork of the e-mission-server repo to work

We should see the changes between the custom fork and the current e-mission-server master/gis branch
See if the changes are still necessary or whether there are implementations in e-mission-server that have superceded them
If they have not been superceded, we need to incorporate them in e-mission-server

So at the end, this should be reproducible against a core e-mission-server repo

Stretch goal: change this repo to work with docker containers that are built with the base e-mission-server container so that people don't have to install e-mission-server and set the EMISSION_SERVER_PATH and all the funky setup steps.

After we are done with this, we should integrate the random forest model + prediction into the e-mission server code for label assist.

• either replace the existing clustering approach OR
• better, keep both methods in the current plugin architecture and maybe even consider an ensemble down the road.

[shankari]

The current implementation of the mode building part of label assist code is at:
https://github.com/e-mission/e-mission-server/tree/master/emission/analysis/modelling/trip_model

There are previous implementations in https://github.com/e-mission/e-mission-server/tree/master/emission/analysis/modelling/tour_model and https://github.com/e-mission/e-mission-server/tree/master/emission/analysis/modelling/tour_model_first_only

• tour model was written by a summer intern and is very bad code
• tour_model_first_only was a last minute hack by me to salvage some of the interns code and get a feature working in production
• trip_model is a rewrite of tour_model_first_only to be more modular. This is what is in production right now.

The custom branch uses some code from tour_model or tour_model_first_only since the TRB_label_assist work was being done in parallel with the trip_model rewrite.

So we should:

• see where that functionality is in trip_model
• if it exists, replace its usage in TRB_label_assist with the implementation from trip_model
• if it doesn't exist, we add it to trip_model and then replace its usage in TRB_label_assist

Eventually we should take the new and improved model(s) from TRB label assist and put the into trip_model

For a high level summary of what the label assist model does, please see this poster:
https://www.osti.gov/biblio/1922617

[humbleOldSage]

Two files, Clustering.py and models.py needs to be updated for us to port the entire code to just our main repo.
I had two questions:

Q1 Among the vast number of models in the models.py file, we are currently using ClusterExtrapolationClassifier, ForestClassifier and NaiveBinningClassifier in our performance_eval.py file. Out of these three , only NaiveBinningClaasifier has dependence on the code from tour_model. But there's one another model ( which we are currently not using) RefactoredNaiveCluster which is dependent on tour_model. Should we, for now, let model be dependent on tour_model or port this to trip model ?

Q2 Is there any way I can check the the working of trip_model by running it ? Is there any module that's using it that I can/should run ??

NOTE : regenerate_classification_performance_results.py. is currently standalone and is not being used by any of the notebooks.

[shankari]

@humbleOldSage I just want to clarify that, while we will eventually port the code around the new models in the e-mission-eval-private-data repo to the e-mission-server repo, the first goal is to port the code required by the special branch of e-mission-server that this repo to the master branch of e-mission-server.

It works, but it requires us to use a custom fork of the e-mission-server repo to work

We should see the changes between the custom fork and the current e-mission-server master/gis branch
See if the changes are still necessary or whether there are implementations in e-mission-server that have superceded them

clustering and models and files in e-mission-eval-private-data not the server.

[shankari]

I apologize for not reading your comment more carefully. You are correct that if there is a mode (e.g. RefactoredNaiveCluster) which is dependent on the tour model, we could choose to remove it instead of migrating it.

So I would suggest:

• verify that RefactoredNaiveCluster is indeed not used (put results of grep)
• migrate the models that are used and see if the changes cover RefactoredNaiveCluster as well.
  • If so, we might as well migrate it (since it will only involve changing the calls)
  • If not, we should remove it

Let's not keep unused, bitrotted code around

[shankari]

Q2 Is there any way I can check the the working of trip_model by running it ? Is there any module that's using it that I can/should run ??

Trip model has unit tests, and a script to launch it against a database.
If you run it against the full CEO dataset, I would suggest launching over the weekend so that you can still use your computer 😄

$ grep -rl trip_model emission | grep -v __pycache__
...
emission/analysis/classification/inference/labels/inferrers.py
emission/analysis/modelling/trip_model/model_type.py
emission/analysis/modelling/trip_model/config.py
emission/analysis/modelling/trip_model/model_storage.py
emission/analysis/modelling/trip_model/run_model.py
emission/analysis/modelling/trip_model/greedy_similarity_binning.py
emission/tests/modellingTests/TestBackwardsCompat.py
emission/tests/modellingTests/TestRunGreedyIncrementalModel.py
emission/tests/modellingTests/TestRunGreedyModel.py
emission/tests/modellingTests/modellingTestAssets.py
emission/tests/modellingTests/TestGreedySimilarityBinning.py
...

NOTE : regenerate_classification_performance_results.py is currently standalone and is not being used by any of the notebooks.

That is by design - if you check the history:

Note that these results can take very long (> 2 days) to regenerate
Running them from a notebook will either not print logs, or will print so many
logs that the notebook buffers will be overwhelmed

Moving the computation code out to a separate script allows us to more easily
redirect the output to a file and track the progress of the execution

[humbleOldSage]

I missed an indirect dependence of RefactoredNaiveCluster , so we'll have to work on it as well. Apologies. The last two results of grep are calls from one of the models we are using, ClusterExtrapolationClassifier .

$ grep -r 'RefactoredNaiveCluster' TRB_label_assist/models.py
TRB_label_assist/models.py:class RefactoredNaiveCluster(Cluster):
TRB_label_assist/models.py:        logging.info("PERF: Initializing RefactoredNaiveCluster")
TRB_label_assist/models.py:        logging.info("PERF: Fitting RefactoredNaiveCluster with size %s" % len(train_df))
TRB_label_assist/models.py:        logging.info("PERF: Predicting RefactoredNaiveCluster for %s" % len(test_df))
TRB_label_assist/models.py:                logging.info("PERF: RefactoredNaiveCluster Working on trip %s/%s" % (idx, len(self.test_df)))
TRB_label_assist/models.py:            self.end_cluster_model = RefactoredNaiveCluster(loc_type='end',
TRB_label_assist/models.py:                self.start_cluster_model = RefactoredNaiveCluster(

[humbleOldSage]

I have been working with the sample data that the script generates for testing for now. It should be enough for my purpose. If it doesn't I'll run it on full data set over the weekend. Also, won't need to run all of them, just need 1 test file, the TestGreedySimilarityBinning.py one.

[shankari]

It should be enough for my purpose.

Curious - what are you running the existing tests for? Is it to test the new functionality that you are moving into trip_model? If so, you should really write a new unit test (which can use similar sample data) instead of ad-hoc testing.

[humbleOldSage]

While I move the functionalities, I am just using them to verify my understanding of the code-flow and I/O to trip_model modules. That's why I felt a small randomly generated sample should serve the purpose. Once done, I do plan on writing new unit tests and may even test on the full CEO dataset.

[humbleOldSage]

The dependence of the 2 files as mentioned previously(clustering.py and models.py)are on the following 2 modules in the custom branch:

1. bin_helper module in theSimilarity Class from similarity.py file.
2. save_models and create_user_input_map modules from the build_save_model.py file.
3. fit module again from the Similarity Class from similarity.py file.

[humbleOldSage]

For i. , I probed the trip_model folder in the main branch and we surely have this functionality already implemented in the main branch.

I can say this because to understand the functionalities and flow of data on the custom branch side, I ran the dependent notebooks on the CEO dataset - i.e. clustering_examples.ipynb. Starting with this file, I noticed that it imports mapping.py, which, in turn, imports clustering.py, which in turn imports import emission.analysis.modelling.tour_model_extended.similarity as eamts

I then looked at the list of diffs in the extended branch
e-mission/e-mission-server@master...hlu109:e-mission-server:eval-private-data-compatibility
pulled the changes, and compared them using diff. The result is attached (as a gzip file).

This notebook only uses eamts.Similarity, which clusters based on O-D of the trips using the ecc.calDistance.

To match this understanding on the main branch side, I ran the existing unit test specifically for the modeling modules using the below command.

PYTHONPATH=. python -m unittest emission/tests/modellingTests/TestGreedySimilarityBinning.py

The trip_model rewrite and the eamts.Similarity were passing origin/destination latitude and longitudes and relied on od_similarity.py file for similarity computation,specifically the ecc.calDistance function. This helped me reach the conclusion that the functionality and the end result of these two had an overlap.

However, both these functions are working on different data-forms, i.e., custom branch uses data in data-frame format while processing whereas the main branch uses them in Entry type wrapper. My first thought was to convert the data frame to Entry type data and use it . But on Further searching, I could figure that the data was being converted from Entry type to data_frame at some-point before similarity computations and then visualizing them. Again converting it back would be unnecessary processing .

So we just need to change the way data is being passed to functions, without breaking any other functionalities.

We have not yet evaluated what are the other changes between tour_model_extended and tour_model, where they are used, and how they should be migrated to trip_model. After finishing this migration (ETA EOD tomorrow), I will submit a draft PR and start on the other two.

All-in-all, we won't have to do new implementations here, just a few changes should do.

[humbleOldSage]

To begin with the changes for (i) at #35 (comment) , we start with the clustering.py file . I can see that the end result of the "naive" algorithm under the add_loc_clusters function is a column being added to the data frame having group labels ( numeric value ), at line that reads:

loc_df.loc[:, f"{loc_type}_{alg}_clusters_{r}_m"] = label

where

• {loc_type} -> start or end or trip
• {alg} -> DBSCAN or naive or OPTIC or fuzzy or mean_shift.
• {r } -> stands for the grouping radius, takes value 50,100 and 150 currently .

These group labels are generated using the bin_helper function present in the eamts.Similarity class. bin_helper calls match which calls distance_helper which calls within_radius which calls ecc.calDistance to calculate the distance between two points.

These generated labels are stored in two places:

1. Inside the eamts.Similarity class they are stores in self.start_bins or self.end_bins or self.trip_bins depending on value of loc_type variable.

2. In the dataframe, the same results is stored in one of three new column amongst start_bin/ end_bin / trip_bin and the other two are filled with NaNs.

The additional column (columns if we pass multiple r values), that I referred to in the beginning of this comment, copies labels from (i) (above) and appends into the data frame.

IF we are able to generate these labels from the existing trip_model in the main branch, we'll be able to remove our dependence on the custom branch.

[shankari]

IF we are able to generate these labels from the existing trip_model in the main branch, we'll be able to remove our dependence on the custom branch.

And what would it take to generate these labels from the existing trip_model in the main branch?