Skip to content

Commit 3973abe

Browse files
trevor-wushankari
authored andcommitted
Gt server6 (#685)
* update to current greentrip branch (#2) * Deleted the changes.patch and applied the patch * Made changes based on first pull-request comments and added suggestion_sys to API file * Converted the original functions to use Nominatim, tested on data and works Kept the old functions just in case the something horribly wrong happens to the current functions Changed the API call to a call to Nominatim * Deleted patch * Suggestion System Debugged (#619) * Made changes to use nominatim.py reverse geocode for any lookup regarding nominatim. Suggestions should pop up now, debugged it and should work. * Removed patch * Resolved comments on pull request for nominatim json, googlemaps json is still being used so will need to restructure code * Function using only Trip ID completed and should be working (#620) * Made changes to use nominatim.py reverse geocode for any lookup regarding nominatim. Suggestions should pop up now, debugged it and should work. * Removed patch * Resolved comments on pull request for nominatim json, googlemaps json is still being used so will need to restructure code * Debugged and finished the single trip function * Got rid of the redundant googlemaps json calls and fixed the functions so that it works for the except case (that was why it was bugging on other trips) * Responded to comments for uuid and uuid first before tripid * Got rid of the edb lines * Got rid of the import statement for edb * Added more comments and addressed the pull-request comments in the code Also changed the indexing to be directly indexing for the trip_id, not sure if it got rid of taking extra space in memory by creating a copy * Finished changing location to lat lon for inputs and switched to using abstract time series function * Fixed all the nits * Fixed suggestion bugs involving cities, towns, and zipcodes and edited the yelpfusion.json.sample to provide information * made changes to json sample * timeseries * Fixed the routing for webapp and added more fields to the return statements for the phone * Fixed nit from routing "overall" to "recent" * Fixed single suggestion and recent suggestion, same as the server version * Changed to tripidstr instead of confusing variable names * Fixed tripidstr and tripid discreption * Improved performance of the main suggestion function, supporting the suggestions page, added trip id for Samantha to pull from the trip diary, fixed Sam and Bill's trips * adding greentrip site * created new branch and applied changes in regards to print statements * got rid of double quotes on trip_id and resolved the trip_id = None * changed tripids to what they were originally and fixed the nominatim files * added link to previous work * fixed zipcode api erroring * moving around files to fit directory structure * added client setup with qr code * fixing client html was wonky * added testing qr code video * added shankari's changes * adding new client change at bottom * removing video * Updated the suggestion functions to output the alias for Trevor's function (#649) * made changes to the yelp-function, so trevor can retrieve the alias * made changes to the single-trip to reflect the new yelp function changes * tested * reverted the timeseries sample * website debugging, alternating javascript, learn more page changes (#650) * now automatic link change for half control and half experimental * alternating qr code image now in place * minor fixes with shankari image and learn_more link * secure jQuery link * learn more page now has more limited information * local changes complete and tested * Committing last set of website changes for Sam This is the final clean version of #653 and #651 * added parameters to make suggestions less wordy * New client setup page. Added new image for page in www/img and also made client page more readable * Removed the specs * Add the additional keys required to the yelpfusion.json file * Add the other fields required for the googlemaps json sample * Merge two almost-identical copies of suggestion system generation + make it easier to test In order to generate the merged version, I extracted the two separate versions and used vimdiff to unify the changes. Even after unifying the largely cosmetic changes, I found two fixes to the recent suggestion method that had not been ported to the trip suggestion method. After merging those, the only differences were to the inputs and the outputs, which made the refactoring straightforward. Testing done: - Pick a location in a residential area (opposite Lincoln Elementary school in Daly City) - Pick a location with lots of establishments (Cinema complex opposite Daly City BART station) - Call the new method to generate a suggestion ``` In [1]: import emission.core.wrapper.suggestion_sys as sugg Connecting to database URL localhost In [2]: import geojson as gj In [3]: start_loc = gj.Point((-122.4831506, 37.6764811)) In [4]: end_loc = gj.Point((-122.4706021, 37.7019950)) In [5]: sugg.calculate_yelp_server_suggestion_for_locations(start_loc, end_loc, 3000) Out[5]: {'businessid': None, 'message': 'Your endpoint has either been a non-serviceable category or a closeby option.', 'method': 'public transportation', 'rating': None} ``` - Load test timeline data - Run pipeline - Call method to generate most recent suggestion ``` In [6]: from uuid import UUID In [7]: test_uuid = UUID("29adc2d9-e157-4fe1-a700-4075619f1e30") In [8]: sugg.calculate_yelp_server_suggestion_nominatim(test_uuid) Out[8]: {'businessid': None, 'message': 'Sorry, unable to retrieve datapoint because datapoint is a house or datapoint does not belong in service categories', 'method': 'bike', 'question': None, 'rating': None, 'suggested_loc': None, 'tripid': ObjectId('5cba04fcf6858fcbf006239f')} ``` - Call method to generate suggestion for a particular trip ``` In [9]: import bson.objectid as boi In [11]: sugg.calculate_yelp_server_suggestion_singletrip_nominatim(test_uuid, boi.Objec ...: tId("5cba04fcf6858fcbf006239f")) Out[11]: {'businessid': None, 'message': 'Sorry, unable to retrieve datapoint because datapoint is a house or datapoint does not belong in service categories', 'method': 'bike', 'question': None, 'rating': None, 'suggested_loc': None} ``` None of these generated any relevant suggestions but also did not fail * Refactor the destination business lookup to avoid repetition For us to make relevant suggestions, we really need to be able to figure out what the destination was. The current code munges the destination lookup and the category lookup, which makes it hard to identify whether the destination lookup was incorrect or only the category. There are also other questions - do we even need to try to look up the destination business, or is it enough to get a rough idea of the address and do an address search... All of these will be much easier to answer if the code is structured in a way that makes sense and the individual components can be tested separately. This change accomplishes that restructuring. It still does not address the copy-pasting in the try/catch blocks in the google code, but I don't want to make this harder to review than it already is Testing done: - Ran the two new functions + overarching suggestion call manually; there are no errors. - Note that the nominatim call failed because there was no 'city' in the response - it looks like the fixes to look for the town instead in `calculate_yelp_server_suggestion_for_locations` were not ported here. We fall through to the google API, which also fails because of the improperly formatted address. But this predates my change. ``` In [1]: import emission.core.wrapper.suggestion_sys as sugg Connecting to database URL localhost In [2]: sugg.find_destination_business("37.7019950", "-122.4706021") Out[2]: (None, '1901G Junipero Serra Blvd', 'San Mateo County', True) In [3]: import logging In [4]: logging.getLogger().setLevel(logging.DEBUG) In [5]: sugg.find_destination_business("37.7019950", "-122.4706021") DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/geocode/json?latlng=37.7019950,-122.4706021& HTTP/1.1" 200 1450 DEBUG:root:Components from address lookup = [{'long_name': '1901G', 'short_name': '1901G', 'types': ['street_number']}, {'long_name': 'Junipero Serra Boulevard', 'short_name': 'Junipero Serra Blvd', 'types': ['route']}, {'long_name': 'Daly City', 'short_name': 'Daly City', 'types': ['locality', 'political']}, {'long_name': 'San Mateo County', 'short_name': 'San Mateo County', 'types': ['administrative_area_level_2', 'political']}, {'long_name': 'California', 'short_name': 'CA', 'types': ['administrative_area_level_1', 'political']}, {'long_name': 'United States', 'short_name': 'US', 'types': ['country', 'political']}, {'long_name': '94014', 'short_name': '94014', 'types': ['postal_code']}] DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/place/nearbysearch/json?location=37.7019950,-122.4706021&radius=10& HTTP/1.1" 200 1231 DEBUG:root:For amenity Daly City, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business Daly City DEBUG:root:For amenity Tomo Sushi & Teriyaki, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business 1901 Junipero Serra Boulevard # G, Daly City DEBUG:root:After checking = 1901G Junipero Serra Boulevard, San Mateo County, got business tuple (False, '') DEBUG:root:Nominatim failed, Google found destination business (None, '1901G Junipero Serra Blvd', 'San Mateo County', True) Out[5]: (None, '1901G Junipero Serra Blvd', 'San Mateo County', True) ``` ``` In [4]: sugg.category_of_business_nominatim("37.7019950", "-122.4706021") DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/geocode/json?latlng=37.7019950,-122.4706021& HTTP/1.1" 200 1450 DEBUG:root:Components from address lookup = [{'long_name': '1901G', 'short_name': '1901G', 'types': ['street_number']}, {'long_name': 'Junipero Serra Boulevard', 'short_name': 'Junipero Serra Blvd', 'types': ['route']}, {'long_name': 'Daly City', 'short_name': 'Daly City', 'types': ['locality', 'political']}, {'long_name': 'San Mateo County', 'short_name': 'San Mateo County', 'types': ['administrative_area_level_2', 'political']}, {'long_name': 'California', 'short_name': 'CA', 'types': ['administrative_area_level_1', 'political']}, {'long_name': 'United States', 'short_name': 'US', 'types': ['country', 'political']}, {'long_name': '94014', 'short_name': '94014', 'types': ['postal_code']}] DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/place/nearbysearch/json?location=37.7019950,-122.4706021&radius=10& HTTP/1.1" 200 1228 DEBUG:root:For amenity Daly City, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business Daly City DEBUG:root:For amenity Tomo Sushi & Teriyaki, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business 1901 Junipero Serra Boulevard # G, Daly City DEBUG:root:After checking = 1901G Junipero Serra Boulevard, San Mateo County, got business tuple (False, '') DEBUG:root:Nominatim failed, Google found destination business (None, '1901G Junipero Serra Blvd', 'San Mateo County', True) DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?location=1901G%2BJunipero%2BSerra%2BBlvd HTTP/1.1" 200 None Out[4]: ['grocery', 'korean', 'bbq'] ``` ``` In [5]: import geojson as gj In [6]: start_loc = gj.Point((-122.4831506, 37.6764811)) ...: In [7]: end_loc = gj.Point((-122.4706021, 37.7019950)) In [8]: sugg.calculate_yelp_server_suggestion_for_locations(start_loc, end_loc, 3000) DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/geocode/json?latlng=37.701995,-122.4706021& HTTP/1.1" 200 1450 DEBUG:root:Components from address lookup = [{'long_name': '1901G', 'short_name': '1901G', 'types': ['street_number']}, {'long_name': 'Junipero Serra Boulevard', 'short_name': 'Junipero Serra Blvd', 'types': ['route']}, {'long_name': 'Daly City', 'short_name': 'Daly City', 'types': ['locality', 'political']}, {'long_name': 'San Mateo County', 'short_name': 'San Mateo County', 'types': ['administrative_area_level_2', 'political']}, {'long_name': 'California', 'short_name': 'CA', 'types': ['administrative_area_level_1', 'political']}, {'long_name': 'United States', 'short_name': 'US', 'types': ['country', 'political']}, {'long_name': '94014', 'short_name': '94014', 'types': ['postal_code']}] DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): maps.googleapis.com DEBUG:requests.packages.urllib3.connectionpool:https://maps.googleapis.com:443 "GET /maps/api/place/nearbysearch/json?location=37.701995,-122.4706021&radius=10& HTTP/1.1" 200 1233 DEBUG:root:For amenity Daly City, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business Daly City DEBUG:root:For amenity Tomo Sushi & Teriyaki, comparing address 1901G Junipero Serra Boulevard, San Mateo County with nearby business 1901 Junipero Serra Boulevard # G, Daly City DEBUG:root:After checking = 1901G Junipero Serra Boulevard, San Mateo County, got business tuple (False, '') DEBUG:root:Nominatim failed, Google found destination business (None, '1901G Junipero Serra Blvd', 'San Mateo County', True) DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?location=1901G%2BJunipero%2BSerra%2BBlvd HTTP/1.1" 200 None DEBUG:root:parsed_response = {'place_id': '269467', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '8933300', 'lat': '37.6739981', 'lon': '-122.4833696', 'display_name': 'Lakeshire Drive, Daly City, San Mateo County, 94015, United States of America', 'address': {'road': 'Lakeshire Drive', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94015', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '525262', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '183198747', 'lat': '37.7008995', 'lon': '-122.4707415', 'display_name': 'Junipero Serra Boulevard, Westlake, Daly City, San Mateo County, 94014, United States of America', 'address': {'road': 'Junipero Serra Boulevard', 'neighbourhood': 'Westlake', 'town': 'Daly City', 'county': 'San Mateo County', 'postcode': '94014', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?location=Junipero%2BSerra%2BBoulevard%2C%2BWestlake%2C%2BDaly%2BCity%2C%2BSan%2BMateo%2BCounty%2C%2B94014%2C%2BUnited%2BStates%2Bof%2BAmerica HTTP/1.1" 200 None DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?term=grocery&location=Daly%2BCity&limit=3 HTTP/1.1" 200 None DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?term=korean&location=Daly%2BCity&limit=3 HTTP/1.1" 200 None DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?term=bbq&location=Daly%2BCity&limit=3 HTTP/1.1" 200 None Out[8]: {'businessid': None, 'message': 'Your endpoint has either been a non-serviceable category or a closeby option.', 'method': 'public transportation', 'rating': None} ``` * clearer for pc vs mobile version and fix button link * Add integration tests to create an artificial dataset for the suggestion system components - README with instructions on how to generate the test cases - test harness with wrappers for the three main algorithms - dataset files with one sample entry, corresponding to a trip from a suburban residential area to a suburban big box store, where there is a closer store available Note that the destination is not currently detected correctly so the accuracy is 0% Testing done Arguments --------- ``` $ ./e-mission-py.bash -d emission/integrationTests/suggestionsys/algorithm_test_harness.py Connecting to database URL localhost usage: algorithm_test_harness.py [-h] [-f INFILE] {find_destination_business,category_of_business_nominatim,calculate_yelp_server_suggestion_for_locations} algorithm_test_harness.py: error: the following arguments are required: algorithm ``` `find_destination_business` --------- ``` (emission) C02KT61MFFT0:e-mission-server shankari$ ./e-mission-py.bash -d emission/integrationTests/suggestionsys/algorithm_test_harness.py find_destination_business Connecting to database URL localhost INFO:root:Mapped algorithm find_destination_business -> <function test_find_destination_business at 0x109262bf8> DEBUG:root:-----Mountain View Kohls------ DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:Nominatim found destination business ('Pacchetti Way', 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'Mountain View', False) DEBUG:root:Test Mountain View Kohls failed, output = ['Pacchetti Way', 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'Mountain View', False], expected ["Kohl's", 'Showers Drive', 'Mountain View', True] DEBUG:root:After instance Mountain View Kohls, successfulTests = 0, failedTests = 1 INFO:root:Test complete, overall accuracy = 0.0 ``` `category_of_business_nominatim` --------- ``` $ ./e-mission-py.bash -d emission/integrationTests/suggestionsys/algorithm_test_harness.py category_of_business_nominatim Connecting to database URL localhost INFO:root:Mapped algorithm category_of_business_nominatim -> <function test_category_of_business_nominatim at 0x1135ec8c8> DEBUG:root:-----Mountain View Kohls------ DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:Nominatim found destination business ('Pacchetti Way', 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'Mountain View', False) DEBUG:root:Test Mountain View Kohls failed, output = [], expected ['deptstores', 'menscloth', 'womenscloth'] DEBUG:root:After instance Mountain View Kohls, successfulTests = 0, failedTests = 1 INFO:root:Test complete, overall accuracy = 0.0 ``` `calculate_yelp_server_suggestion_for_locations` ....................... ``` $ ./e-mission-py.bash -d emission/integrationTests/suggestionsys/algorithm_test_harness.py calculate_yelp_server_suggestion_for_locations Connecting to database URL localhost INFO:root:Mapped algorithm calculate_yelp_server_suggestion_for_locations -> <function test_calculate_yelp_server_suggestion_for_locations at 0x11805f950> DEBUG:root:-----Sunnyvale -> Mountain View Kohls------ DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.mapquestapi.com DEBUG:requests.packages.urllib3.connectionpool:http://www.mapquestapi.com:80 "GET /directions/v2/route?key=AuwuGlPC5f3Ru7PGahKAtGcs4WdvARem&from=37.3635335,-122.0154871&to=37.40355,-122.10783 HTTP/1.1" 200 None DEBUG:root:distance in meters = 11695.106466185258 DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:Nominatim found destination business ('Pacchetti Way', 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'Mountain View', False) DEBUG:root:parsed_response = {'place_id': '268772', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '8932529', 'lat': '37.3638317', 'lon': '-122.0170368', 'display_name': 'Gladiola Drive, Sunnyvale, Santa Clara County, 94086, United States of America', 'address': {'road': 'Gladiola Drive', 'city': 'Sunnyvale', 'county': 'Santa Clara County', 'postcode': '94086', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/Pacchetti-Way-Mountain%20View HTTP/1.1" 400 186 DEBUG:root:parsed_response = {'place_id': '201857', 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://www.openstreetmap.org/copyright', 'osm_type': 'way', 'osm_id': '5066896', 'lat': '37.4030262', 'lon': '-122.1102601', 'display_name': 'Pacchetti Way, Castro City, Mountain View, Santa Clara County, 94023, United States of America', 'address': {'road': 'Pacchetti Way', 'neighbourhood': 'Castro City', 'city': 'Mountain View', 'county': 'Santa Clara County', 'postcode': '94023', 'country': 'United States of America', 'country_code': 'us'}} DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.yelp.com DEBUG:requests.packages.urllib3.connectionpool:https://api.yelp.com:443 "GET /v3/businesses/search?location=Pacchetti%2BWay%2C%2BCastro%2BCity%2C%2BMountain%2BView%2C%2BSanta%2BClara%2BCounty%2C%2B94023%2C%2BUnited%2BStates%2Bof%2BAmerica HTTP/1.1" 200 None DEBUG:root:Test Sunnyvale -> Mountain View Kohls failed, output = None, expected target-sunnyvale-2 DEBUG:root:After instance Sunnyvale -> Mountain View Kohls, successfulTests = 0, failedTests = 1 INFO:root:Test complete, overall accuracy = 0.0 ``` * Added some more documentation on how to get categories To make people's lives easier * Clarifying configuration required * merged * remove ipynb * added new dataset to test only restaurants * Revert "remove ipynb" This reverts commit d31d8ef. * Revert "Revert "remove ipynb"" This reverts commit 70add76. * Revert "added new dataset to test only restaurants" This reverts commit ef365d5. * Revert "added new dataset to test only restaurants" This reverts commit ef365d5. * new dataset to test accuracy for certain establishments * small edit to increase accuracy * update * Improvements to Yelp API and dataset
1 parent 5eeac04 commit 3973abe

File tree

4 files changed

+99
-66
lines changed

4 files changed

+99
-66
lines changed

emission/core/wrapper/suggestion_sys.py

Lines changed: 67 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,27 @@ def search(api_key, term, location):
105105
}
106106
return request(API_HOST, SEARCH_PATH, api_key, url_params=url_params)
107107

108+
def lat_lon_search(api_key, lat, lon, radius):
109+
"""Query Search API using latitude and longitude.
110+
Args:
111+
lat (float) : latitude
112+
lon (float) : longitude
113+
radius (int) : radius of search in meters
114+
Returns:
115+
dict: The JSON response form the request.
116+
"""
117+
url_params = {
118+
'latitude': lat,
119+
'longitude': lon,
120+
'radius' : radius,
121+
'limit': SEARCH_LIMIT,
122+
'sort_by': 'distance',
123+
'categories' : 'food,restaurants,shopping,hotels,beautysvc,auto,education,collegeuniv,financialservices,publicservicesgovt'
124+
}
125+
return request(API_HOST, SEARCH_PATH, api_key, url_params=url_params)
126+
108127
"""
109-
YELP API: Function to retrieve all reviews related to the business.
128+
YELP API: Function to retrieve all reviews related to the business.
110129
"""
111130
def business_reviews(api_key, business_id):
112131
business_path = BUSINESS_PATH + business_id
@@ -129,7 +148,7 @@ def get_business_id(api_key, lat, lon):
129148
return request(API_HOST, SEARCH_PATH, api_key, url_params=url_params)
130149

131150
"""
132-
NOMINATIM API: Creates a Nominatim API Call, returns address in string form and dictionary form separated by streetname,
151+
NOMINATIM API: Creates a Nominatim API Call, returns address in string form and dictionary form separated by streetname,
133152
road, neighborhood, etc
134153
"""
135154
def return_address_from_location_nominatim(lat, lon):
@@ -164,37 +183,51 @@ def is_service_nominatim(business):
164183
return True
165184

166185
'''
167-
NOMINATIM VERS: Function that RETURNS a list of categories that the business falls into
186+
NOMINATIM VERS: Function that RETURNS a list of categories that the business falls into
168187
169-
Using the Google reverse lookup in the except clause, in case Nominatim's results are too vague.
170-
Will first try Nominatim's reverse lookup, but if Nominatim returns a broad "address"
171-
of the street and the city, without a full address with a specific location
188+
Using the Google reverse lookup in the except clause, in case Nominatim's results are too vague.
189+
Will first try Nominatim's reverse lookup, but if Nominatim returns a broad "address"
190+
of the street and the city, without a full address with a specific location
172191
Such as Piedmont Ave, Berkeley, CA
173192
174-
Then the function will enter the Google reverse lookup and choose a business that is closest to
193+
Then the function will enter the Google reverse lookup and choose a business that is closest to
175194
latitude and longitude given
176195
'''
177196

197+
178198
### BEGIN: Pulled out candidate functions so that we can evaluate individual accuracies
179199
def find_destination_business_google(lat, lon):
180200
return return_address_from_google_nomfile(lat, lon)
181201

182202
def find_destination_business_yelp(lat, lon):
183-
return (None, None, None, None)
203+
yelp_from_lat_lon = lat_lon_search(YELP_API_KEY, lat, lon, 250)
204+
if yelp_from_lat_lon == {}:
205+
return (None, None, None, False)
206+
businesses = yelp_from_lat_lon['businesses']
207+
if businesses == []:
208+
return find_destination_business(lat, lon)
209+
business_name = businesses[0]['name']
210+
address = businesses[0]['location']['address1']
211+
city = businesses[0]['location']['city']
212+
#If there is no commercial establishment in a 50 meter (1/2 block) radius of coordinate
213+
#It is safe to assume the area is not a commercial establishment
214+
location_is_service = True
215+
print((business_name, address, city, location_is_service))
216+
return (business_name, address, city, location_is_service)
184217

185218
def find_destination_business_nominatim(lat, lon):
186219
string_address, address_dict = return_address_from_location_nominatim(lat, lon)
187220
business_key = list(address_dict.keys())[0]
188221
business_name = address_dict[business_key]
189222
try:
190223
city = address_dict['city']
191-
except:
224+
except:
192225
try:
193226
city = address_dict["town"]
194227
except:
195228
try:
196229
zipcode = address_dict["postcode"]
197-
city = zipcode_to_city(zipcode)
230+
city = zipcode_to_city(zipcode)
198231
except:
199232
city = ''
200233

@@ -222,6 +255,7 @@ def find_destination_business(lat, lon):
222255
% str(return_tuple))
223256
return return_tuple
224257

258+
225259
### BEGIN: Pulled out candidate functions so that we can evaluate individual accuracies
226260
def category_of_business_awesome(lat, lon):
227261
return []
@@ -313,8 +347,8 @@ def geojson_to_lat_lon_separated(geojson):
313347
return lat, lon
314348

315349
'''
316-
REWRITE def check_mode_from_trip(cleaned_trip, cleaned_sections, section_counter, trip_counter):
317-
Mode number correspondence:
350+
REWRITE def check_mode_from_trip(cleaned_trip, cleaned_sections, section_counter, trip_counter):
351+
Mode number correspondence:
318352
0: "IN_VEHICLE"
319353
1: "BIKING"
320354
2: "ON_FOOT"
@@ -364,7 +398,7 @@ def dummy_starter_suggestion(uuid):
364398
return modes_from_trips
365399

366400
'''
367-
NOMINATIM VERSION: Function to find the review of the original location of the end point of a trip
401+
NOMINATIM VERSION: Function to find the review of the original location of the end point of a trip
368402
'''
369403
def review_start_loc_nominatim(lat, lon):
370404
try:
@@ -391,35 +425,35 @@ def review_start_loc_nominatim(lat, lon):
391425
392426
As nominatim sometimes is unable to provide a specific location with the city and instead returns
393427
a postcode (zipcode) and the country name. For the suggestions that we built, the suggestions
394-
require which city (city name) it is in order to look for other similar categoried services
395-
in the area. Thus, this function takes in the INPUT of a zipcode, and RETURNS the name of the city.
428+
require which city (city name) it is in order to look for other similar categoried services
429+
in the area. Thus, this function takes in the INPUT of a zipcode, and RETURNS the name of the city.
396430
397431
'''
398432
def zipcode_retrieval(zipcode):
399-
400-
# Use this API key first.
433+
434+
# Use this API key first.
401435
url = ZIP_HOST_URL + ZIPCODE_API_KEY + ZIP_FORMAT + zipcode + ZIP_DEGREE
402436
response = requests.request('GET', url=url)
403437
results = response.json()
404438

405439
if "error_code" in results:
406-
# In case the first API key runs out of requests per hour.
440+
# In case the first API key runs out of requests per hour.
407441
url = ZIP_HOST_URL + BACKUP_ZIP_KEY + ZIP_FORMAT + zipcode + ZIP_DEGREE
408442
response = requests.request('GET', url=url)
409443
return response.json()
410444
else:
411445
return results
412446

413-
414-
447+
448+
415449
def zipcode_to_city(zipcode):
416450
response = zipcode_retrieval(zipcode)
417451
return response['city']
418452
'''
419453
NOMINATIM
420454
In progress-nominatim yelp server suggestion function, first just trying to make end-to-end work before robustifying this function.
421455
422-
Mode number correspondence:
456+
Mode number correspondence:
423457
0: "IN_VEHICLE"
424458
1: "BIKING"
425459
2: "ON_FOOT"
@@ -444,7 +478,7 @@ def calculate_yelp_server_suggestion_singletrip_nominatim(uuid, tripidstr):
444478
start_location = cleaned_trips.data.start_loc
445479
end_location = cleaned_trips.data.end_loc
446480
'''
447-
Distance in miles because the current calculated distances is through MapQuest which uses miles,
481+
Distance in miles because the current calculated distances is through MapQuest which uses miles,
448482
still working on changing those functions, because haven't found any functions through nominatim
449483
that calculates distance between points.
450484
'''
@@ -463,7 +497,7 @@ def calculate_yelp_server_suggestion_for_locations(start_location, end_location,
463497

464498
begin_string_address, begin_address_dict = return_address_from_location_nominatim(start_lat, start_lon)
465499
end_string_address, end_address_dict = return_address_from_location_nominatim(end_lat, end_lon)
466-
try:
500+
try:
467501
city = end_address_dict["city"]
468502
except:
469503
try:
@@ -491,12 +525,12 @@ def calculate_yelp_server_suggestion_for_locations(start_location, end_location,
491525
if q['rating'] >= location_review:
492526
#'Coordinates' come out as two elements, latitude and longitude
493527
ratings_bus[q['name']] = (q['rating'], q['alias'])
494-
obtained = q['location']['display_address'][0] + q['location']['display_address'][1]
528+
obtained = q['location']['display_address'][0] + q['location']['display_address'][1]
495529
obtained.replace(' ', '+')
496530
business_locations[q['name']] = obtained
497531
else:
498532
return {'message' : error_message_categor, 'question': None, 'suggested_loc': None, 'method': 'bike', 'rating': None, 'businessid': None}
499-
except:
533+
except:
500534
return {'message' : error_message_categor, 'question': None, 'suggested_loc': None, 'method': 'bike', 'rating': None, 'businessid': None}
501535

502536
#THIS PART WILL BE FIXED ACCODRING TO NOMINATIM AND GET RID OF MAPQUEST (find some other way to calculate distance)
@@ -509,8 +543,8 @@ def calculate_yelp_server_suggestion_for_locations(start_location, end_location,
509543
if calculate_distance < distance_in_miles and calculate_distance < 5 and calculate_distance >= 1:
510544
try:
511545
question = "How about this location?"
512-
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
513-
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
546+
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
547+
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
514548
rating_mess = "Rating of " + str(ratings_bus[a][0])
515549
#Not sure to include the amount of carbon saved
516550
#Still looking to see what to return with this message, because currently my latitude and longitudes are stacked together in one string
@@ -519,20 +553,20 @@ def calculate_yelp_server_suggestion_for_locations(start_location, end_location,
519553
except ValueError as e:
520554
continue
521555
elif calculate_distance < distance_in_miles and calculate_distance < 1:
522-
try:
556+
try:
523557
question = "How about this location?"
524-
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
525-
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
558+
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
559+
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
526560
rating_mess = "Rating of " + str(ratings_bus[a][0])
527561
# insert_into_db(tripDict, i, yelp_suggestion_trips, uuid)
528562
return {'message' : new_message, 'question': question, 'suggested_loc': suggested_loc, 'method': 'walk', 'rating': str(ratings_bus[a][0]), 'businessid': ratings_bus[a][1]}
529563
except ValueError as e:
530564
continue
531565
elif calculate_distance < distance_in_miles and calculate_distance >= 5 and calculate_distance <= 15:
532-
try:
566+
try:
533567
question = "How about this location?"
534-
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
535-
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
568+
new_message = "We saw that you took a vehicle from" + begin_string_address + "to" + address
569+
suggested_loc = "Instead, there is " + a + "which has better reviews and closer to your original starting point"
536570
rating_mess = "Rating of " + str(ratings_bus[a][0])
537571
# insert_into_db(tripDict, i, yelp_suggestion_trips, uuid)
538572
return {'message' : new_message, 'question': question, 'suggested_loc': suggested_loc, 'method': 'public', 'rating': str(ratings_bus[a][0]), 'businessid': ratings_bus[a][1]}

emission/integrationTests/suggestionsys/algorithm_test_harness.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,4 +189,3 @@ def test_single_instance(test_fn, cfn, instance, noise_in_meters):
189189
logging.info("Test complete, comparison results = ")
190190
for cfn_name, successfulTests, failedTests in cfn2resultlist:
191191
logging.info("candidate: %s, accuracy = %s" % (cfn_name, (successfulTests * 100) / (successfulTests + failedTests)))
192-

emission/integrationTests/suggestionsys/category_of_business_nominatim.dataset.json

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,11 @@
3535
},
3636

3737
{
38-
"test_name": "Manhattan Beach FISHBAR",
38+
"test_name": "Manhattan Beach FISHBAR",
3939
"test_type": "Suburban local bar/restaurant, non-chain",
4040
"input": {
4141
"loc": {
42-
"coordinates": [-118.4182952, 33.9017119],
42+
"coordinates": [-118.4182952, 33.9017119],
4343
"type": "Point"
4444
}
4545
},
@@ -128,7 +128,7 @@
128128
}
129129
},
130130
"output": ["medcenters"]
131-
},
131+
},
132132
{
133133
"test_name": "Walmart Supercenter",
134134
"test_type": "Suburban big box store by itself in a plaza",
@@ -139,8 +139,8 @@
139139
}
140140
},
141141
"output": ["deptstores", "grocery"]
142-
},
143-
{
142+
},
143+
{
144144
"test_name": "Loving Hut",
145145
"test_type": "Restaurant in a large plaza with other restaurants",
146146
"input": {
@@ -150,7 +150,7 @@
150150
}
151151
},
152152
"output": ["vegetarian", "vegan", "asianfusion"]
153-
},
153+
},
154154
{
155155
"test_name": "San Francisco Premium Outlets",
156156
"test_type": "A large outlet with a lot of stores inside",
@@ -205,7 +205,7 @@
205205
}
206206
},
207207
"output": ["hotdog", "pizza", "hotdogs"]
208-
},
208+
},
209209
{
210210
"test_name": "Layang Layang",
211211
"test_type": "A restaurant in a plaza with Save Marts",
@@ -216,7 +216,7 @@
216216
}
217217
},
218218
"output": ["malaysian"]
219-
},
219+
},
220220
{
221221
"test_name": "Kaiser Permanente Pediatrics Building",
222222
"test_type": "Clinic next to a road",
@@ -227,7 +227,7 @@
227227
}
228228
},
229229
"output": ["medcenters"]
230-
},
230+
},
231231
{
232232
"test_name": "Taco Bell",
233233
"test_type": "Chain fast-food restaurant",
@@ -390,7 +390,7 @@
390390
"type": "Point"
391391
}
392392
},
393-
"output": ["highschools"]
393+
"output": ["highschools"]
394394
},
395395
{
396396
"test_name": "Giraffe Learning Center",

0 commit comments

Comments
 (0)