From 0f1981e0534f11461c322d29aa5d87872f85bc11 Mon Sep 17 00:00:00 2001 From: Michael Kopp Date: Fri, 20 Dec 2024 23:42:03 +0100 Subject: [PATCH 1/4] Factor development setup into DEVELOPMENT.md Signed-off-by: Michael Kopp --- bindings/kepler.gl-jupyter/DEVELOPMENT.md | 205 ++++++++++++++++++ bindings/kepler.gl-jupyter/README.md | 162 +------------- bindings/kepler.gl-jupyter/js/.nvmrc | 1 + .../kepler.gl-jupyter/keplergl/keplergl.py | 1 + bindings/kepler.gl-jupyter/setup.py | 4 +- 5 files changed, 217 insertions(+), 156 deletions(-) create mode 100644 bindings/kepler.gl-jupyter/DEVELOPMENT.md create mode 120000 bindings/kepler.gl-jupyter/js/.nvmrc diff --git a/bindings/kepler.gl-jupyter/DEVELOPMENT.md b/bindings/kepler.gl-jupyter/DEVELOPMENT.md new file mode 100644 index 0000000000..aa867c39b1 --- /dev/null +++ b/bindings/kepler.gl-jupyter/DEVELOPMENT.md @@ -0,0 +1,205 @@ +# Local Development + +## Preface + +This file describes the keplergl bindings for jupyter, i.e. the python package you can install as `keplergl` (see below). + +It makes use of the `kepler.gl` javascript package, which is defined in `../..`, but pulls this dependency from [upstream](https://www.npmjs.com/package/kepler.gl). +Hence to build this binding it is _not_ required to build the parent project. + +### Provided entities + +The bindings actually provide two things: + +- A JupyterLab/Jupyter Notebook extension, which is displayed/interacted with in your browser when using `jupyter`. +- A python package, that can be interacted with via `import keplergl` in python. + +The python package bundles all of these together. + + +### Code organization + +The python package is declared by `./pyproject.toml`. +The actual python code is in `./keplergl`. +This provides a jupyter widget (a `ipywidgets.DOMWidget`). + +This python package contains and makes use of the javascript package `keplergl-jupyter`, which resides in [`./js`](./js). +It provides a [JupyterLab extension](https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html) (see `./js/labplugin.js`) and a [Jupyter Notebook extension](https://nbclassic.readthedocs.io/en/latest/extending/index.html) (see `./js/extension.js`) which share all the 'functional' code to display the kepler.gl-widget. + +When building the js code, this will generate javascript files in `./keplergl/static`. +When installing the python package, the code in `./js` is also automatically built (unless the build results are already there, i.e. if you manually built it). + +Some example notebooks can be found in `./notebooks`. + + +### Release + +See [`RELEASE.md`](RELEASE.md) for how this package is released. + + +## Setup + +### Environment Setup +You will need to install node, yarn and Jupyter Notebook. + +#### 1. Node and Yarn +We recommended to use [`nvm`](https://github.com/nvm-sh/nvm) to install `node` and to use `yarn` via `corepack`: + + cd js + nvm install # this will install `node` in the version specified in `.nvmrc` + corepack enable + yarn # this will install yarn in the version specified in `yarn.lock`. + +It may work to use other [node](https://nodejs.org/en/download/package-manager) `> 12` and [yarn](https://yarnpkg.com/getting-started/install) installations. + +#### Jupyter + + +- Using conda +```shell +conda install jupyter +conda install notebook 6.0.1 +``` + +- Using pip + (When working with `pip` you may want to first create a virtual environment via `python -m venv venv && source ./venv/bin/activate` -- on Ubuntu you will need to install `python3-venv` for that) +```shell +pip install jupyter +pip install notebook==6.0.1 +``` + +#### 3. Install GeoPandas + +- Using conda +```shell +conda install geopandas +``` + +- Using pip + +```shell +pip install geopandas +``` + +### Download and run keplergl in your local Jupyter Notebook + +#### Clone Repo +```shell +git clone https://github.com/keplergl/kepler.gl.git +``` + +### Setup JS +#### 1. Install Js dependencies +```sh +cd bindings/kepler.gl-jupyter +cd js +yarn +``` + +Rerun `yarn` if any js dependency has changed (usually after pulling new changes from master). + +#### 2. Load mapbox token +Add a [Mapbox access token](https://docs.mapbox.com/help/how-mapbox-works/access-tokens/) to the environment. + +```sh +export MapboxAccessTokenJupyter= +``` + +#### 3. Build the js module + +For development of the js code, it is convenient to start a local server to watch for changes: + +```shell +yarn start +``` + +If you edit a file in the `js` directory, the local server will pick up those changes. + +To build without a local server that watches for changes, use + +```shell +yarn build +``` + +### Setup jupyter + +#### 1. Install python module and enable extension from local files +This command must be run **AFTER** the `js` setup, and folder `./keplergl/static/` was created. It only needs to be run once. + +```sh +# dev install from folder containing setup.py +pip install -e . + +# only needed in dev mode, not in normal mode. +jupyter nbextension install --py --symlink --sys-prefix keplergl + +# only needed in dev mode, not in normal mode. +jupyter nbextension enable --py --sys-prefix keplergl +``` + +NOTE: +The above command `jupyter nbextension install -py --symlink --sys-prefix keplergl` is trying to create a symoblic link from the folder `bindings/kepler.gl-jupyter/keplergl/static` into jupyter's folder `nbextensions`. +Please check if there folder `nbextensions/kepler-jupyter` already exists and if so, remove (or rename) it first. + +To find the location of `nbextensions` folder, you can use the following command: +```shell +realpath $(dirname $(dirname $(which jupyter)))/share/jupyter/nbextensions +``` +For example for +``` +$ where jupyter +/Users/test/opt/anaconda3/envs/test37/bin/jupyter +``` +the nbextensions should be at: +``` +/Users/test/opt/anaconda3/envs/test37/share/jupyter/nbextensions +``` + + +#### 2. Start jupyter notebook + +```shell +cd notebooks +jupyter notebook +``` + + +#### Have fun! + +You can now start editing the `.js` and `.py` files to see changes reflected in your local notebook. After changing files in the `./js` folder, the local start script will recompile the `.js` files and put them in to `keplergl/static` folder. You need to reload the jupyter notebook page to reload the files. + + +#### 3. Development for JupyterLab + +To target JupyterLab instead of Jupyter Notebook, use + +```shell +pip install -e . # in this folder +cd js && jupyter labextension develop . --overwrite +cd ../notebooks && jupyter lab +``` +for development. +This will automatically create the correct symlinks to let `jupyter lab` pick up the extension. + +To build and install it use (in `./js`) +```shell +jupyter labextension build . +jupyter labextension install . +``` + +Alternatively you can build the extension via `jupyter labextension build .` and then manually create a symbolic link from the folder output folder `./kepler-jupyter/labextension` to JupyterLab's folder `labextensions`, e.g. `/Users/test/opt/anaconda3/envs/test37/share/jupyter/labextensions`. You will need to reload the jupyter lab page. + +The output of the jupyter labextension is defined in the file `js/package.json`: +```javascript +... +"jupyterlab": { + "extension": "babel/labplugin", + "outputDir": "../keplergl-jupyter/labextension", + "sharedPackages": { + "@jupyter-widgets/base": { + "bundled": false, + "singleton": true + } + } +} +``` diff --git a/bindings/kepler.gl-jupyter/README.md b/bindings/kepler.gl-jupyter/README.md index 417ca56a8a..89062d68e3 100644 --- a/bindings/kepler.gl-jupyter/README.md +++ b/bindings/kepler.gl-jupyter/README.md @@ -1,16 +1,15 @@ # kepler.gl for Jupyter -This is the [kepler.gl](http://kepler.gl) jupyter widget, an advanced geospatial visualization tool, to render large-scale interactive maps in Jupyter Notebook. +This is the [kepler.gl](http://kepler.gl) jupyter widget, an advanced geospatial visualization tool, to render large-scale interactive maps in Jupyter Notebook and JupyterLab. -![Kepler.gl for Jupyter][jupyter_widget] +![Kepler.gl for Jupyter](https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_widget.png) -Table of contacts +Table of contents - [Installation](#installation) - [Quick Start](#quick-start) - [Demo Notebooks](#demo-notebooks) - [Usage](#usage) - [Local Development Setup](#local-development-setup) -- [FAQ & Troubleshoot](#faq--troubleshoot) ## Installation @@ -199,8 +198,10 @@ Please note that the map is not interactive due to the limitation of Google Cola - [GeoDataFrame](https://github.com/keplergl/kepler.gl/blob/master/bindings/kepler.gl-jupyter/notebooks/GeoDataFrame.ipynb): Load GeoDataFrame to kepler.gl -https://docs.kepler.gl/docs/keplergl-jupyter#1-load-keplergl-map ## Usage + +See documentation at https://docs.kepler.gl/docs/keplergl-jupyter + - [1. Load kepler.gl](https://docs.kepler.gl/docs/keplergl-jupyter#1-load-keplergl-map) - [`keplergl.KeplerGl()`](https://docs.kepler.gl/docs/keplergl-jupyter#keplergl) - [2. Add Data](https://docs.kepler.gl/docs/keplergl-jupyter#2-add-data) @@ -221,155 +222,6 @@ https://docs.kepler.gl/docs/keplergl-jupyter#1-load-keplergl-map ## Local Development Setup -### Environment Setup -You will need to install node, yarn and Jupyter Notebook. - -#### 1. Node and Yarn -Install [node](https://nodejs.org/en/download/package-manager/#macos) `> 12`, and [yarn](https://yarnpkg.com/en/docs/install#mac-stable). Use [nvm](https://github.com/creationix/nvm) for better node version management e.g. `nvm install 12`. - - -#### 2. Install Jupyter - -- Using conda -```shell -conda install jupyter -conda install notebook 6.0.1 -``` - -- Using pip - -```shell -pip install jupyter -pip install notebook==6.0.1 -``` - -#### 3. Install GeoPandas - -- Using conda -```shell -conda install geopandas -``` - -- Using pip - -```shell -pip install geopandas -``` - -### Download and run keplergl in your local Jupyter Notebook - -#### Clone Repo -```shell -git clone https://github.com/keplergl/kepler.gl.git -``` - -### Setup JS -#### 1. Install Js module -```sh -cd bindings/kepler.gl-jupyter -cd js -yarn -``` - -#### 2. Load mapbox token -Add [Mapbox access token](https://docs.mapbox.com/help/how-mapbox-works/access-tokens/) to Node env. - -```sh -export MapboxAccessTokenJupyter= -``` - -#### 3. Build js module, start a local server to watch for changes - -```shell -npm start -``` - -You need to run step 2 and 3 to restart the js program. And step 1-3 if any js dependency has changed (Usually after pulling new changes from master). - -### Setup jupyter - -#### 1. Install python module and enable extension from local files -This command must be run **AFTER** the `js` setup, and folder `static/` was created. It only needs to be run once. - -```sh -# dev install from folder containing setup.py -pip install -e . - -# only needed in dev mode, not in normal mode. -jupyter nbextension install --py --symlink --sys-prefix keplergl - -# only needed in dev mode, not in normal mode. -jupyter nbextension enable --py --sys-prefix keplergl -``` - -NOTE: The above command `jupyter nbextension install -py --symlink --sys-prefix keplergl` is trying to create a symoblic link of the folder `bindings/kepler.gl-jupyter/keplergl/static` under the jupyter's folder `nbextensions`. Please check if there is already a folder "nbextensions/kepler-jupyter" existed, and you might need to remove it first. - -To find the location of `nbextensions` folder, you can use the following command: -```shell -$ where jupyter -/Users/test/opt/anaconda3/envs/test37/bin/jupyter - -# the nbextensions should be at: /Users/test/opt/anaconda3/envs/test37/share/jupyter/nbextensions -``` +See file [`DEVELOPMENT.md`](DEVELOPMENT.md) -#### 2. Start jupyter notebook - -```shell -cd notebooks -jupyter notebook -``` - - -#### Have fun! - -You can now start editing the .js and .py files to see changes reflected in your local notebook. After changing files in the js folder, the local start script will recompile the js files and put them in to `keplergl/static` folder. You need to reload the jupyter notebook page to reload the files. - - -#### 3. Development for JupyterLab - -To test the development work in previous 2 steps for JupyterLab. You can build the `keplergl labextension` under the `js` directory: - -```shell -jupyter labextension build . -``` - -The output of the jupyter labextension is defined in the file `js/package.json`: -```javascript -... -"jupyterlab": { - "extension": "babel/labplugin", - "outputDir": "../keplergl-jupyter/labextension", - "sharedPackages": { - "@jupyter-widgets/base": { - "bundled": false, - "singleton": true - } - } -} -``` - -Then, you can either install this labextension to test it: -```shell -jupyter labextension install . -``` -or, you can manually create a symbolic link for the folder `bindings/kepler.gl-jupyter/kepler-jupyter/labextension` under the jupyter's folder `labextensions`, e.g. `/Users/test/opt/anaconda3/envs/test37/share/jupyter/labextensions`. You will need to reload the jupyter lab page. - -[jupyter_widget]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_widget.png -[empty_map]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_empty_map.png -[geodataframe_map]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_geodataframe.png -[map_interaction]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_custom_map.gif -[load_map_w_data]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_load_map_w_data.gif -[map_add_data]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_add_data.png -[connect_data_config]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_connect_data_w_config.png -[save_widget_state]: https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/jupyter_save_state.png - -[wkt]: https://dev.mysql.com/doc/refman/5.7/en/gis-data-formats.html#gis-wkt-format -[geojson]: https://tools.ietf.org/html/rfc7946 -[feature_collection]: https://tools.ietf.org/html/rfc7946#section-3.3 -[features]: https://tools.ietf.org/html/rfc7946#section-3.2 -[data_frame]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html -[geo_data_frame]: https://geopandas.readthedocs.io/en/latest/data_structures.html#geodataframe - -[match-config-w-data]: #match-config-with-data -[data_format]: #3-data-format diff --git a/bindings/kepler.gl-jupyter/js/.nvmrc b/bindings/kepler.gl-jupyter/js/.nvmrc new file mode 120000 index 0000000000..6cffac2e11 --- /dev/null +++ b/bindings/kepler.gl-jupyter/js/.nvmrc @@ -0,0 +1 @@ +../../../.nvmrc \ No newline at end of file diff --git a/bindings/kepler.gl-jupyter/keplergl/keplergl.py b/bindings/kepler.gl-jupyter/keplergl/keplergl.py index bd3b468de6..846d021009 100644 --- a/bindings/kepler.gl-jupyter/keplergl/keplergl.py +++ b/bindings/kepler.gl-jupyter/keplergl/keplergl.py @@ -314,6 +314,7 @@ def save_to_html(self, data=None, config=None, file_name='keplergl_map.html', re - config: map config dictionary, if not provided, will use current map config - file_name: the html file name, default is keplergl_map.html - read_only: if read_only is True, hide side panel to disable map customization + - center_map: if center_map is True, the bound of the map will be updated acoording to the current map data Returns: - an html file will be saved to your notebook diff --git a/bindings/kepler.gl-jupyter/setup.py b/bindings/kepler.gl-jupyter/setup.py index a3a50b0548..2715909e32 100644 --- a/bindings/kepler.gl-jupyter/setup.py +++ b/bindings/kepler.gl-jupyter/setup.py @@ -48,7 +48,9 @@ install_npm(js_dir, npm=["yarn"], build_cmd='build'), ensure_targets(jstargets), ) -is_repo = os.path.exists(os.path.join(here, '.git')) +# This is a sub-sub-folder in a git repository, hence we check for the existence +# of `../../.git` to find out, whether we are in a cloned git repository. +is_repo = os.path.exists(os.path.join(here, os.path.pardir, os.path.pardir, '.git')) if is_repo: cmdclass['jsdeps'] = js_command else: From 8c540f0e2ce4b206c6635fa24516552356882942 Mon Sep 17 00:00:00 2001 From: Xun Li Date: Thu, 9 Jan 2025 20:22:30 -0700 Subject: [PATCH 2/4] Update bindings/kepler.gl-jupyter/DEVELOPMENT.md Co-authored-by: kopp --- bindings/kepler.gl-jupyter/DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/kepler.gl-jupyter/DEVELOPMENT.md b/bindings/kepler.gl-jupyter/DEVELOPMENT.md index aa867c39b1..d4abffb95d 100644 --- a/bindings/kepler.gl-jupyter/DEVELOPMENT.md +++ b/bindings/kepler.gl-jupyter/DEVELOPMENT.md @@ -187,7 +187,7 @@ jupyter labextension build . jupyter labextension install . ``` -Alternatively you can build the extension via `jupyter labextension build .` and then manually create a symbolic link from the folder output folder `./kepler-jupyter/labextension` to JupyterLab's folder `labextensions`, e.g. `/Users/test/opt/anaconda3/envs/test37/share/jupyter/labextensions`. You will need to reload the jupyter lab page. +Alternatively you can build the extension via `jupyter labextension build .` and then manually create a symbolic link from the folder output folder `./kepler-jupyter` to JupyterLab's folder `labextensions`, e.g. `/Users/test/opt/anaconda3/envs/test37/share/jupyter/labextensions`. You will need to reload the jupyter lab page. The output of the jupyter labextension is defined in the file `js/package.json`: ```javascript From 57e8599018c70842ca083c364f32685eabc584da Mon Sep 17 00:00:00 2001 From: Xun Li Date: Thu, 9 Jan 2025 20:23:43 -0700 Subject: [PATCH 3/4] Delete bindings/kepler.gl-jupyter/js/.nvmrc --- bindings/kepler.gl-jupyter/js/.nvmrc | 1 - 1 file changed, 1 deletion(-) delete mode 120000 bindings/kepler.gl-jupyter/js/.nvmrc diff --git a/bindings/kepler.gl-jupyter/js/.nvmrc b/bindings/kepler.gl-jupyter/js/.nvmrc deleted file mode 120000 index 6cffac2e11..0000000000 --- a/bindings/kepler.gl-jupyter/js/.nvmrc +++ /dev/null @@ -1 +0,0 @@ -../../../.nvmrc \ No newline at end of file From 018b233426f2d9172a32673d5d32b1cb0c622864 Mon Sep 17 00:00:00 2001 From: Xun Li Date: Thu, 9 Jan 2025 20:24:30 -0700 Subject: [PATCH 4/4] Apply suggestions from code review --- bindings/kepler.gl-jupyter/DEVELOPMENT.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bindings/kepler.gl-jupyter/DEVELOPMENT.md b/bindings/kepler.gl-jupyter/DEVELOPMENT.md index d4abffb95d..18a21b134c 100644 --- a/bindings/kepler.gl-jupyter/DEVELOPMENT.md +++ b/bindings/kepler.gl-jupyter/DEVELOPMENT.md @@ -58,14 +58,14 @@ It may work to use other [node](https://nodejs.org/en/download/package-manager) - Using conda ```shell conda install jupyter -conda install notebook 6.0.1 +conda install notebook ``` - Using pip (When working with `pip` you may want to first create a virtual environment via `python -m venv venv && source ./venv/bin/activate` -- on Ubuntu you will need to install `python3-venv` for that) ```shell pip install jupyter -pip install notebook==6.0.1 +pip install notebook ``` #### 3. Install GeoPandas @@ -98,7 +98,7 @@ yarn Rerun `yarn` if any js dependency has changed (usually after pulling new changes from master). -#### 2. Load mapbox token +#### 2. Load mapbox token (Optional) Add a [Mapbox access token](https://docs.mapbox.com/help/how-mapbox-works/access-tokens/) to the environment. ```sh