Merge pull request #582 from dynamicslab/example-directory

Template for removing examples, including removing example 3

Author: Jacob-Stevens-Haas

.github/workflows/notify-experiments.yml

@@ -0,0 +1,16 @@
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  notify-notebook:
+    runs-on: ubuntu-latest
+    steps:
+      # Add more like these when linking external example CI
+      - name: Inform original paper
+        uses: peter-evans/repository-dispatch@v3
+        with:
+          token: ${{ secrets.PYSINDY_EXAMPLE_PAT }}
+          repository: dynamicslab/sindy-original-example
+          event-type: pysindy-commit

README.rst

@@ -173,25 +173,24 @@ Community guidelines
 
 Contributing examples
 ^^^^^^^^^^^^^^^^^^^^^
-We love seeing examples of PySINDy being used to solve interesting problems! If you would like to contribute an example, reach out to us by creating an issue.
-
-At a minimum, we need to be able to run the example notebooks in the normal mode as well as in a test mode that uses smaller data in order to run faster and simply verify that cells execute without error.  In order to do that, your example should obey the following directory tree
-
-.. code-block::
-
-  ./<name_of_example>/
-  \
-   |-example.py # save your notebook as a python script
-   |-example_data.py # has functions to create/load data
-   |-mock_data.py # has functions with same name as in example_data.py which create/load smaller datasets
-   |-example.ipynb # run python examples/publish_notebook/<name_of_example> to generate this.  Needs packages in requirements-dev.txt
-   |-utils.py (Any other names example.py needs to import.  Any additional local modules imported by example.py need to be submodules of utils.py, e.g. utils.plotting)
-
-You can optimize your notebook for testing by checking ``__name__``.  When our tests run ``example.py`` they set the ``__name__`` global to ``"testing"``.  For instance, your notebook should determine whether to import from ``mock_data`` or ``example_data`` using this method (another example: you could also use this method to set ``max_iter``).  It's a bit arbitrary, but try to make your examples run in under ten seconds using the mock data.  You can use our test to verify your example in testing mode:
-
-.. code-block::
-
-   pytest -k test_external --external-notebook="path/to/<name_of_example>"
+We love seeing examples of PySINDy being used to solve interesting problems! If you would like to contribute an example to the documentation, reach out to us by creating an issue.
+
+Examples are external repositories that
+`follow a structure <https://github.com/dynamicslab/pysindy-example>`_ that pysindy
+knows how to incorporate into its documentation build.  They tend to be pinned to
+a set of dependencies and may not be kept up to date with breaking API changes.
+
+The linked repository has information on how to set up your example.  To PR the example
+into this repository, (a) edit examples/external.yml and examples/README.rst with your
+repository information and (b) verify your own build passes in your repository,
+including publishing on github pages.  If you want to keep your example up to date with
+the pysindy main branch, (c) add your repository information to the ``notify-experiments``
+workflow so that pysindy will trigger your notebooks to be run in CI in your own repo.
+This will require adding a
+`fine-grained PAT <https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens>`_
+with the permissions ``contents: read & write`` and ``metadata: read only`` to the
+pysindy GH secrets. Alternatively, you can just trigger your builds based on cron timing.
+See the pysindy experiments repo for more information.
 
 
 Contributing code

docs/conf.py

@@ -1,6 +1,17 @@
 import importlib
+import os
+import re
 import shutil
 from pathlib import Path
+from typing import TypeVar
+
+import requests
+import yaml
+from docutils import nodes
+from docutils.statemachine import StringList
+from sphinx.application import Sphinx
+from sphinx.directives.other import TocTree
+from sphinx.util.docutils import SphinxDirective
 
 author = "dynamicslab"
 project = "pysindy"  # package name
@@ -42,9 +53,6 @@
 
 here = Path(__file__).parent.resolve()
 
-if (here / "static/custom.css").exists():
-    html_static_path = ["static"]
-
 exclude_patterns = ["build", "_build", "Youtube"]
 # pygments_style = "sphinx"
 
@@ -105,7 +113,7 @@ def patched_parse(self):
 GoogleDocstring._parse = patched_parse
 
 
-def setup(app):
+def setup(app: Sphinx):
     """Our sphinx extension for copying from examples/ to docs/examples
 
     Since nbsphinx does not handle glob/regex paths, we need to
@@ -135,3 +143,113 @@ def setup(app):
     )
     if (here / "static/custom.css").exists():
         app.add_css_file("custom.css")
+
+    _grab_external_examples(example_source)
+    app.add_directive("pysindy-example", PysindyExample)
+
+
+EXTERNAL_EXAMPLES: dict[str, list[tuple[str, str]]] = {}
+
+
+def _load_ext_config(example_source: Path) -> list[dict[str, str]]:
+    ext_config = example_source / "external.yml"
+    with open(ext_config) as f:
+        ext_examples = yaml.safe_load(f)
+    return ext_examples
+
+
+def _grab_external_examples(example_source: Path):
+    ext_examples = _load_ext_config(example_source)
+    for example in ext_examples:
+        ex_name = example["name"]
+        user = example["user"]
+        repo = example["repo"]
+        ref = example["ref"]
+        dir = example["dir"]
+        base = f"https://raw.githubusercontent.com/{user}/{repo}/{ref}/{dir}/"
+        notebooks = fetch_notebook_list(base)
+        base = f"https://raw.githubusercontent.com/{user}/{repo}/{ref}/"
+        local_nbs = [(name, copy_nb(base, pth, repo)) for name, pth in notebooks]
+        EXTERNAL_EXAMPLES[ex_name] = local_nbs
+
+
+class PysindyExample(SphinxDirective):
+    required_arguments = 0
+    optional_arguments = 0
+    option_spec = {"key": str, "title": str}
+    has_content = True
+
+    def run(self) -> list[nodes.Node]:
+        key = self.options["key"]
+        example_config = _load_ext_config((here / "../examples").resolve())
+        try:
+            this_example = [ex for ex in example_config if ex["name"] == key][0]
+        except IndexError:
+            RuntimeError("Unknown configuration key for external example")
+        heading_text: str = self.options.get("title")
+        base_repo = f"https://github.com/{this_example['user']}/{this_example['repo']}"
+        repo_ref = nodes.reference(name="Source repo", refuri=base_repo)
+        ref_text = nodes.Text("Source repo")
+        repo_ref += ref_text
+        repo_par = nodes.paragraph()
+        repo_par += repo_ref
+        normalized_text = re.sub(r"\s", "_", heading_text)
+        tgt_node = nodes.target(refid=normalized_text)
+        title_node = nodes.title()
+        title_text = nodes.Text(heading_text)
+        title_node += [title_text, tgt_node]
+        content_nodes = self.parse_content_to_nodes()
+        toc_items = []
+        for name, relpath in EXTERNAL_EXAMPLES[key]:
+            if name:
+                toc_str = f"{name} <{relpath}>"
+            if not name:
+                toc_str = relpath
+            toc_items.append(toc_str)
+        toc_nodes = TocTree(
+            name="PysindyExample",
+            options={"maxdepth": 1},
+            arguments=[],
+            content=StringList(initlist=toc_items),
+            lineno=self.lineno,
+            block_text="",
+            content_offset=self.content_offset,
+            state=self.state,
+            state_machine=self.state_machine,
+        ).run()
+        section_node = nodes.section(ids=[heading_text], names=[heading_text])
+        section_node += [title_node, *content_nodes, *toc_nodes, repo_par]
+        return [section_node]
+
+
+def fetch_notebook_list(base: str) -> list[tuple[str, str]]:
+    """Gets the list of example notebooks from a repo's index.html
+
+    Each entry is a tuple of the title name of a link and the address
+    """
+    index = requests.get(base + "index.rst")
+    if index.status_code != 200:
+        raise RuntimeError("Unable to locate external example directory")
+    text = str(index.content, encoding="utf-8")
+    link_line = r"^\s+(.*)[^\S\r\n]+(\S+.ipynb)"
+    T = TypeVar("T")
+
+    def deduplicate(mylist: list[T]) -> list[T]:
+        return list(set(mylist))
+
+    rellinks = deduplicate(re.findall(link_line, text, flags=re.MULTILINE))
+    return rellinks
+
+
+def copy_nb(base: str, relpath: str, repo: str) -> str:
+    """Create a local copy of external file, modifying relative reference"""
+    example_dir = Path(__file__).parent / "examples"
+    repo_local_dir = example_dir / repo
+    repo_local_dir.mkdir(exist_ok=True)
+    page = requests.get(base + relpath)
+    if page.status_code != 200:
+        raise RuntimeError(f"Unable to locate external notebook at {base + relpath}")
+    filename = repo_local_dir / relpath.rsplit("/", 1)[1]
+    with open(filename, "wb") as f:
+        f.write(page.content)
+    return os.path.relpath(filename, start=example_dir)

examples/3_original_paper.ipynb

@@ -1,9 +1,15 @@
 PySINDy Examples
 ================
 
-This directory showcases the following examples of PySINDy in action.
+This directory showcases examples of PySINDy in action.  Not all examples are run on
+the current master branch. They serve to show what is possible with pysindy, but do
+not necessarily use the current API.
+Each is copied from another repository that contains dependency information and
+potentially a greater description.
 
-`Feature overview <./1_feature_overview/example.ipynb>`_
+Some notebooks require substantial computing resources.
+
+Feature overview
 -----------------------------------------------------------------------------------------------------------
 This notebook gives an almost exhaustive overview of the different features available in PySINDy. It's a good reference for how to set various options and work with different types of datasets.
 
@@ -24,23 +30,20 @@ We recommend that people new to SINDy start here. We give a gentle introduction
 
     ./2_introduction_to_sindy/example
 
-`Original paper <./3_original_paper/example.ipynb>`_
--------------------------------------------------------------------------------------------------------
-This notebook uses PySINDy to reproduce the examples in the `original SINDy paper <https://www.pnas.org/content/pnas/113/15/3932.full.pdf>`_. Namely, it applies PySINDy to the following problems:
-
-* Linear 2D ODE
-* Cubic 2D ODE
-* Linear 3D ODE
-* Lorenz system
-* Fluid wake behind a cylinder
-* Logistic map
-* Hopf system
 
-.. toctree::
-    :hidden:
-    :maxdepth: 1
+.. pysindy-example::
+    :key: original
+    :title: Original Paper
 
-    ./3_original_paper/example
+    This repository recreates the results from the `original SINDy paper <https://www.pnas.org/content/pnas/113/15/3932.full.pdf>`_.
+    It applies SINDy to the following problems:
+    * Linear 2D ODE
+    * Cubic 2D ODE
+    * Linear 3D ODE
+    * Lorenz system
+    * Fluid wake behind a cylinder
+    * Logistic map
+    * Hopf system
 
 `Scikit-learn compatibility <./4_scikit_learn_compatibility/example.ipynb>`_
 -------------------------------------------------------------------------------------------------------------------------------

examples/README.rst

@@ -0,0 +1,5 @@
+- name: "original"
+  user: "dynamicslab"
+  repo: "sindy-original-example"
+  ref: "e68efeb"
+  dir: "examples"

examples/data/PODcoefficients.mat

@@ -51,11 +51,13 @@ dev = [
 ]
 docs = [
     "ipython",
+    "nbsphinx",
     "pandoc",
+    "requests",
     "sphinx-rtd-theme",
-    "sphinx==7.1.2",
+    "sphinx==7.4.7",
+    "pyyaml",
     "sphinxcontrib-apidoc",
-    "nbsphinx"
 ]
 miosr = [
     "gurobipy>=9.5.1,!=10.0.0"