Testing Jupyter / Colab Notebooks

[PhilipMay]

From last SIG Addons community meeting I picked up this task.

We want to discuss and implement a way to test jupyter / colab notebooks.
All discussions and ideas should go to this issue.

Furthermore the protocol sais that:

• TF team is looking at ways to test doc strings (not sure if this should go to an other issue)
• May be able to inline doc string test cases (not sure if this should go to an other issue)
• First pass can we just test to make sure the notebook executes
• Be sure to tag @karmel

[PhilipMay]

I am not even sure which jupyter / colab notebooks should be tested. I guess these here:
https://github.com/tensorflow/addons/tree/master/examples

Is that right?

[PhilipMay]

@karmel Should I open a separate issue about docstring testing?

[PhilipMay]

Since these are norml jupyter notebooks we can smoke test them by just running them. This is possible with: nbconvert --ExecutePreprocessor.timeout=None --to notebook --execute <noteboo_name>.ipynb

Also see here: https://nbconvert.readthedocs.io/en/latest/execute_api.html#executing-notebooks-from-the-command-line

If this fails because some exception is raised the smoke test does not pass.
Here is are links where they do exactly that:

1. example: https://github.com/ghego/travis_anaconda_jupyter
2. notebook testing lib: https://github.com/pyviz-dev/nbsmoke
3. notebook testing lib: https://github.com/opengeophysics/testipynb

My suggestion is to test the libs (link 2 and 3) and see if they fit our needs and decide which to use. The ppl. from link 2 (nbsmoke) say they are still "still undergoing heavy development". Lib 3. (testipynb) seems to be more mature to me.

What do you think?

[seanpmorgan]

Since these are norml jupyter notebooks we can smoke test them by just running them. This is possible with: nbconvert --ExecutePreprocessor.timeout=None --to notebook --execute <noteboo_name>.ipynb

Also see here: https://nbconvert.readthedocs.io/en/latest/execute_api.html#executing-notebooks-from-the-command-line

If this fails because some exception is raised the smoke test does not pass.
Here is are links where they do exactly that:

1. example: https://github.com/ghego/travis_anaconda_jupyter
2. notebook testing lib: https://github.com/pyviz-dev/nbsmoke
3. notebook testing lib: https://github.com/opengeophysics/testipynb

My suggestion is to test the libs (link 2 and 3) and see if they fit our needs and decide which to use. The ppl. from link 2 (nbsmoke) say they are still "still undergoing heavy development". Lib 3. (testipynb) seems to be more mature to me.

What do you think?

Thanks for taking this on @PhilipMay. First impression is that either of the two libs looks like it could work for our first pass.

[karmel]

CC @tomerk and @yashk2810 , who worked on this on the TF side.

[Smokrow]

I tried running jupyter notebook cells in a docker container a while back, but as far as I remember I failed 😄 . In case something like this is of interested I am happy to help.

[PhilipMay]

I which folder should the test be placed? I see 2 possibilities:

• tensorflow_addons <- where the other tests are also located
• examples <- where the examples are located

[yashk2810]

@MarkDaoust @lamberta

[tomerk]

CC'ing @MarkDaoust who worked on this as well for tensorflow_docs. We test the tensorflow tutorials using a custom bazel test rule that combines nbconvert calls w/ small regex rewrites of the notebook code to make it execute less work.
(e.g. we replace the number of epochs to run w/ 1, and swap real dataset downloads with mock data).

We can explore open sourcing this rule if you think it would be useful. Otherwise if the libraries you linked above work fine for you they're worth considering.

[lamberta]

Re: location
Noted in this comment, for notebooks that are intended as doc pages on tensorflow.org, they should go in the docs/tutorials/ directory to be consistent with the rest of the site. (We use "examples" to mean demo apps and other non-narrative samples.)

[PhilipMay]

I have a first prototype solution here:

NBDIR = '../../examples'

class JupyterNotebooksTest(absltest.TestCase):

    def test_jupyter_notebooks(self):
        tnb = testipynb.TestNotebooks(directory=NBDIR, timeout=2100)
        self.assertTrue(len(tnb._nbpaths) > 0)
        self.assertTrue(tnb.run_tests())

if __name__ == '__main__':
    absltest.main()

The problem is that the path (NBDIR) is hard wired.
Any idears how to improve this?
Where should we place this script? In examples directory or tensorflow_addons/utils ?

[Smokrow]

I think it would be best to keep this away from the examples folder since this folder might get more "normal user clicks" than other module directories. Maybe you can put it to the other tools. @seanpmorgan maybe here https://github.com/tensorflow/addons/tree/master/tools/ci_testing ?

[Smokrow]

@PhilipMay could you open a PR with your changes?