Run web extension tests

[kumar303]

This command should execute automated tests for a web extension within Firefox

[lidel]

Will web-ext test be a thin, battle-tested orchestration of mocha, karma and geckorunner
(as in npm run test:func of mozilla/example-addon-repo), or something else?

[kumar303]

I'm not too sure. The jpm test command seemed a little overkill to me.

In the example you cited, I think hooking into existing tools such as Mocha, Selenium, etc, is the way to go. Maybe we don't need a web-ext test at all?

[lidel]

I really appreciated jpm test as it provided me with access to real browser internals (no stubs) and "just worked". There is no easy way to do the same right now. I feel this kind of sorcery should be handled by an upstream library, not developers themselves :-)

What I am missing is an upstream test library for WebExtension that provides:

• cross-platform support (Unix/MS Windows)
• transitive dependency on well-tested libraries (karma, mocha, fx-runner, geckodriver)
• automatic addon installation in Firefox
• a library of reusable functions to test common WebExtension features
  (eg. checking if browser button was added)
• sugar-coated access to CHROME context of geckodriver, to ease implementation of missing functions

It could be under web-ext test, but due to heavy dependencies I agree, it may be an overkill.

Maybe extracting it from example-addon-repo to a separate, optional lib/tool (eg. web-ext-test) would be a better approach?

[Standard8]

@lidel fyi I just created https://github.com/Standard8/example-webextension which is going to hold the web extension example in future, as the example-addon-repo is likely to revert to hybrid for the time being (and have different structures).

It does use sinon-chrome which automatically provides stubs for the chrome.* and browser.* namespaces, which I think would help with some of your requirements.

[kumar303]

I feel this kind of sorcery should be handled by an upstream library

We probably don't need anything as low level as this. We can re-use the way web-ext run connects to the remote debugger and installs a temporary add-on. How hard would it be to run this in the setup of a geckodriver test suite?

What I am missing is ... a library of reusable functions to test common WebExtension features
(eg. checking if browser button was added)

I have a hard time understanding the value of testing that a browser button was added for a given extension. You add a button by declaring it in your manifest. The actual button adding is performed by the Firefox WebExtension platform itself and features like this are already well tested. Why do you need to test it again? Is it because you don't want to be bitten by a regression in the WebExtension platform? Regressions are certainly possible but unlikely; you will get more mileage by trying to prevent bugs in your own program first (which aren't already tested by Firefox :)).

What I am missing is ... sugar-coated access to CHROME context of geckodriver, to ease implementation of missing functions

This seems useful but can't you already do this in geckodriver easily?

I think it would be helpful to see some more iterations on real world test suites so we can better understand what to bake into web-ext test. It might be good to start by refactoring example-addon-repo to use the aforementioned temporary add-on install code (if possible).

[Standard8]

I feel this kind of sorcery should be handled by an upstream library

As commented just before the section you highlighted - that's going to be implemented in geckodriver, so it will be a simple function to call.

We probably don't need anything as low level as this. We can re-use the way web-ext run connects to the remote debugger and installs a temporary add-on. How hard would it be to run this in the setup of a geckodriver test suite?

If I understand your question correctly, then selenium/geckodriver start up Firefox themselves (with its own profile etc) for testing. I guess you could probably set it up for remote control from selenium, but then you might loose some of the profile control that selenium gives you.

What I am missing is ... a library of reusable functions to test common WebExtension features
(eg. checking if browser button was added)

I have a hard time understanding the value of testing that a browser button was added for a given extension. You add a button by declaring it in your manifest. The actual button adding is performed by the Firefox WebExtension platform itself and features like this are already well tested. Why do you need to test it again? Is it because you don't want to be bitten by a regression in the WebExtension platform?

I understand both sides of the discussion here. Taking experience from what I've worked on previously, what I would normally do is:

• Unit test to test that the function to create/setup the button is called.
  • This is a bit more difficult in WebExtension land as the manifest is declaring the button.
  • However, I do unit test that the listeners are set up correctly.
  • This isn't testing the sdk.
• Functional test that the button does what is expected.
  • In my extension when the button is clicked, it opens a new tab at a specific address. Hence, the test is for the button being clicked through to opening the tab.
  • As a side effect of that test, we check the button does what is expected, and effectively we're re-testing the SDK. You can't get away from that with functional testing.

[lidel]

Thanks! At this point, my understanding is:

1. Unit tests can be provided by mocha + sinon-chrome (as long as sinon stubs cover Firefox-specific APIs). This is fairly solid and time invested in writing unit tests with these tools will not be wasted.

2. Functional tests are possible with karma, selenium, geckodriver, but there are kinks to iron out upstream first (eg. the way temporary add-on is deployed and run),

   I have a hard time understanding the value of testing that a browser button was added for a given extension. [..]
   Is it because you don't want to be bitten by a regression in the WebExtension platform? [..]

   "Trust, but verify" -- I want to have early warning system to see if there are any issues in Nightly/Beta and either file an upstream bug or plan an update of my own code.
   Button is just an easy target for writing PoC, the real value shows in more advanced tests of course.

   As a side effect of that test, we check the button does what is expected, and effectively we're re-testing the SDK. You can't get away from that with functional testing.

   It is not a bad thing, it is a feature :-)
   The added value of jpm test from old SDK was that I was able to run regression tests against multiple Firefox versions (esr, release, beta, nightly).
   My goal is to have a similar test matrix for WebExtension.

[kumar303]

My goal is to have a similar test matrix for WebExtension.

Good point. This would be useful because it's a bit hard to track down which version of Firefox shipped which WebExtension API. We do make an effort at this in the MDN docs though.