Instrument test code not working

[basickarl]

I'd like to have coverage on the test files themselves to make sure all of our test files are being hit and that all of the lines in them are being executed. I've gotten this to work with mocha, but not with cypress yet.

I followed the following instructions: https://github.com/cypress-io/code-coverage#instrument-unit-tests.

Current behavior:

Cypress:

karl@karl-Dell-Precision-M3800:~/dev/dummy$ npx nyc --nycrc-path=./.nycrc.json cypress run --config reporter=spec

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.4.0                                                                              │
  │ Browser:    Electron 61 (headless)                                                             │
  │ Specs:      1 found (integration/example-integration-client.spec.js)                           │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running: integration/example-integration-client.spec.js...                               (1 of 1) 


  example client integration test
    ✓ should just pass (113ms)


  1 passing (138ms)


  (Results)

  ┌──────────────────────────────────────────────────────────────┐
  │ Tests:        1                                              │
  │ Passing:      1                                              │
  │ Failing:      0                                              │
  │ Pending:      0                                              │
  │ Skipped:      0                                              │
  │ Screenshots:  0                                              │
  │ Video:        true                                           │
  │ Duration:     0 seconds                                      │
  │ Spec Ran:     integration/example-integration-client.spec.js │
  └──────────────────────────────────────────────────────────────┘


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /home/karl/dev/dummy/cypress/videos/integration/example-integration-client.spec.js.mp4 (0 seconds)


====================================================================================================

  (Run Finished)


      Spec                                                Tests  Passing  Failing  Pending  Skipped 
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔ integration/example-integration-clie…     132ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    All specs passed!                           132ms        1        1        -        -        -  

-------------------------------------|----------|----------|----------|----------|-------------------|
File                                 |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
-------------------------------------|----------|----------|----------|----------|-------------------|
All files                            |        0 |        0 |        0 |        0 |                   |
 dummy                               |        0 |        0 |        0 |        0 |                   |
  sandbox.js                         |        0 |      100 |        0 |        0 |... 61,64,67,68,71 |
  server.js                          |        0 |        0 |        0 |        0 |... 55,65,66,68,69 |
 dummy/client                        |        0 |        0 |        0 |        0 |                   |
  Chart.min.js                       |        0 |        0 |        0 |        0 |                 7 |
  client.js                          |        0 |        0 |        0 |        0 |... 76,279,290,292 |
 dummy/cypress/integration           |        0 |      100 |        0 |        0 |                   |
  example-integration-client.spec.js |        0 |      100 |        0 |        0 |             1,2,3 |
 dummy/cypress/plugins               |        0 |      100 |        0 |        0 |                   |
  index.js                           |        0 |      100 |        0 |        0 |                14 |
 dummy/cypress/support               |        0 |        0 |        0 |        0 |                   |
  commands.js                        |        0 |        0 |        0 |        0 |                   |
  index.js                           |        0 |        0 |        0 |        0 |                   |
 dummy/tests/integration             |        0 |      100 |        0 |        0 |                   |
  example-integration-server.spec.js |        0 |      100 |        0 |        0 |           1,3,4,5 |
-------------------------------------|----------|----------|----------|----------|-------------------|

Mocha:

karl@karl-Dell-Precision-M3800:~/dev/dummy$ npx nyc --nycrc-path=./.nycrc.json mocha ./tests/**/*.spec.js


  example server integration test
    ✓ should just pass


  1 passing (6ms)

-------------------------------------|----------|----------|----------|----------|-------------------|
File                                 |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
-------------------------------------|----------|----------|----------|----------|-------------------|
All files                            |     0.12 |        0 |     0.23 |     2.22 |                   |
 dummy                               |        0 |        0 |        0 |        0 |                   |
  sandbox.js                         |        0 |      100 |        0 |        0 |... 61,64,67,68,71 |
  server.js                          |        0 |        0 |        0 |        0 |... 55,65,66,68,69 |
 dummy/client                        |        0 |        0 |        0 |        0 |                   |
  Chart.min.js                       |        0 |        0 |        0 |        0 |                 7 |
  client.js                          |        0 |        0 |        0 |        0 |... 76,279,290,292 |
 dummy/cypress/integration           |        0 |      100 |        0 |        0 |                   |
  example-integration-client.spec.js |        0 |      100 |        0 |        0 |             1,2,3 |
 dummy/cypress/plugins               |        0 |      100 |        0 |        0 |                   |
  index.js                           |        0 |      100 |        0 |        0 |                14 |
 dummy/cypress/support               |        0 |        0 |        0 |        0 |                   |
  commands.js                        |        0 |        0 |        0 |        0 |                   |
  index.js                           |        0 |        0 |        0 |        0 |                   |
 dummy/tests/integration             |      100 |      100 |      100 |      100 |                   |
  example-integration-server.spec.js |      100 |      100 |      100 |      100 |                   |
-------------------------------------|----------|----------|----------|----------|-------------------|

Desired behavior:

The following in the Cypress test should be at 100%, since it was hit.

 dummy/cypress/integration           |        0 |      100 |        0 |        0 |                   |
  example-integration-client.spec.js |        0 |      100 |        0 |        0 |             1,2,3 |

Steps to reproduce: (app code and test code)

cypress/integration/example-integration-client.spec.js:

describe('example client integration test', function() {
    it('should just pass', async function() {
        assert(true);
    });
});

cypress/plugins/index.js:

module.exports = (on, config) => {
    on('task', require('@cypress/code-coverage/task'));
    on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'));
};

cypress/support/index.js:

import './commands';
import '@cypress/code-coverage/support';

.babelrc:

{
    "plugins": ["istanbul"]
}

.mocharc.json:

{
    "ui": "tdd"
}

.nycrc.json:

{
    "branches": 0,
    "lines": 0,
    "functions": 0,
    "statements": 0,
    "check-coverage": true,
    "exclude-after-remap": false,
    "reporter": ["text", "html"],
    "clean": true,
    "cache": false,
    "all": true
}

cypress.json:

{
    "integrationFolder": "./cypress/",
    "testFiles": "**/*.spec.js"
}

package.json:

{
    "name": "dummy",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "scripts": {
        "start": "browser-refresh server.js",
    },
    "author": "",
    "license": "ISC",
    "dependencies": {
        "express": "^4.16.4"
    },
    "devDependencies": {
        "@cypress/code-coverage": "^1.8.0",
        "babel-plugin-istanbul": "^5.2.0",
        "browser-refresh": "^1.7.3",
        "cypress": "^3.4.0",
        "eslint": "^5.15.2",
        "istanbul-lib-coverage": "^2.0.5",
        "mocha": "^6.2.0",
        "nyc": "^14.1.1"
    }
}

Versions

Default everything on Cypress 3.4.0
Ubuntu 14.04

[bahmutov]

Hi @basickarl

2 things:

• can you remove async from your test? That will just not work except for very limited circumstances

    it('should just pass', async function() {
        assert(true);
    });

• we strip code coverage information from the spec files, because I don't think you are interested in the spec files, just in the source files, see https://github.com/cypress-io/code-coverage/blob/master/support.js#L63 - maybe it could be an option.

[basickarl]

@bahmutov Ah I see! So it's stripped via Cypress, I would like this information as it helps me and my team keep track if every line in the test files are executed and that there is no dead code in the test files (hence coverage).

Thanks for the link, I'll have a look!