Proposal: Testing infrastructure

[smothiki]

After working on the test suite for a good amount of time and getting help from fellow folks.
These are some issues I'm thinking of proposing to make the suite better. During our last retro, we have already talked about running smoke tests.

Some Proposals about proceeding further with current CI/CD infrastructure.

view of the current Deis architecture

The control Plane:

• Controller, Builder, Registry, Database, Minio
  Any changes made to these components or Deis Cli or controller-sdk-go repository will affect workflow functionality and should run the full test suite.

The Logging and Monitoring stack:

• Fluentd, Logger, Grafana, NSQ, InfluxDB.
  A Pr change to this stack will not affect any CLI functionality or workflow features that are related to control plane except Deis logs

sub-components:

• Slugbuilder, slugrunner, dockerfilebuilder
  A pr change to these will only affect a few tests in the workflow e2e and instead of running entire test suite and git push with different app types test should be sufficinet for testing.

Ideas

• Each component in the architecture should have it's own dedicated test suite and clusters.
• For example :
• control plane should have a helm chart or manifests, which will install the control plane components on to Deis namespace. The cluster should already be running Logging and monitoring stack.
• Logging and monitoring stack should have it's own dedicated cluster where control plane is already present and helm chart or manifests are needed to install this stack on to existing control plane in Deis namespace.
• subcomponents like slugbuilder, slugrunner and dockerfile builder etc., should run specific tests in e2e or should have a dedicated test suite that would run specific tests to check if these components are working.
• Note This point is a complete outlier but if implemented would cut GCS costs. We should also look at installing Deis in different namespaces within the same cluster. This might require some code change in each component but would save cluster costs and also provides user a chance to install deis in a custom namespace.

Please provide valuable feedback and comments to proceed further in this proposal and come up with action items that we can work on to make Jenkins a green place .

[bacongobbler]

Great write-up @smothiki. Going to sit down and think about this one overnight.

[bacongobbler]

Some great thoughts in here.

On the view of the current architecture, one of our most recent projects is registry-proxy, and it was not mentioned here. I think registry-proxy should be considered as a sub-component, since it's just a dumb reverse proxy for the registry.

When you mean

Each component in the architecture should have it's own dedicated test suite and clusters.

What would this look like? workflow-e2e becomes strictly for the control plane, then we would implement a separate logging-e2e and monitoring-e2e suites? Would the sub-components just run local functional tests like deis/postgres's test suite, or something different?

[mboersma]

WRT the last point, I think beefy unit tests and local functional tests are the place to start for a component-specific suite. Ideally we can get to a point where we test all visible interfaces thoroughly enough with mocks that workflow-e2e is not required to merge a PR.

[vdice]

This definitely represents a smarter approach for when and how to utilize the e2e suite(s); thanks, @smothiki.

Here are my ideas of possible immediate next steps that can be pursued:

1. No change for 'control plane' ci pipelines at this point (keep running full e2e suite as per usual against changes). This could (and should) definitely be revisited as functional test coverage is added/improved in a given repo.
2. Break out logging/monitoring stack from full e2e by identifying the current e2e specs that touch this functionality and 'tagging' or otherwise labeling them so that the e2e stage of the ci pipeline only runs these. This would involve an organizational change/update in workflow-e2e and similarly to the pipeline job(s) in jenkins-jobs. This can be done while unit/functional tests are beefed up in these repos. Eventually, the ci pipeline would add a 'functional' stage of tests in between component build and (minimal, sub-suite) e2e.
3. Similarly for the other sub-components, break out from full e2e by identifying current specs that touch this functionality, done in parallel w/ unit/functional test coverage improvements.
4. Build momentum on functional test efforts in all component repos above by creating the necessary tickets and prioritizing work.

[smothiki]

@mboersma

WRT the last point, I think beefy unit tests and local functional tests are the place to start for a ....

I forgot to mention this one in few ideas I discussed with @vdice. We will still run the entire test suite like the current e2e but not for every PR but as a cron Job which runs 4 times every day on a dedicated cluster.

[bacongobbler]

We will still run the entire test suite like the current e2e but not for every PR but as a cron Job which runs 4 times every day on a dedicated cluster.

Will we be doing this as well as the entire test suite for control plane PRs?

[smothiki]

Just had a discussion with @vdice thanks to his suggestions.
We would love to proceed with control plane testing and below are the changes we would love to make to the existing suite.

• Create a separate helm chart for control plane. which has all the logging and monitoring charts as manifests instead of templates and use helm keep.
• Don't reap the cluster instead do helm uninstall with out force which will persist all helm keep manifests

So for every PR to control plane components we have a clusters which has logging and monitoring and helm keep manifests already installed . Just doing helmc install chart-name will install only control plane components. If we have consent on this we would love to proceed to implementation on this front.

Pros : turn around test time is optimized .
cons: book keeping for helm- keep manifest changes and changes in logging stack.

[smothiki]

@bacongobbler as per vdice suggestion cron jobs are not helping much in identifying bugs. So thinking of better approach as of now.

[jchauncey]

If all you are testing is the control plane there is no benefit to installing the monitoring and logging stack. Second you gain no benefit to keeping the cluster around.

There is also no reason to stand up the control plane if you are testing the monitoring stack.

I'm not sure I understand the benefits here.