Skip to content

ref(*): Refactor the whole thing #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
krancour merged 1 commit into from
Apr 12, 2016
Merged

ref(*): Refactor the whole thing #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
FROM quay.io/deis/go-dev:0.9.0
FROM quay.io/deis/go-dev:0.10.0

ENV JUNIT=true

RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates curl wget openssh-client git g++ gcc libc6-dev make bzr git mercurial openssh-client subversion procps && \
Expand All @@ -16,3 +18,7 @@ RUN curl -O https://storage.googleapis.com/kubernetes-release/release/v1.1.8/bin

COPY . /go/src/github.com/deis/workflow-e2e
WORKDIR /go/src/github.com/deis/workflow-e2e

RUN glide install

CMD ["/usr/bin/make", "test-integration"]
90 changes: 57 additions & 33 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,63 +1,87 @@
export GO15VENDOREXPERIMENT=1

SHORT_NAME := deis-e2e
SHORT_NAME := workflow-e2e

SRC_PATH := /go/src/github.com/deis/workflow-e2e
DEV_IMG := quay.io/deis/go-dev:0.9.1

RUN_CMD := docker run --rm -e DEIS_ROUTER_SERVICE_HOST=${DEIS_ROUTER_SERVICE_HOST} -e DEIS_ROUTER_SERVICE_PORT=${DEIS_ROUTER_SERVICE_PORT} -v ${CURDIR}:${SRC_PATH} -w ${SRC_PATH} ${DEV_IMG}
DEV_CMD := docker run --rm -e GO15VENDOREXPERIMENT=1 -v ${CURDIR}:${SRC_PATH} -w ${SRC_PATH} ${DEV_IMG}

TEST_OPTS := -slowSpecThreshold=120.00 -noisyPendings=false
PARALLEL_TEST_OPTS := ${TEST_OPTS} -p

MUTABLE_VERSION ?= canary
VERSION ?= git-$(shell git rev-parse --short HEAD)

ifdef GINKGO_NODES
export GINKO_NODES_ARG=-nodes=${GINKGO_NODES}
else
export GINKO_NODES_ARG=-p
endif

TEST_OPTS := -slowSpecThreshold=120.00 -noisyPendings=false ${GINKO_NODES_ARG}

DEIS_REGISTRY ?= quay.io/
IMAGE_PREFIX ?= deis
IMAGE := ${DEIS_REGISTRY}${IMAGE_PREFIX}/${SHORT_NAME}:${VERSION}
MUTABLE_IMAGE := ${DEIS_REGISTRY}${IMAGE_PREFIX}/${SHORT_NAME}:${MUTABLE_VERSION}

.PHONY: bootstrap
ifndef DEIS_CONTROLLER_URL
ifdef DEIS_ROUTER_SERVICE_HOST
export DEIS_CONTROLLER_URL=http://deis.${DEIS_ROUTER_SERVICE_HOST}.nip.io
endif
endif
Copy link
Copy Markdown
Member

@arschles arschles Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krancour do you think it's worth putting this logic into the test code itself?

Copy link
Copy Markdown
Contributor Author

@krancour krancour Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had done it this way because I felt it was more transparent. Speaking personally, if I don't know how to run something (like a test suite), the Makefile is where I look first. That said, I don't mind relocating this to the test code itself if you think there's a strong argument to be made for it.

Copy link
Copy Markdown
Member

@arschles arschles Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The program should at least be self documenting. How about a middle ground, where the test code at least prints out an explanation if it doesn't find DEIS_CONTROLLER_URL (unless it does already?)

Copy link
Copy Markdown
Contributor Author

@krancour krancour Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems reasonable. I still want to fix this before merge.

Copy link
Copy Markdown
Contributor Author

@krancour krancour Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arschles as we discussed offline, I've attempted this and there's a problem that arises from Ginkgo's architecture. When executing tests in parallel, each Ginkgo node runs as its own process. If the DEIS_CONTROLLER_URL environment variable is not set and is also not derivable from DEIS_ROUTER_SERVICE_HOST, you are warned about this once per node... and there isn't anything immediately obvious we can do about that.

We decided together to defer fixing this until beta3.

[kent@mbp workflow-e2e]$ make test-integration
ginkgo -slowSpecThreshold=120.00 -noisyPendings=false -nodes=15 tests/

     -------------------------------------------------------------------
    |                                                                   |
    |  Ginkgo timed out waiting for all parallel nodes to report back!  |
    |                                                                   |
     -------------------------------------------------------------------

[1] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[2] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[3] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[4] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[5] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[6] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[7] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[8] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[9] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[10] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[11] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[12] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[13] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[14] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!
[15] 2016/04/12 15:59:19 Environment variable DEIS_CONTROLLER_URL must be exported to proceed!

Ginkgo ran 1 suite in 3.560606126s
Test Suite Failed
make: *** [test-integration] Error 1
[kent@mbp workflow-e2e]$

Copy link
Copy Markdown
Member

@bacongobbler bacongobbler Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's file an issue and merge this.


DEV_IMG := quay.io/deis/go-dev:0.10.0
DEV_CMD_ARGS := --rm -v ${CURDIR}:${SRC_PATH} -w ${SRC_PATH} ${DEV_IMG}
DEV_CMD := docker run ${DEV_CMD_ARGS}
DEV_CMD_INT := docker run -it ${DEV_CMD_ARGS}
RUN_CMD := docker run --rm -e GINKGO_NODES=${GINKGO_NODES} \
-e DEIS_CONTROLLER_URL=${DEIS_CONTROLLER_URL} \
-e DEFAULT_EVENTUALLY_TIMEOUT=${DEFAULT_EVENTUALLY_TIMEOUT} \
-e MAX_EVENTUALLY_TIMEOUT=${MAX_EVENTUALLY_TIMEOUT} \
-e JUNIT=${JUNIT} \
-e DEBUG=${DEBUG} \
-v ${HOME}/.kube:/root/.kube \
-w ${SRC_PATH} ${IMAGE}

check-controller-url:
@if [ -z "$$DEIS_CONTROLLER_URL" ]; then \
echo "DEIS_CONTROLLER_URL is not exported. You must export this variable to proceed."; \
echo "Its value should match the Deis Controller URL you would ordinarily use with"; \
echo "the \`deis register\` or \`deis login\` commands."; \
exit 2; \
fi
Copy link
Copy Markdown
Member

@arschles arschles Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here RE putting this logic into test code itself


dev-env:
${DEV_CMD_INT} bash

bootstrap:
${DEV_CMD} glide install
glide install

.PHONY: test-integration
test-integration:
DEFAULT_EVENTUALLY_TIMEOUT="30s" ginkgo ${TEST_OPTS} tests/
docker-bootstrap:
${DEV_CMD} make bootstrap

.PHONY: test-integration
test-integration-parallel:
ginkgo ${PARALLEL_TEST_OPTS} tests/
test-integration: check-controller-url
ginkgo ${TEST_OPTS} tests/

.PHONY: docker-build
docker-build:
docker build -t ${IMAGE} ${CURDIR}
docker tag -f ${IMAGE} ${MUTABLE_IMAGE}

.PHONY: docker-push
docker-push: docker-immutable-push docker-mutable-push

.PHONY: docker-immutable-push
docker-immutable-push:
docker push ${IMAGE}

.PHONY: docker-mutable-push
docker-mutable-push:
docker push ${MUTABLE_IMAGE}

.PHONY: docker-test-integration
# run tests inside of a container
# run tests in parallel inside of a container
docker-test-integration:
docker run -e DEIS_ROUTER_SERVICE_HOST=${DEIS_ROUTER_SERVICE_HOST} \
-e DEIS_ROUTER_SERVICE_PORT=${DEIS_ROUTER_SERVICE_PORT} \
-e TEST_OPTS=${TEST_OPTS}
-e DEFAULT_EVENTUALLY_TIMEOUT="30s" ${IMAGE}

.PHONY: docker-test-integration-parallel
# run tests inside of a container
docker-test-integration-parallel:
docker run -e DEIS_ROUTER_SERVICE_HOST=${DEIS_ROUTER_SERVICE_HOST} \
-e DEIS_ROUTER_SERVICE_PORT=${DEIS_ROUTER_SERVICE_PORT} \
-e TEST_OPTS=${PARALLEL_TEST_OPTS} ${IMAGE}
${RUN_CMD} make test-integration

.PHONY: check-controller-url \
dev-env \
bootstrap \
docker-bootstrap \
test-integration \
docker-build \
docker-push \
docker-immutable-push \
docker-mutable-push \
docker-test-integration
94 changes: 86 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Deis End to End Tests v2
# Deis Workflow End to End Tests v2

[![Build Status](https://travis-ci.org/deis/workflow-e2e.svg?branch=master)](https://travis-ci.org/deis/workflow-e2e)
[![Go Report Card](http://goreportcard.com/badge/deis/workflow-e2e)](http://goreportcard.com/report/deis/workflow-e2e)
Expand All @@ -10,7 +10,7 @@ For more information about the Deis Workflow, please visit the main project page

## Beta Status

This Deis component is currently in beta status, and we welcome your input! If you have feedback, please [submit an issue][issues]. If you'd like to participate in development, please read the "Development" section below and [submit a pull request][prs].
Deis Workflow (including these tests) is currently in beta status, and we welcome your input! If you have feedback, please [submit an issue][issues]. If you'd like to participate in development, please read the "Development" section below and [submit a pull request][prs].

# About

Expand All @@ -32,21 +32,99 @@ Before you run the tests, you'll need a full Deis cluster up and running in Kube

## Run the Tests

To run the entire test suite:
There are three options for how to execute the tests. These include two options for executing the tests against Deis Workflow installed on a _remote_ Kubernetes cluster, and one option for installing the same tests directly into a Kubernetes cluster and executing them there.

### Remote Execution

Either of two options for remote execution of the test suite require the `DEIS_CONTROLLER_URL` environment variable to be exported. Its value should be the the controller endpoint you would normally use with the `deis register` or `deis login` commands:

```console
$ export DEIS_CONTROLLER_URL=http://deis.your.cluster
```

Tests execute in parallel by default. If you wish to control the number of executors, export a value for the `GINKGO_NODES` environment variable:

```console
$ export GINKGO_NODES=5
```

If this is not set, Ginkgo will automatically choose a number of test nodes (executors) based on the number of CPU cores _on the machine executing the tests_. It is important to note, however, that test execution is constrained more significantly by the resources of the cluster under test than by the resources of the machine executing the tests. The number of test nodes, therefore, should be explicitly set and scaled in proportion to the resources available in the cluster.

For reference, Workflow's own CI pipeline uses the following:

| Test Nodes | Kubernetes Worker Nodes | Worker Node CPU | Worker Node Memory |
|------------|-------------------------|-----------------|--------------------|
| 5 | 3 | 4 vCPUs | 15 GB |

Setting the `GINKGO_NODES` environment variable to a value of `1` will allow serialized execution of all tests in the suite.

#### Native Execution

If you have Go 1.5 or greater already installed and working properly and also have the [Glide](https://github.com/Masterminds/glide) dependency management tool for Go installed, you may clone this repository into your `$GOPATH`:

```console
git clone git@github.com:deis/workflow-e2e.git $GOPATH/src/github.com/deis/workflow-e2e
```

One-time execution of the following will resolve the test suite's own dependencies:

```console
$ make bootstrap
```

To execute the entire test suite:

```console
$ make test-integration
```

To run a single test or set of tests, you'll need the [ginkgo](https://github.com/onsi/ginkgo) tool installed. You can then use the `--focus` option:
To run a single test or set of tests, you'll need the [Ginkgo](https://github.com/onsi/ginkgo) tool installed on your machine:

```console
$ go get github.com/onsi/ginkgo/ginkgo
```

You can then use the `--focus` option to run subsets of the test suite:

```console
$ ginkgo --focus="deis apps" tests
```

#### Containerized Execution

If you do not have Go 1.5 or greater installed locally, but do have a Docker daemon running locally (or are using docker-machine), you can quite easily execute tests against a remote cluster from within a container.

In this case, you may clone this repository into a path of your own choosing (does not need to be on your `$GOPATH`):

```console
git clone git@github.com:deis/workflow-e2e.git /path/of/your/choice
```

Then build the test image and execute the test suite:

```console
$ make docker-build docker-test-integration
```

### Within the Cluster

A third option is to run the test suite from within the very cluster that is under test.

To install and start the tests:

```console
helm install workflow-beta2-e2e
```

To monitor tests as they execute:

```console
$ ginkgo --focus=Apps .
$ kubectl --namespace=deis logs -f workflow-beta2-e2e tests
```

## Special Note on Resetting Cluster State

Periodically, tests may not clean up after themselves and leave projects, users or other state behind, which will cause lots of test failures (often all tests will fail). If you see this behavior, run these commands to clean up (replace `deis-workflow-qoxhz`) with the name of the deis/workflow pod in your cluster):
All tests clean up after themselves, however, in the case of test failures or interruptions, automatic cleanup may not always proceed as intended. This may leave projects, users or other state behind, which may impact future executions of the test suite against the same cluster. (Often all tests will fail.) If you see this behavior, run these commands to clean up. (Replace `deis-workflow-qoxhz` with the name of the deis/workflow pod in your cluster.)

```console
$ kubectl exec -it deis-workflow-qoxhz python manage.py shell
Expand All @@ -55,10 +133,10 @@ Python 2.7.10 (default, Aug 13 2015, 12:27:27)
>>> from django.contrib.auth import get_user_model
>>> m = get_user_model()
>>> m.objects.exclude(username='AnonymousUser').delete()
>>> m.objects.all()
>>> m.objects.all()
```

Note that this is an ongoing issue for which we're planning a more comprehensive fix in [this issue](https://github.com/deis/workflow-e2e/issues/12)).
Note that this is an ongoing issue for which we're planning [a more comprehensive fix](https://github.com/deis/workflow-e2e/issues/12).
Copy link
Copy Markdown
Member

@arschles arschles Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that #12 is closed, but there are a few other related issues, PRs and commits:

Anyway, nothing actionable in this PR. This comment is purely informational. I've created #162 to track this work.


## License

Expand Down
96 changes: 0 additions & 96 deletions tests/admin_test.go
View file
Open in desktop

This file was deleted.

Loading