cli: automatic multi environment integration tests

[iliapolo]

Describe the feature

When the CLI deploys a stack as part of an integration test, automatically deploy the stack to multiple environments within a single CLI execution

Use Case

A single CLI execution can deploy multiple stacks to multiple environments. This opens the door for bugs that arise from code that doesn't take into account that each environment may contain information that affects the deployment of the stack.

For example, code that uses any bootstrap stack resource must not assume that the resource is the same in all environments.

Automatically deploying stacks to multiple environments will catch these bugs without needing to explicitly write a test for it.

Proposed Solution

There are several technical approaches we can take:

• Provide a base stack to be used in tests which duplicates itself across multiple environments.
• Mutate the assembly at runtime.
• ...

For simplicity from an auth perspective, we can narrow this down to just multi-region tests and not deal with multi account.

Regardless though, since these tests require multiple environments, we'll need to make a change to our region allocator. We might be able to get away with just acquiring an additional region lease here:

aws-cdk/packages/@aws-cdk-testing/cli-integ/lib/resource-pool.ts

Lines 69 to 76 in 33872f3

	public async using<B>(block: (x: A) => B | Promise<B>): Promise<B> {
	const lease = await this.take();
	try {
	return await block(lease.value);
	} finally {
	await lease.dispose();
	}
	}

But since we are planning a revamp to our environment allocation service, its likely better to just hold off until we implement it - taking into account requirements surfaced by this use-case.

Other Information

Some peripheral considerations:

• We need to provide a way for tests to opt out of this behavior (there may be good reasons for it).
• Do we add this as a permutation of the test? or instead?

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

ALL

Environment details (OS name and version, etc.)

ALL

[mrgrain]

I like this idea!