(applicationsignals-alpha): deployment fails on windows hosts with user/password error

[caveman-dick]

Describe the bug

When attaching application-signals functionality to a windows conainter it fails to deploy with the following error on the cloudwatch-agent container:

CannotStartContainerError: Error response from daemon: container 1800c3f9395a4353c70a4f793900a4986988604db812cfd0f45d0eb16ee4763b encountered an error during hcs::System::CreateProcess: start-amazon-cloudwatch-agent.exe: failure in a Windows system call: The user name or password is incorrect. (0x52e)

Currently the cloudwatch-agent container has it's user set to '0:1338':

aws-cdk/packages/@aws-cdk/aws-applicationsignals-alpha/lib/enablement/ecs-cloudwatch-agent.ts

Line 166 in 0bce2f0

user: '0:1338',

This is only valid for Linux hosts but is being set irrespective of the OS. This is not valid for windows hosts as per the CF documention here:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#:~:text=uid%3Agroup-,Note,-This%20parameter%20is

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

On windows hosts the container user field is not set.

Current Behavior

CannotStartContainerError: Error response from daemon: container 1800c3f9395a4353c70a4f793900a4986988604db812cfd0f45d0eb16ee4763b encountered an error during hcs::System::CreateProcess: start-amazon-cloudwatch-agent.exe: failure in a Windows system call: The user name or password is incorrect. (0x52e)

Reproduction Steps

const taskDefinition = new ecs.Ec2TaskDefinition(
    this,
    'TaskDefinition',
    {
        networkMode: ecs.NetworkMode.NAT,
        taskRole,
    },
); 

taskDefinition.addContainer('windowsservercore', {
  logging: ecs.LogDriver.awsLogs({ streamPrefix: 'win-iis-on-fargate' }),
  portMappings: [{ containerPort: 80 }],
  image: ecs.ContainerImage.fromRegistry('mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022'),
});

new appsignals.ApplicationSignalsIntegration(
    this,
    'ApplicationSignalsIntegration',
    {
        taskDefinition,
        instrumentation: {
            sdkVersion:
                appsignals.DotnetInstrumentationVersion
                    .V1_7_0_WINDOWS2022,
            runtimePlatform: {
                cpuArchitecture: ecs.CpuArchitecture.X86_64,
                operatingSystemFamily,
            },
        },
        serviceName: 'windowsservercore',
        cloudWatchAgentSidecar: {
            containerName: 'cloudwatch-agent',
            enableLogging: true,
            operatingSystemFamily,
            essential: false,
            cpu: 256,
            memoryLimitMiB: 512,
        },
    },
);

Possible Solution

Set the user to undefined for windows containers

Additional Information/Context

Escape-hatch workaround (index will be variable depending on your task definition):

const cfnTaskDef = taskDefinition.node.defaultChild as ecs.CfnTaskDefinition;
cfnTaskDef.addOverride('Properties.ContainerDefinitions.2.User', undefined);

AWS CDK Library version (aws-cdk-lib)

2.197.0

AWS CDK CLI version

2.1016.1 (build 6de56b2)

Node.js Version

24.0.1

OS

Windows 11/Windows Server 2022

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @caveman-dick,

Thank you for reporting this issue. I've analyzed the problem and confirmed that this is a bug in the AWS CDK ApplicationSignals integration.

The issue occurs because the CloudWatchAgentIntegration class sets the container user to '0:1338' regardless of the operating system. While this works for Linux containers, it's not valid for Windows containers and causes the deployment to fail.

The fix would be to conditionally set the user property based on the operating system family - only setting it for Linux containers and leaving it undefined for Windows containers.

For now, you can use the escape hatch workaround you mentioned:

const cfnTaskDef = taskDefinition.node.defaultChild as ecs.CfnTaskDefinition;
cfnTaskDef.addOverride('Properties.ContainerDefinitions.2.User', undefined);

We'll work on implementing a proper fix in the CDK. Would you be interested in contributing a PR? The change would be relatively straightforward - we just need to add a condition to check the operating system family before setting the user property.

Thank you for helping us improve the AWS CDK!

[caveman-dick]

@pahud Thanks for the confirmation. I have got a PR almost ready for it, code is done just updating the tests. However the codespace I was trying to do it in ran out of disk-space building the project! 🤣 Just pulling it locally to try and fix there.

[pahud]

@caveman-dick Interesting. Sounds like a cloud-based codespace. What was that?

[caveman-dick]

@pahud Yeah I was trying to use a Github codespace to make the change as it was pretty small.

However, after raising this issuem, I've come to the realisation that there are a lot more problems getting ApplicationSignals working on windows containers than just fixing this issue. It all seems to stem from the restrictive networking setup on windows container hosts.

Currently I'm looking at using cfn-init to deploy the cw-agent to the host machine rather than running inside a task. Once I've got an end-to-end setup working I'll revisit this and see if I can get it working in a task.

Potentially this might work differently when using Fargate but the app I'm deploying atm doesn't support Fargate due to the use of environmentFiles.