[MLOB-2522] Add --llmobs flag for instrumenting Lambdas with LLM Observability

[sabrenner]

What and why?

This PR adds a --llmobs flag to the AWS Lambda instrument/uninstrument commands, which takes in a string. This sets

• DD_LLMOBS_ENABLED="true"
• DD_LLMOBS_AGENTLESS_ENABLED="false" (enabled as part of feat: add llmobs proxy paths to trace agent datadog-lambda-extension#628, our instructions will point users to make sure to install the extension layer)
• DD_LLMOBS_ML_APP to the string provided

Some additional changes/explanations/clarifications

1. Our documentation will highlight intended usage as:

datadog-ci lambda instrument -f <YOUR_LAMBDA_FUNCTION_NAME> -r <AWS_REGION> -v 106 -e 73 --llmobs <YOUR_ML_APP>

i.e, this should be used to instrument one layer, with both the language layer and extension layer specified (layer versions will be auto-populated).

2. With both layers, these three variables are the only ones needed to enable LLM Observability. DD_LLMOBS_AGENTLESS_ENABLED="false" will use the extension layer's agent as a proxy, enabled by feat: add llmobs proxy paths to trace agent datadog-lambda-extension#628

How?

Adds an --llmobs option to read, and parses it onto the llmobsMlApp setting, which if it is set, sets DD_LLMOBS_ENABLED, DD_LLMOBS_AGENTLESS_ENABLED, and DD_LLMOBS_ML_APP. This is my first time contributing, so I did my best to add tests where I saw appropriate. Happy to add more where I should if I missed some spots!

Review checklist

• Feature or bugfix MUST have appropriate tests (unit, integration)

[datadog-datadog-prod-us1]

Datadog Report

Branch report: sabrenner/lambda-instrument-llmobs
Commit report: f9cfa24
Test service: datadog-ci-tests

✅ 0 Failed, 1272 Passed, 0 Skipped, 1m 46.23s Total duration (34.48s time saved)

[duncanista]

Just curious, what happens if you don't set the app name but enable LLMObs? as in, just the two env vars

      "DD_LLMOBS_ENABLED": "true",
      "DD_LLMOBS_AGENTLESS_ENABLED": "false"

What would happen to the ML_APP name?

[sabrenner]

yeah good question. when I tried doing this in the tests, ie

const code = await cli.run(
  [
    'lambda',
    'instrument',
    // ...
    '--llmobs',
    // 'my-ml-app',
  ],
  context
)

i got a mismatched output of

Unknown Syntax Error: Command not found; did you mean one of:
...
While running lambda instrument -f arn:aws:lambda:us-east-1:123456789012:function:lambda-hello-world --dry-run --layerVersion 10 --logLevel debug --service middletier --env staging --version 0.2 --extra-tags layer:api,team:intake --no-source-code-integration --llmobs

and assumed that when using Options.String('llmobs') expects a value or otherwise causes a failure somewhere. But, i'm not too well-versed on this behavior here.

but, in general, if ml_app is not set, the LLMObs SDKs will throw/raise at runtime.

[duncanista]

Oh yeah, I didn't expected it to work in the CI command, just curious about what would happen in LLMObs in general, but

if ml_app is not set, the LLMObs SDKs will throw/raise at runtime.

answers my question.

I'm just curious because it's interesting to see a product that requires a pair of env vars to work properly, normally I'd just see the enabling and that's it – thanks!