Bug: Conflicting dependencies with Pydantic V2

[barreeeiroo]

Expected Behaviour

When defining a requirements.txt file with dependencies, it should be possible to define it like this:

aws-lambda-powertools[all]~=2.22.0
pydantic[email]~=2.1.1

Running pip install -r requirements.txt should install both aws-lambda-powertools and pydantic packages. AWS Lambda Powertools is compatible with Pydantic V2 thanks to #2733.

Current Behaviour

Running pip install -r requirements.txt raises the following error:

ERROR: Cannot install aws-lambda-powertools[all]==2.22.0 and pydantic[email]==2.1.1 because these package versions have conflicting dependencies.

The conflict is caused by:

• aws-lambda-powertools[all] 2.22.0 depends on pydantic<2.0.0 and >=1.8.2; extra == "parser" or extra == "all"
• pydantic[email] 2.1.1 depends on pydantic 2.1.1 (Installed)

Code snippet

aws-lambda-powertools[all]~=2.22.0
pydantic[email]~=2.1.1

pip install -r requirements.txt

Possible Solution

I was surprised when finding this error because, based on #2672 and #2733, it seems like AWS Lambda Powertools was already compatible with Pydantic V2.

However, when looking at the pyproject.toml file I found out about the dependency being still in V1:

[tool.poetry.dependencies]
...
pydantic = { version = "^1.8.2", optional = true }

Shouldn't this be set to be either ^1.8.2 or ^2.0.3 (based on Pydantic V2 usage docs)?

So, a possible solution could be:

[tool.poetry.dependencies]
...
pydantic = { version = ">=1.8.2, !=2.0.0, !=2.0.1, !=2.0.2, <3.0.0", optional = true }

Ideally, it would be nice to have something like ^1.8.2 || ^2.0.3, but not sure if that's allowed in Poetry.

Steps to Reproduce

1. Create a requirements.txt file.
2. Include both AWS Lambda Powertools and Pydantic V2 as direct dependencies.
3. Try to install dependencies through pip install -r requirements.txt.

Powertools for AWS Lambda (Python) version

2.22.0

AWS Lambda function runtime

3.11

Packaging format used

Lambda Layers, PyPi

Debugging logs

No response

[leandrodamascena]

Hello @barreeeiroo! Thank you for opening this issue! This is actually not a bug. We've kept Pydantic v1 as the default in Powertools to avoid issues with dependency resolutions for customers who are using Powertools with Pydantic v1 and don't want to switch to Pydantic v2.

To add the use of Pydantic v2 with Powertools, you must remove the optional dependency for Pydantic. In you case you can change your file:

aws-lambda-powertools[aws-xray-sdk, fastjsonschema]~=2.22.0
pydantic[email]~=2.1.1

Reference: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parser/#using-pydantic-v2

Thanks

[barreeeiroo]

Hi @leandrodamascena, you're totally right. I've updated the requirements.txt file to your suggestion and it's now working (I didn't think about removing the parser extra), sorry about the not-bug.

As a possible improvement though, maybe the documentation regarding Pydantic V2 could be improved a little bit. I have already checked the link you mention before (Using Pydantic V2) and my assumption was that I "just" had to include the pydantic v2 dependency, which is what I did. I didn't think about removing the parser extra from aws-lambda-powertools.
Now that I read more carefully the RFC, I can see that you mentioned "customers should refrain from using [all] or [parser] when bringing Pydantic v2 as part of their dependencies" but couldn't actually find it in the documentation where its mentioned how to use Parser with Pydantic V2 (that's why I assumed I only had to include Pydantic V2 as dependency, without any additional task).

In the documentation, right below where it says "Add aws-lambda-powertools and pydantic>=2.0.3 as a dependency in your preferred tool", maybe a warning alert box could be created stating something like "Make sure your aws-lambda-powertools dependency doesn't include the [all] or [parser] extras to avoid the dependency conflict.", just like the RFC said.
I can take care of adding it if you'd like to add that clarification about Pydantic V2 usage.

---

EDIT: requirements.txt should be:

aws-lambda-powertools[tracer,validation]~=2.22.0

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.