Skip to content

Improvement to build, install, and distribution mechanisms #1888

Open
@sdavtaker

Description

@sdavtaker
Contributor

Describe the issue

Feedback from our customers show that AWS SDK for C++ complexity to build is high. Our customers come from very diverse disciplines and backgrounds, that requires AWS SDK for C++ to build under a vast set of constraints. These constraints usually come from using different operating systems, toolchains, memory management processes, and domain specific practices to mention a few.

Since AWS SDK for C++ was first released, we extended our cmake scripts to fulfill needs from new platforms, new uses cases we identified, and new tooling mostly as a collection of small individual changes with no significant impact on the build as a whole. However, as the collection of them grew, the build as a whole became a very complex piece of code to work with. This building complexity many times translates in time spent for developers who want to install the AWS SDK for C++ to work in their projects, for contributors and reviewers wanting to implement changes to fix issues with it, for package managers to release distributions, for release managers to distribute it as a dependency of their projects.

We are working in a redesign of our cmake scripts to be released at v1.10. We expect to provide a simpler and idiomatic modern cmake experience based in standard industry patterns.
Here is a list of some of the changes we will introduce:

  • Implement superbuild pattern.
    Remove the need to pull dependencies via git submodules.
    Restructure project directories to easily distinct between generated-src and src.
    Provide CPack targets.
    Declaring dependencies closer to their targets so they are not required when not used. For example, no test tools will be required when tests are not expected to be run.
    Make configuration parameters names reflect better cmake community practices.
    A legacy mode that will mantain backward compatibility with current build scripts for allowing an easy transition to customers.
    Simplify toolchain customizations.
    Set cmake_minimum_required to 3.12 version.

We will develop these changes in branch build-dev-1.10 until ready for release. We are going to publish PRs in each step of the way and link them to this Issue. We encourage everyone interested to participate in this Issue discussion thread, and the PRs.

Steps to Reproduce

No response

Current behavior

No response

AWS CPP SDK version used

1.9

compiler and version used

any

Operating System and version

any

Activity

added
guidanceQuestion that needs advice or information.
needs-triageThis issue or PR still needs to be triaged.
breaking-changeIssue requires a breaking change to remediate.
and removed
guidanceQuestion that needs advice or information.
needs-triageThis issue or PR still needs to be triaged.
on Mar 23, 2022
sdavtaker

sdavtaker commented on Mar 23, 2022

@sdavtaker
ContributorAuthor

First preparation PR: #1886

pinned this issue on Mar 23, 2022
kaihirota

kaihirota commented on Apr 21, 2022

@kaihirota

I'm trying to use DynamoDB Streams and Kinesis within Unreal Engine, and the process is extremely complex and painful. Of course, part of it is also due to UE's fault, but that's another story. Anyway, to use a third party library within Unreal Engine, I have to package the library as a Unreal Engine "plugin" that contains a "third party library" folder which includes compiled and source code for Windows, Mac, etc. Like many people, I could not figure out how to make this work. Fortunately, I was able to purchase a plugin for a fee from the UE marketplace, and the plugin's sole value proposition is enabling users to use the AWS SDK (but only dynamodb submodule) within UE.

While I'm sure this represents a niche and rare use case of AWS SDK, this is one area which would benefit largely from easier integration of AWS.

41 remaining items

Mike4Online

Mike4Online commented on Nov 27, 2023

@Mike4Online

I run into two difficulties with building and linking to the AWS SDK for C++:

Background

I need to build past and current versions of my software product, as well as current and past versions of the AWS SDK for C++ on which it depends. My build PC runs on older version of Ubuntu (18.04 Bionic) intentionally to ensure my customers can run my program on their older Linux systems.

Issue 1: CMake builds don't let you specify paths for all dependencies

So I cannot ever rely on system LIB and INCLUDE paths for transitive dependencies such as OpenSSL, zlib, libCURL, liblzma, libuuid, etc. These would all be too old in the Ubuntu 18 system directories.

Version 1.11 of the AWS SDK for C++ allows you to explicitly specify the path to the CURL library, i.e. by defining CURL_LIBRARY='/path/to/curl/libcurl.a'. But have not not found similar support for explicitly defining paths to libraries or headers for other dependencies (OPENSSL, ZLIB, LIBUUID, LIBLZMA, etc.). Adding such support would facilitate CMake builds of the AWS SDK for C++, when they need to depend on specific versions of dependencies. Until then, I have to rely on building with vcpkg, which somehow manages to keep the dependencies straight and contained within the vcpkg build tree.

Issue 2: AWS SDK for C++ header include files are spread across multiple subdirectories

I have built the AWS SDK for C++ libraries successfully multiple times, only to find that when I run the S3-Crt demo project I get SEGFAULTS. Turns out I had not copied all of the header include files from the CMake build tree to my own header include directory. So there was a mix of AWS SDK headers from a previous build as well as updated headers from the latest build.

While this is a novice mistake to make, the AWS SDK for C++ makes it easier to mess this up, specifically:

The AWS SDK for C++ often includes API changes between two tags, even when the major and minor version number have not changed. This should be highlighted in build documentation, as some libraries follow semantic versioning practices and will bump the minor version if there is an API change.

The CMake and vcpkg builds of the AWS SDK for C++ place header include files in several directories. vcpkg does a decent job of this:

cp -r ${VCPKG_ROOT}/installed/x64-linux/include/aws/*             /my_include_path/aws/aws/
cp -r ${VCPKG_ROOT}/installed/x64-linux/include/s2n/*             /my_include_path/aws/s2n/
cp -r ${VCPKG_ROOT}/installed/x64-linux/include/smithy/*          /my_include_path/aws/smithy/
cp    ${VCPKG_ROOT}/installed/x64-linux/include/z*                /my_include_path/aws/
cp    ${VCPKG_ROOT}/installed/x64-linux/include/s2n.h             /my_include_path/aws/

But the CMake build places headers is many far-flung directories. And these directories can change from tag to tag:

cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-core/include/*                                  /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/src/aws-cpp-sdk-core/include/aws/*                          /my_include_path/aws/aws/
cp -R /CodeRepos/aws-sdk-cpp/src/aws-cpp-sdk-core/include/smithy/*                       /my_include_path/aws/smithy/
cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-s3/include/*                                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-s3-crt/include/*                                /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/include/*                                   /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-common/include/*                  /my_include_path/aws/
cp -R /CodeRepos/sdk_build_debug/crt/aws-crt-cpp/crt/aws-c-common/generated/include/*    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-auth/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-cal/include/*                     /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-compression/include/*             /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-event-stream/include/*            /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-checksums/include/*                 /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-http/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/include/*                      /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-mqtt/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-s3/include/*                      /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-sdkutils/include/*                /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-lc/include/*                        /my_include_path/aws/

Enhancing the CMake build so it prepares a consolidated set of header directories (similar to what vcpkg does) would be a handy way to ensure the entire set of headers could be easily found and copied elsewhere.

The S3 Crt demo project will log an API version mismatch error if it detects that the version in <aws/core/VersionConfig.h> does not correspond with the version within the linked or loaded AWS SDK libraries. If a similar check were to be added early in Aws::InitAPI() and made a runtime error, it might help catch this common mistake, rather allowing execution to continue to a SEGFAULT.

Mike4Online

Mike4Online commented on Dec 4, 2023

@Mike4Online

The AWS SDK online documentation still shows that building the AWS SDK for C++ on Linux requires CMake - minimum version 3.13, maximum version 3.21.

Is this maximum CMake version still valid? Ideally the SDK could be built with the latest CMake.

sbiscigl

sbiscigl commented on Dec 6, 2023

@sbiscigl
Contributor

Is this maximum CMake version still valid? Ideally the SDK could be built with the latest CMake.

there is no maximum version and it will buid on latest, i routinely use latest. let me see if i can track down the documentation to update that wording to remove maximum.

Mike4Online

Mike4Online commented on Dec 7, 2023

@Mike4Online

There is another documentation page stating that the CMake maximum version is 3.21. This one is for Windows:

Build the AWS SDK for C++ on Windows

ahmedyarub

ahmedyarub commented on Jan 14, 2024

@ahmedyarub

Any updates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CmakeCmake related submissionsbreaking-changeIssue requires a breaking change to remediate.feature-requestA feature should be added or improved.p2This is a standard priority issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @Mike4Online@sdavtaker@glaubitz@sbiscigl@jmklix

        Issue actions

          Improvement to build, install, and distribution mechanisms · Issue #1888 · aws/aws-sdk-cpp