control-service: secrets service integration test#2289
Merged
Conversation
Add an integration test for the secrets service. Revert dependabot changes to spring-vault version. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Collaborator
Author
|
pre-commit.ci autofix |
Add an integration test for the secrets service. Revert dependabot changes to spring-vault version. Signed-off-by: Dako Dakov <ddakov@vmware.com>
…https://github.com/vmware/versatile-data-kit into person/ddakov/vault-secrets-service-integration-test
for more information, see https://pre-commit.ci
Codacy is mindless. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Codacy violations Signed-off-by: Dako Dakov <ddakov@vmware.com>
ivakoleva
approved these changes
Jun 21, 2023
...ce/src/integration-test/java/com/vmware/taurus/datajobs/it/TestVaultJobSecretsServiceIT.java
Show resolved
Hide resolved
...ce/src/integration-test/java/com/vmware/taurus/datajobs/it/TestVaultJobSecretsServiceIT.java
Outdated
Show resolved
Hide resolved
...ce/src/integration-test/java/com/vmware/taurus/datajobs/it/TestVaultJobSecretsServiceIT.java
Outdated
Show resolved
Hide resolved
projects/control-service/projects/versions-of-external-dependencies.gradle
Show resolved
Hide resolved
Address review feedback. Signed-off-by: Dako Dakov <ddakov@vmware.com>
...ol_service/src/main/java/com/vmware/taurus/secrets/service/vault/VaultJobSecretsService.java
Outdated
Show resolved
Hide resolved
antoniivanov
approved these changes
Jun 21, 2023
# Why A bug creeped in to the code where we were deploying data jobs to control namespace and we weren't catching it. We couldn't catch it because in CICD data jobs and control deploy to the same namespace. # What control and data jobs now use differnt namespace in cicd. --------- Signed-off-by: murphp15 <murphp15@tcd.ie> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…2279) What: Added string function to the stepBuilder, which is being called when a new notebook step is added to the job. This ensures that the steps are executed alphanumerically. Why: currently, when we execute "vdk run" firstly all the python and sql files are executed and then the notebook files (which is not expected since we expect vdk run to run files alphanumerically) Signed-off-by: Duygu Hasan [hduygu@vmware.com](mailto:hduygu@vmware.com)
mivanov1988
approved these changes
Jun 21, 2023
Review feedback Signed-off-by: Dako Dakov <ddakov@vmware.com>
# Why The script deploy-testing-pipelines-service.sh needs kubectl installed if you are setting up the env. Here I install it so we can use the setup env in this step. I discovered this because I am trying to create secrets in a new namespace in this PR #2269. This must be a regression from when we moved away from the aws image. --------- Signed-off-by: murphp15 <murphp15@tcd.ie>
murphp15
reviewed
Jun 21, 2023
...ol_service/src/main/java/com/vmware/taurus/secrets/service/vault/VaultJobSecretsService.java
Show resolved
Hide resolved
# Why At the moment the KubernetesService class as much to much responsibilities. It is extended by DataJobsKubernetesService and controlKubernetesService class. However it includes the logic that is unique to those individual classes also. For example only the datajobsKS creates or deploys cron jobs. So in this PR I move those external facing functions to the dataJobsclass and fix the tests. I have not move alot of the more private functions to the datajobs class yet because alot of other tests need to be changed to achieve this as they are being tested using reflection and I want to keep the PRs small. # How has this been tested? Locally. Signed-off-by: murphp15 <murphp15@tcd.ie> --------- Co-authored-by: github-actions <>
This change introduces VEP-2272, which aims at proposing an improvement to Versatile Data Kit by switching the source of truth from Kubernetes to a database. Note: This PR will cover only the first four sections. Signed-off-by: Miroslav Ivanov miroslavi@vmware.com --------- Signed-off-by: miroslavi@vmware.com Signed-off-by: Miroslav Ivanov miroslavi@vmware.com
# Why There is a regression in the build pipelines. we need to merge this so the builds can pass again. # How was this tested? On a previous commit I ran all the failing steps on this branch and it passed. --------- Signed-off-by: murphp15 <murphp15@tcd.ie> Co-authored-by: paulm2 <paulm2@vmware.com>
…s/control-service/projects/pipelines_control_service (#2301) Bumps [org.json:json](https://github.com/douglascrockford/JSON-java) from 20230227 to 20230618. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/douglascrockford/JSON-java/releases">org.json:json's releases</a>.</em></p> <blockquote> <h2>20230618</h2> <table> <thead> <tr> <th>Pull Request</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/749">#749</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/749">Prep for release 20230618</a></td> </tr> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/740">#740</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/734">Fixed Flaky Tests Caused by JSON permutations</a></td> </tr> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/734">#734</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/734">Fixed Flaky Tests Caused by JSON permutations</a></td> </tr> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/733">#733</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/733">JSONTokener implemented java.io.Closeable</a></td> </tr> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/731">#731</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/731">Removing commented out code in JSONObject optDouble()</a></td> </tr> <tr> <td><a href="https://redirect.github.com/douglascrockford/JSON-java/issues/729">#729</a></td> <td><a href="https://redirect.github.com/stleary/JSON-java/pull/729">Refactor ParserConfiguration class hierarchy</a></td> </tr> </tbody> </table> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/stleary/JSON-java/blob/master/docs/RELEASES.md">org.json:json's changelog</a>.</em></p> <blockquote> <p>20230618 Final release with Java 1.6 compatibility. Future releases will require Java 1.8 or greater.</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/douglascrockford/JSON-java/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…12.492 in /projects/control-service/projects/pipelines_control_service (#2297) Bumps [com.amazonaws:aws-java-sdk-core](https://github.com/aws/aws-sdk-java) from 1.12.490 to 1.12.492. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">com.amazonaws:aws-java-sdk-core's changelog</a>.</em></p> <blockquote> <h1><strong>1.12.492</strong> <strong>2023-06-20</strong></h1> <h2><strong>AWS Config</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Updated ResourceType enum with new resource types onboarded by AWS Config in May 2023.</li> </ul> </li> </ul> <h2><strong>AWS Lambda</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds RecursiveInvocationException to the Invoke API and InvokeWithResponseStream API.</li> </ul> </li> </ul> <h2><strong>Amazon Appflow</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds new API to reset connector metadata cache</li> </ul> </li> </ul> <h2><strong>Amazon Elastic Compute Cloud</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Adds support for targeting Dedicated Host allocations by assetIds in AWS Outposts</li> </ul> </li> </ul> <h2><strong>Amazon Redshift</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Added support for custom domain names for Redshift Provisioned clusters. This feature enables customers to create a custom domain name and use ACM to generate fully secure connections to it.</li> </ul> </li> </ul> <h1><strong>1.12.491</strong> <strong>2023-06-19</strong></h1> <h2><strong>AWS CloudFormation</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Specify desired CloudFormation behavior in the event of ChangeSet execution failure using the CreateChangeSet OnStackFailure parameter</li> </ul> </li> </ul> <h2><strong>AWS Glue</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds support for creating cross region table/database resource links</li> </ul> </li> </ul> <h2><strong>AWS Price List Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release updates the PriceListArn regex pattern.</li> </ul> </li> </ul> <h2><strong>Amazon EC2 Container Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Documentation only update to address various tickets.</li> </ul> </li> </ul> <h2><strong>Amazon Elastic Compute Cloud</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>API changes to AWS Verified Access to include data from trust providers in logs</li> </ul> </li> </ul> <h2><strong>Amazon Route 53 Domains</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Update MaxItems upper bound to 1000 for ListPricesRequest</li> </ul> </li> </ul> <h2><strong>Amazon SageMaker Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Amazon Sagemaker Autopilot releases CreateAutoMLJobV2 and DescribeAutoMLJobV2 for Autopilot customers with ImageClassification, TextClassification and Tabular problem type config support.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java/commit/2d16a8dc4839948468ac964a4761025765261646"><code>2d16a8d</code></a> AWS SDK for Java 1.12.492</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/71353f994f0a78efac98c50db4da59d1a8a8cdcc"><code>71353f9</code></a> Update GitHub version number to 1.12.492-SNAPSHOT</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/41b4cb20e0dc15bbac231994e6f503b6d2f5ff84"><code>41b4cb2</code></a> AWS SDK for Java 1.12.491</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/bd921f97536210d281200e06a9fb99653151db8b"><code>bd921f9</code></a> Update GitHub version number to 1.12.491-SNAPSHOT</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-java/compare/1.12.490...1.12.492">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 5.5.0 in /projects/control-service/projects/pipelines_control_service (#2299) Bumps [net.javacrumbs.shedlock:shedlock-spring](https://github.com/lukas-krecan/ShedLock) from 5.4.0 to 5.5.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/c78909f65ed035853dc74818985dd96d63868a94"><code>c78909f</code></a> [maven-release-plugin] prepare release shedlock-parent-5.5.0</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/3b7c72702d2176e34635425f2f397fdda05a886f"><code>3b7c727</code></a> Merge pull request <a href="https://redirect.github.com/lukas-krecan/ShedLock/issues/1458">#1458</a> from lukas-krecan/dependabot/maven/org.mockito-mocki...</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/fdc0d7d7fa33606c48093242113b88af5392e043"><code>fdc0d7d</code></a> Merge pull request <a href="https://redirect.github.com/lukas-krecan/ShedLock/issues/1459">#1459</a> from lukas-krecan/dependabot/maven/org.apache.maven....</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/542da089df14a126ec59ada53e9703487f2c4400"><code>542da08</code></a> Merge pull request <a href="https://redirect.github.com/lukas-krecan/ShedLock/issues/1457">#1457</a> from mmastika/feature/provider-datastore</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/e7c208187e689758c3f346e4fff4fe743fd5afcf"><code>e7c2081</code></a> Bump maven-shade-plugin from 3.4.1 to 3.5.0</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/2b513496b54817522a8f153497921801d7b0f747"><code>2b51349</code></a> Bump mockito-core from 5.3.1 to 5.4.0</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/5b0e7449d86ddccd4346c8cb8be95703176d9b52"><code>5b0e744</code></a> Review feedback, rename fields to fieldNames and re-jig transaction commits</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/7732d7606fa35e11463ac046e0b98a956db899b1"><code>7732d76</code></a> Merge pull request <a href="https://redirect.github.com/lukas-krecan/ShedLock/issues/1454">#1454</a> from lukas-krecan/dependabot/maven/quarkus.platform....</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/de12addc0ed330e486bcd827b46b6c0d1e871df5"><code>de12add</code></a> Merge pull request <a href="https://redirect.github.com/lukas-krecan/ShedLock/issues/1455">#1455</a> from lukas-krecan/dependabot/maven/org.springframewo...</li> <li><a href="https://github.com/lukas-krecan/ShedLock/commit/686d30643151b352b7a8e04b8689c6ccab3d0c8b"><code>686d306</code></a> GCP Datastore provider</li> <li>Additional commits viewable in <a href="https://github.com/lukas-krecan/ShedLock/compare/shedlock-parent-5.4.0...shedlock-parent-5.5.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Add an integration test for the secrets service. Revert dependabot changes to spring-vault version. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Add an integration test for the secrets service. Revert dependabot changes to spring-vault version. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Codacy is mindless. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Codacy violations Signed-off-by: Dako Dakov <ddakov@vmware.com>
Address review feedback. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Review feedback Signed-off-by: Dako Dakov <ddakov@vmware.com>
…https://github.com/vmware/versatile-data-kit into person/ddakov/vault-secrets-service-integration-test
Switch to docker-in-docker image for integration tests. Signed-off-by: Dako Dakov <ddakov@vmware.com>
Switch to docker-in-docker image for integration tests. Signed-off-by: Dako Dakov <ddakov@vmware.com>
dakodakov
deleted the
person/ddakov/vault-secrets-service-integration-test
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an integration test for the secrets service.
Revert dependabot changes to spring-vault version as the newer versions require spring 6.0+.