CI: Unify CI jobs to reduce machine count by ajpallares · Pull Request #6332 · RevenueCat/purchases-ios

ajpallares · 2026-02-23T12:59:58Z

Checklist

N/A — CI-only changes, no SDK code modified

Motivation

Each CI job spins up a separate macOS machine. Many jobs that share the same Xcode version can run sequentially on a single machine, reducing the number of billable machines per CI run.

Additionally, an Apple bug (FB22032719) causes SKProductsRequest to fail on iOS 18.6 simulators under Xcode 26 (missingValue for StoreKit.ProductResponse.Key.price). This forces iOS 18 tests to run under Xcode 16 instead of Xcode 26.

Description

Merges several CI jobs that share an Xcode version into single jobs, and removes standalone jobs that are now redundant:

Unified job	Xcode	Replaces
`run-test-ios-15-and-14`	14.3.1	`run-test-ios-14` + `run-test-ios-15` + `spm-release-build-xcode-14` + `validate-package-swift-5.8`
`run-test-ios-16`	15.4.0	`run-test-ios-16` + `spm-release-build-xcode-15`
`run-test-ios-18-and-17`	16.4	`run-test-ios-17` + `run-test-ios-18`
`run-test-ios-26`	26.3	`run-test-ios-26`
`run-test-tvos-and-macos`	16.4	`run-test-tvos` + `run-test-macos` + `spm-release-build`
`run-revenuecat-ui-ios-18-and-17`	16.4	`run-revenuecat-ui-ios-17` + `run-revenuecat-ui-ios-18`

Removed standalone jobs (absorbed into the unified jobs above):

Removed job	Now covered by
`spm-release-build`	`run-test-tvos-and-macos` (Xcode 16 SPM build)
`spm-release-build-xcode-14`	`run-test-ios-15-and-14` (Xcode 14 SPM build)
`spm-release-build-xcode-15`	`run-test-ios-16` (Xcode 15 SPM build)
`validate-package-swift-5.8`	All `run-test-ios-*` jobs run `swift package describe` (Xcode 14, 15, 16, 26)

Grouping rationale:

Xcode 14.3.1 ships with iOS 15 runtime; iOS 14 is downloaded in the background while iOS 15 tests run
Xcode 15.4.0 ships with iOS 15 and 16 runtimes; standalone job for iOS 16 since iOS 15 tests are run in Xcode 14.3.1
Xcode 16.4 ships with iOS 17 and 18 runtimes; iOS 18 must run here (not Xcode 26) due to the SK1 Apple bug (reported to Apple as FB22032719)
Xcode 26.3 is required for iOS 26 only

Within each unified job, tests run from newest to oldest iOS version. Each run-test-ios-* job also runs an SPM release build as its first step, absorbing the standalone spm-release-build* jobs.

Other changes:

tvOS snapshot generation: Added tvOS-specific snapshot file prefix (tvOS{version}-) in CurrentTestCaseTracker.osVersionAndTestName and enabled snapshot generation support in the test_tvos Fastlane lane and CI job. Previously, tvOS snapshots were silently merged with iOS snapshots due to sharing the same iOS file prefix.

Known tradeoffs

Sequential test runs in unified jobs: If an earlier test run fails, later iOS versions in the same job are skipped. Previously each version was independent. This will be tackled in CI: Add test-run caching for unified job retry optimization #6333

Diff

The only changes are done in CircleCI's config.yml and fastfile. The rest of the diff are the new tvOS snapshot files

Note

Medium Risk
CI-only but broad workflow refactoring: combining previously parallel jobs into sequential runs and changing artifact/snapshot handling could cause missed coverage on early failures or break CI expectations until validated.

Overview
Reduces CircleCI macOS machine usage by merging multiple platform/version test jobs into a few sequential “combo” jobs (e.g. run-test-ios-18-and-17, run-test-ios-15-and-14, run-test-tvos-and-macos, run-revenuecat-ui-ios-18-and-17) and removing now-redundant standalone validate-package-swift/spm-release-build* jobs while keeping an SPM release build + swift package describe at the start of the unified jobs.

Improves CI artifact handling by making compress_result_bundle safe when the .xcresult directory is missing, cleaning fastlane/test_output between sequential runs, and preserving per-OS-version .xcresult tarballs with distinct names/artifact destinations.

Adds new tvOS 18 snapshot baselines under Tests/UnitTests/**/__Snapshots__ to support tvOS snapshot generation in CI.

^{Written by Cursor Bugbot for commit 671831f. This will update automatically on new commits. Configure here.}

ajpallares · 2026-02-23T14:59:39Z

.circleci/config.yml

-  spm-release-build-xcode-14:
-    executor:
-      name: macos-executor
-      xcode_version: "14.3.1"
-
-    steps:
-      - checkout
-      - run:
-          name: SPM Release Build
-          command: swift build -c release --target RevenueCat
-          no_output_timeout: 30m
-      - slack-notify-on-fail


This is now part of the new run-test-ios-15-and-14 job

ajpallares · 2026-02-23T15:00:48Z

.circleci/config.yml

-      - run:
-          name: SPM Release Build
-          command: swift build -c release --target RevenueCat
-          no_output_timeout: 30m


This step is now part of the new run-test-ios-18-and-17 job

ajpallares · 2026-02-23T15:01:13Z

.circleci/config.yml

-      - run:
-          name: SPM RevenueCatUI Release Build
-          command: swift build -c release --target RevenueCatUI
-          no_output_timeout: 30m


This step is now part of the new run-revenuecat-ui-ios-18-and-17 job

Merge several pairs of CI jobs that run on the same Xcode version into single jobs that execute tests sequentially, reducing the total number of macOS machines needed per CI run. Unified jobs: - run-test-ios-14-and-15 (Xcode 14.3.1, includes background iOS 14 sim install) - run-test-ios-16-and-17 (Xcode 15.4.0) - run-test-ios-18-and-26 (Xcode 26.3) - run-test-tvos-and-macos (Xcode 16.4) - run-revenuecat-ui-ios-17-and-18 (Xcode 16.4) Each unified job also runs an SPM release build as a preceding step, absorbing the standalone spm-release-build jobs. Other changes: - compress_result_bundle guards against missing .xcresult directories - push_snapshot_pr in Fastfile saves/restores the original branch Co-authored-by: Cursor <cursoragent@cursor.com>

PR #6303 updated Package.swift to use exact: "1.18.9" but the Tuist/Package.swift was left on the old revision pin. Co-authored-by: Cursor <cursoragent@cursor.com>

…ify-some-xcode-14-jobs

…719) iOS 18 SK1 tests fail under Xcode 26 due to an Apple bug where SKProductsRequest cannot parse the price field (missingValue for StoreKit.ProductResponse.Key.price). Restructure jobs so iOS 18 runs under Xcode 16 instead: - run-test-ios-26: standalone on Xcode 26 (iOS 26 only) - run-test-ios-17-and-18: Xcode 16.4 (iOS 17 + 18) - run-test-ios-14-15-and-16: Xcode 14.3.1 (iOS 14 + 15 + 16) Co-authored-by: Cursor <cursoragent@cursor.com>

Rename and reorder unified jobs so the highest iOS version runs first: - run-test-ios-18-and-17 (was 17-and-18) - run-test-ios-16-15-and-14 (was 14-15-and-16) - run-revenuecat-ui-ios-18-and-17 (was 17-and-18) iOS 14 still runs last since its runtime needs to be downloaded first. Co-authored-by: Cursor <cursoragent@cursor.com>

… exception-safe - Remove test-all-unified workflow and action parameter (was temporary) - Wrap push_snapshot_pr git operations in begin/ensure to guarantee the original branch is restored even if push or PR creation fails Co-authored-by: Cursor <cursoragent@cursor.com>

The standalone spm-release-build-xcode-15 was removed during unification but no other Xcode 15 job picked up the RevenueCat release build. Add it to the only remaining Xcode 15 job. Co-authored-by: Cursor <cursoragent@cursor.com>

…e-swift with swift package describe - Extract iOS 16 tests from run-test-ios-16-15-and-14 into a standalone run-test-ios-16 job on Xcode 15.4.0 with SPM release build - Rename remaining job to run-test-ios-15-and-14 (Xcode 14.3.1) - Add `swift package describe` as first step in all run-test jobs, making the validate-package-swift job redundant - Remove validate-package-swift job and all workflow references - Remove SPM RevenueCat build from spm-revenuecat-ui-ios-16 (now covered by run-test-ios-16) Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

The iOS test jobs already cover all 4 Xcode versions (14, 15, 16, 26), so tvOS, macOS and watchOS jobs don't need to duplicate the validation. Co-authored-by: Cursor <cursoragent@cursor.com>

…ots workflow Co-authored-by: Cursor <cursoragent@cursor.com>

The old pattern `../*/__Snapshots__/*` only matched one directory level on each side of __Snapshots__, but snapshot files are 3+ levels deep (e.g. Tests/UnitTests/Misc/__Snapshots__/AnyEncodableTests/file.json). Use `**` for recursive matching in the git pathspec. Co-authored-by: Cursor <cursoragent@cursor.com>

…ts__ dirs" This reverts commit 7cf475f.

…sage snapshots Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares · 2026-02-24T08:55:47Z

.circleci/config.yml

-  spm-release-build-xcode-15:
-    executor:
-      name: macos-executor
-      xcode_version: "15.4.0"
-
-    steps:
-      - checkout
-      - run:
-          name: SPM Release Build
-          command: swift build -c release --target RevenueCat
-          no_output_timeout: 30m
-      - slack-notify-on-fail


This is now part of the run-test-ios-16 job

tvOS was falling through to the #else branch in osVersionAndTestName, generating snapshots with the iOS prefix (e.g. iOS18-testFoo.1.json). This meant tvOS snapshots were indistinguishable from iOS snapshots. Add a #elseif os(tvOS) branch to produce tvOS-prefixed snapshot files (e.g. tvOS18-testFoo.1.json), enabling independent tvOS snapshot generation and validation. Co-authored-by: Cursor <cursoragent@cursor.com>

…tion" This reverts commit c06e438.

…ation" This reverts commit c486f7f.

ajpallares · 2026-02-24T09:41:19Z

.circleci/config.yml

-  validate-package-swift:
-    parameters:
-      xcode_version:
-        type: string
-      mac_os_executor_name:
-        type: string
-    executor:
-      name: << parameters.mac_os_executor_name >>
-      xcode_version: << parameters.xcode_version >>
-    steps:
-      - checkout
-      - run:
-          name: Validate Package.swift
-          command: swift package describe


Added as part of each of the run-test-ios-* jobs, so now this is run on every Xcode version

ajpallares · 2026-02-24T09:43:16Z

.circleci/config.yml

+      - create-snapshot-pr-if-needed:
+          condition: << pipeline.parameters.generate_snapshots >>
+          job: "create_snapshot_pr"
+          version: "tvos"


This is new to this PR. Now we'll have tvOS-specific snapshot generation as well

ajpallares · 2026-02-24T09:55:21Z

fastlane/Fastfile

+        )
+      end
+    ensure
+      sh("git", "checkout", original_branch)


Since we're now potentially running multiple tests in the same job, now push_snapshot_pr checkouts the original branch

RevenueCat-Danger-Bot · 2026-02-24T11:37:09Z

	1 Warning
⚠️	This PR increases the size of the repo by more than 100.00 KB (increased by 184.57 KB).

Generated by 🚫 Danger

tonidero

I think this looks great! Let's hope this improves CI 🙏

tonidero · 2026-02-25T08:35:55Z

.circleci/config.yml

+          version: "revenuecatui-18"
      - create-snapshot-pr-if-needed:
          condition: << pipeline.parameters.generate_revenuecatui_snapshots >>
          job: "create_snapshots_repo_pr"


I would say, these 2 steps are quite confusing, one executing the create_snapshot_pr and this one create_snapshots_repo_pr... Might be worth documenting the difference?

Not a problem from this PR I would say though.

Yep. 100% agree

tonidero · 2026-02-25T08:39:55Z

.circleci/config.yml

+      # === iOS 17 tests ===
      - run:
-          name: Run tests
+          name: Run iOS 17 tests


I guess this won't be executed if the iOS 18 tests fail right? I was wondering whether it was worth to execute them anyways... But after thinking about it, it would complicate things and make the results harder to read IMO... So I think it's ok as it is :)

Right. Actually this exact thing I'll tackle separately in #6333

tonidero · 2026-02-25T08:40:49Z

.circleci/config.yml

+              fi
+            } &
+            disown
+            echo "Started iOS 14.5 runtime installation in background"


I do wonder if we should try to move most of this logic to fastlane... But I'm ok keeping it here for now.

Right. I see the point. I'll check how to achieve that

I think I'm going to keep this for now, as it deals with the background process and it's mostly CI orchestration rather than test execution/artifact handling, which should be in fastlane grounds. Happy to discuss further or to iterate though!

.circleci/config.yml

The existing no_output_timeout doesn't apply because the loop prints periodic "Still waiting..." messages. Add an explicit elapsed-time check so the step fails cleanly instead of waiting indefinitely. Co-authored-by: Cursor <cursoragent@cursor.com>

The loop prints output every 10 seconds, so no_output_timeout never triggers. The explicit 30-minute elapsed-time check is sufficient. Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares added the pr:other label Feb 23, 2026

RevenueCat deleted a comment from RevenueCat-Danger-Bot Feb 23, 2026

ajpallares changed the title ~~CI: Unify run-test-ios-14 and run-test-ios-15 into a single job~~ CI: Unify CI jobs to reduce machine count and improve retry efficiency Feb 23, 2026

ajpallares force-pushed the pallares/ci/unify-some-xcode-14-jobs branch from c15cc7a to bb603a5 Compare February 23, 2026 14:30

ajpallares changed the title ~~CI: Unify CI jobs to reduce machine count and improve retry efficiency~~ CI: Unify CI jobs to reduce machine count Feb 23, 2026

ajpallares mentioned this pull request Feb 23, 2026

CI: Add test-run caching for unified job retry optimization #6333

Draft

1 task

ajpallares commented Feb 23, 2026

View reviewed changes

ajpallares force-pushed the pallares/ci/unify-some-xcode-14-jobs branch from bb603a5 to fed250a Compare February 23, 2026 15:06

Update Tuist swift-snapshot-testing dependency to match Package.swift

b87f6aa

PR #6303 updated Package.swift to use exact: "1.18.9" but the Tuist/Package.swift was left on the old revision pin. Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares changed the base branch from main to ajp/fix-tuist-snapshot-testing-dep February 23, 2026 15:25

Merge branch 'ajp/fix-tuist-snapshot-testing-dep' into pallares/ci/un…

a858553

…ify-some-xcode-14-jobs

Base automatically changed from ajp/fix-tuist-snapshot-testing-dep to main February 23, 2026 15:29

ajpallares and others added 7 commits February 23, 2026 18:27

Add missing swift package describe to run-test-tvos-and-macos

bc5933b

Co-authored-by: Cursor <cursoragent@cursor.com>

Limit swift package describe to run-test-ios jobs only

679e14e

The iOS test jobs already cover all 4 Xcode versions (14, 15, 16, 26), so tvOS, macOS and watchOS jobs don't need to duplicate the validation. Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares force-pushed the pallares/ci/unify-some-xcode-14-jobs branch from 8f14708 to 679e14e Compare February 23, 2026 19:13

ajpallares and others added 4 commits February 23, 2026 20:14

[TEMP] Corrupt testNestedDictionary snapshots to test generate_snapsh…

7a695c0

…ots workflow Co-authored-by: Cursor <cursoragent@cursor.com>

Revert "Fix snapshot PR git add glob to match deeply nested __Snapsho…

feb9c73

…ts__ dirs" This reverts commit 7cf475f.

[TEMP] Remove X-Retry-Count from some testRepeatedRequestsLogDebugMes…

3fef904

…sage snapshots Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares force-pushed the pallares/ci/unify-some-xcode-14-jobs branch from 2a39c12 to 3fef904 Compare February 24, 2026 06:55

Merge branch 'main' into pallares/ci/unify-some-xcode-14-jobs

cc0dbf6

ajpallares commented Feb 24, 2026

View reviewed changes

ajpallares and others added 2 commits February 24, 2026 10:25

TEMP: only run run-test-tvos-and-macos for snapshots generation

c06e438

ajpallares mentioned this pull request Feb 24, 2026

Generating new test snapshots for pallares/ci/unify-some-xcode-14-jobs - macos #6350

Closed

Revert "TEMP: only run run-test-tvos-and-macos for snapshots genera…

519b7b4

…tion" This reverts commit c06e438.

ajpallares mentioned this pull request Feb 24, 2026

Generating new test snapshots for pallares/ci/unify-some-xcode-14-jobs - macos #6352

Closed

ajpallares and others added 2 commits February 24, 2026 10:40

Revert "TEMP: corrupt some macOS snapshots to test the snapshot gener…

de9f9df

…ation" This reverts commit c486f7f.

[skip ci] Generating new test snapshots (#6351)

58c8bc1

ajpallares commented Feb 24, 2026

View reviewed changes

ajpallares marked this pull request as ready for review February 24, 2026 12:43

ajpallares requested a review from a team as a code owner February 24, 2026 12:43

tonidero approved these changes Feb 25, 2026

View reviewed changes

ajpallares and others added 3 commits February 25, 2026 18:33

Merge branch 'main' into pallares/ci/unify-some-xcode-14-jobs

10d90ed

Remove redundant no_output_timeout from iOS 14 runtime wait step

671831f

The loop prints output every 10 seconds, so no_output_timeout never triggers. The explicit 30-minute elapsed-time check is sufficient. Co-authored-by: Cursor <cursoragent@cursor.com>

ajpallares enabled auto-merge (squash) February 25, 2026 17:45

ajpallares merged commit 8aa9344 into main Feb 25, 2026
32 of 37 checks passed

ajpallares deleted the pallares/ci/unify-some-xcode-14-jobs branch February 25, 2026 18:09

ajpallares mentioned this pull request Feb 27, 2026

AdMob adapter #6278

Merged

2 tasks

This was referenced Mar 4, 2026

[AUTOMATIC] Release/5.60.1 #6401

Closed

[AUTOMATIC] Release/5.60.1 #6403

Closed

[AUTOMATIC] Release/5.61.0 #6410

Closed

[AUTOMATIC] Release/5.61.0 #6414

Merged

Conversation

ajpallares commented Feb 23, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Motivation

Description

Known tradeoffs

Diff

Uh oh!

ajpallares Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajpallares Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajpallares Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RevenueCat-Danger-Bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonidero left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ajpallares commented Feb 23, 2026 •

edited by cursor bot

Loading

ajpallares Feb 23, 2026 •

edited

Loading

ajpallares Feb 23, 2026 •

edited

Loading

ajpallares Feb 23, 2026 •

edited

Loading

RevenueCat-Danger-Bot commented Feb 24, 2026 •

edited

Loading