Skip to content

fix(add-portal): keep generated tests offline in CI - #714

Merged
MadsLorentzen merged 3 commits into
ai-job-search-dev:masterfrom
rudycelekli:fix/portal-offline-tests
Nov 9, 2026
Merged

MadsLorentzen merged 3 commits into
ai-job-search-dev:masterfrom
rudycelekli:fix/portal-offline-tests

Conversation

@rudycelekli

@rudycelekli rudycelekli commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

/add-portal Step 3 tells the generator to create a live portal smoke test. Step 5 then registers that CLI for automatic CI, whose discovery job runs every *.test.ts through bun test on each push. This contradicts the CI workflow's explicit rule against live portal requests.

Generate fixture/mock tests using captured responses and the shipped local HTTP/fetch-preload pattern instead. Keep Step 4's mandatory, bounded live search and detail checks as manual registration checks.

Failing case / reproduction (for fixes)

Follow the persisted Step 3 instruction to put a live search test under .agents/skills/<name>/cli/tests/, then follow Step 5 registration: .github/workflows/ci.yml discovers its package and executes the live test. The revised instructions separate those paths; no live portal was contacted from CI to demonstrate the conflict.

Verification

Existing /add-portal command test, lint, framework-version and security guards passed; git diff --check clean. Python 3.13.5 with openpyxl 3.1.5: 800 tests OK (6 optional-dependency skips). No workflow, permissions or portal implementation changed.

The source, specification and verification evidence were reviewed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 issues found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="CHANGELOG.md">

<violation number="1" location="CHANGELOG.md:3">
P2: This PR adds about 27 entries for unrelated changes to a file its own notice says is frozen. Those entries belong in their own PRs, or this file should not be edited here.</violation>

<violation number="2" location="CHANGELOG.md:39">
P2: This release note promises an automatic weekly release workflow, but HEAD contains none of the referenced workflow, Commitizen config, or release-workflow test. Add the implementation or remove the claim; otherwise no process creates the releases described here.</violation>

<violation number="3" location="CHANGELOG.md:47">
P2: This migration note is false in the checked-in tree: README, security/setup docs, CI, and upstream tests still contain `MadsLorentzen/ai-job-search`. Update those references or qualify the note, because users and CI still consume the old slug.</violation>

<violation number="4" location="CHANGELOG.md:56">
P2: `/scrape` cannot have the behavior described here because the referenced `tools/scrape_state.py` and `tests/test_scrape_state.py` are absent from HEAD. Ship the helper and its integration or remove this release entry.</violation>

<violation number="5" location="CHANGELOG.md:70">
P2: The entry claims each CLI pins `@types/node` 26.6.5, but `jobbank-search/cli/package.json` does not declare it. Correct the claim, or add the pin to the manifests before shipping this note.</violation>

<violation number="6" location="CHANGELOG.md:307">
P2: This entry describes a User-Agent that the code does not send. `helpers.ts` still uses the `MadsLorentzen` UA, so restore the entry to match the code until the code changes.</violation>

<violation number="7" location="CHANGELOG.md:362">
P2: The changelog claims a `framework_version` bump in `09-web-research.md` that is not in the tree. Either bump the marker in that file in this PR or correct the entry.</violation>
</file>

<file name=".claude/commands/add-portal.md">

<violation number="1" location=".claude/commands/add-portal.md:134">
P3: CI runs `bun test` directly, not the package's `test` script, so the `--timeout 30000` flag from `package.json` does not apply in CI. Reword to say CI runs the CLI's test files with `bun test`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread CHANGELOG.md
installed portal instead of getting the `⚠ Unverified source host` warning. The User-Agent keeps
the `owner/project/version` name freehire's API docs ask for and drops the optional URL:
`MadsLorentzen/ai-job-search/1.0 (freehire-search)`. The host-check test now lists `github.com` and
`ai-job-search-dev/ai-job-search/1.0 (freehire-search)`. The host-check test now lists `github.com` and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This entry describes a User-Agent that the code does not send. helpers.ts still uses the MadsLorentzen UA, so restore the entry to match the code until the code changes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 307:

<comment>This entry describes a User-Agent that the code does not send. `helpers.ts` still uses the `MadsLorentzen` UA, so restore the entry to match the code until the code changes.</comment>

<file context>
@@ -260,7 +304,7 @@ per-file diff commands.
   installed portal instead of getting the `⚠ Unverified source host` warning. The User-Agent keeps
   the `owner/project/version` name freehire's API docs ask for and drops the optional URL:
-  `MadsLorentzen/ai-job-search/1.0 (freehire-search)`. The host-check test now lists `github.com` and
+  `ai-job-search-dev/ai-job-search/1.0 (freehire-search)`. The host-check test now lists `github.com` and
   `gist.github.com` postings among the hosts that must stay unverified.
 
</file context>

Comment thread CHANGELOG.md
All notable changes to this project are documented here. The format is based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres
to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
**This file is frozen at 1.7.4.** Later releases are published weekly and documented in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This PR adds about 27 entries for unrelated changes to a file its own notice says is frozen. Those entries belong in their own PRs, or this file should not be edited here.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 3:

<comment>This PR adds about 27 entries for unrelated changes to a file its own notice says is frozen. Those entries belong in their own PRs, or this file should not be edited here.</comment>

<file context>
@@ -1,17 +1,20 @@
-All notable changes to this project are documented here. The format is based on
-[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres
-to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+**This file is frozen at 1.7.4.** Later releases are published weekly and documented in
+[GitHub Releases](https://github.com/ai-job-search-dev/ai-job-search/releases), with notes
+generated from the titles of the PRs each release contains. Don't add entries here.
</file context>

Comment thread CHANGELOG.md

- **`fetch_page.py` checks `robots.txt` before every redirect it follows** (#680, `tools/fetch_page.py`,
`tests/test_fetch_page.py`, `SECURITY.md`, `tests/test_posting_gate_wiring.py`,
`.claude/skills/job-application-assistant/09-web-research.md` `framework_version: 1.1.6 -> 1.1.7`) -

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The changelog claims a framework_version bump in 09-web-research.md that is not in the tree. Either bump the marker in that file in this PR or correct the entry.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 362:

<comment>The changelog claims a `framework_version` bump in `09-web-research.md` that is not in the tree. Either bump the marker in that file in this PR or correct the entry.</comment>

<file context>
@@ -313,6 +357,127 @@ per-file diff commands.
 
+- **`fetch_page.py` checks `robots.txt` before every redirect it follows** (#680, `tools/fetch_page.py`,
+  `tests/test_fetch_page.py`, `SECURITY.md`, `tests/test_posting_gate_wiring.py`,
+  `.claude/skills/job-application-assistant/09-web-research.md` `framework_version: 1.1.6 -> 1.1.7`) -
+  the gate ran once, on the URL it was given, and `curl -L` or urllib's redirect handler then followed
+  up to five redirects unchecked, so an allowed posting URL that redirected to a path the site's
</file context>

Comment thread CHANGELOG.md

### Fixed

- **The four `@bunli/core` portal CLIs typecheck on a clean install again** (#768, closes #765,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The entry claims each CLI pins @types/node 26.6.5, but jobbank-search/cli/package.json does not declare it. Correct the claim, or add the pin to the manifests before shipping this note.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 70:

<comment>The entry claims each CLI pins `@types/node` 26.6.5, but `jobbank-search/cli/package.json` does not declare it. Correct the claim, or add the pin to the manifests before shipping this note.</comment>

<file context>
@@ -25,15 +28,56 @@ per-file diff commands.
+
 ### Fixed
 
+- **The four `@bunli/core` portal CLIs typecheck on a clean install again** (#768, closes #765,
+  `.agents/skills/{jobbank,jobdanmark,jobindex,jobnet}-search/cli/package.json`,
+  `tests/test_portal_cli_types.py`) - `bun-types` accepts any `@types/node`, while `@bunli/core`
</file context>

Comment thread CHANGELOG.md
existing clones and forks keep working; update your `upstream` remote when convenient:
`git remote set-url upstream https://github.com/ai-job-search-dev/ai-job-search.git`. Every
in-tree reference (README, CONTRIBUTING, SECURITY, the upstream-check tool and its tests, the
workflows' upstream guard) now points at the new slug.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This migration note is false in the checked-in tree: README, security/setup docs, CI, and upstream tests still contain MadsLorentzen/ai-job-search. Update those references or qualify the note, because users and CI still consume the old slug.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 47:

<comment>This migration note is false in the checked-in tree: README, security/setup docs, CI, and upstream tests still contain `MadsLorentzen/ai-job-search`. Update those references or qualify the note, because users and CI still consume the old slug.</comment>

<file context>
@@ -25,15 +28,56 @@ per-file diff commands.
+  existing clones and forks keep working; update your `upstream` remote when convenient:
+  `git remote set-url upstream https://github.com/ai-job-search-dev/ai-job-search.git`. Every
+  in-tree reference (README, CONTRIBUTING, SECURITY, the upstream-check tool and its tests, the
+  workflows' upstream guard) now points at the new slug.
+
 - **`freehire-search` documents the `OR` and `title:` query operators** (#674,
</file context>

Comment thread CHANGELOG.md
or `-q "golang rust"`, which returns fewer jobs than `golang` alone. Measured live: `golang`
3,550, `golang OR rust` 6,805. The CLI already passes `-q` through unchanged.

- **`/scrape` drops already-seen hits before fetching details and keeps its history out of the conversation**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: /scrape cannot have the behavior described here because the referenced tools/scrape_state.py and tests/test_scrape_state.py are absent from HEAD. Ship the helper and its integration or remove this release entry.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 56:

<comment>`/scrape` cannot have the behavior described here because the referenced `tools/scrape_state.py` and `tests/test_scrape_state.py` are absent from HEAD. Ship the helper and its integration or remove this release entry.</comment>

<file context>
@@ -25,15 +28,56 @@ per-file diff commands.
   or `-q "golang rust"`, which returns fewer jobs than `golang` alone. Measured live: `golang`
   3,550, `golang OR rust` 6,805. The CLI already passes `-q` through unchanged.
 
+- **`/scrape` drops already-seen hits before fetching details and keeps its history out of the conversation**
+  (#710, `tools/scrape_state.py`, `tests/test_scrape_state.py`, `.claude/skills/job-scraper/SKILL.md`,
+  `.claude/settings.json`, `tools/security_guards.py`) - previously `/scrape` read the whole
</file context>

Comment thread CHANGELOG.md
commitizen from the merged commit types: `type!:` or a `BREAKING CHANGE:` footer is major, `feat` is
minor, `fix`, `refactor` and `perf` are patch. A week with nothing to bump is skipped, and a
`master` commit whose CI failed is never released. Forks keep updating to tags as before; release
notes now live in GitHub Releases, and a test fails any PR that still edits this file.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This release note promises an automatic weekly release workflow, but HEAD contains none of the referenced workflow, Commitizen config, or release-workflow test. Add the implementation or remove the claim; otherwise no process creates the releases described here.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 39:

<comment>This release note promises an automatic weekly release workflow, but HEAD contains none of the referenced workflow, Commitizen config, or release-workflow test. Add the implementation or remove the claim; otherwise no process creates the releases described here.</comment>

<file context>
@@ -25,15 +28,56 @@ per-file diff commands.
+  commitizen from the merged commit types: `type!:` or a `BREAKING CHANGE:` footer is major, `feat` is
+  minor, `fix`, `refactor` and `perf` are patch. A week with nothing to bump is skipped, and a
+  `master` commit whose CI failed is never released. Forks keep updating to tags as before; release
+  notes now live in GitHub Releases, and a test fails any PR that still edits this file.
+
+- **The repository moved to the `ai-job-search-dev` organization** - the canonical URL is now
</file context>

(Skip if the skill is zero-dependency and they don't care about typecheck types.)
3. Note that the skill auto-triggers from its `SKILL.md` description - no other wiring is needed.
4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` and `test` scripts run on every push to the fork without editing the workflow.
4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` and network-free `test` scripts run on every push to the fork without editing the workflow. CI must not contact the live portal, including when personal use was approved in Step 2.4.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: CI runs bun test directly, not the package's test script, so the --timeout 30000 flag from package.json does not apply in CI. Reword to say CI runs the CLI's test files with bun test.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .claude/commands/add-portal.md, line 134:

<comment>CI runs `bun test` directly, not the package's `test` script, so the `--timeout 30000` flag from `package.json` does not apply in CI. Reword to say CI runs the CLI's test files with `bun test`.</comment>

<file context>
@@ -131,7 +131,7 @@ Do not proceed to Step 5 until search, detail, and tests all pass.
    (Skip if the skill is zero-dependency and they don't care about typecheck types.)
 3. Note that the skill auto-triggers from its `SKILL.md` description - no other wiring is needed.
-4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` and `test` scripts run on every push to the fork without editing the workflow.
+4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` and network-free `test` scripts run on every push to the fork without editing the workflow. CI must not contact the live portal, including when personal use was approved in Step 2.4.
 
 ---
</file context>
Suggested change
4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` and network-free `test` scripts run on every push to the fork without editing the workflow. CI must not contact the live portal, including when personal use was approved in Step 2.4.
4. CI coverage is also automatic: the `cli-checks` job discovers every `.agents/skills/*/cli/package.json`, so the new CLI's `typecheck` runs and its network-free tests run via `bun test` on every push to the fork without editing the workflow. CI must not contact the live portal, including when personal use was approved in Step 2.4.

@MadsLorentzen MadsLorentzen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Real problem with real evidence: every portal that arrived in community-portals this week had to replace the live smoke tests /add-portal told its author to write. I dropped the CHANGELOG hunk on your branch since the file is frozen by #800.

@MadsLorentzen
MadsLorentzen merged commit e9ab29d into ai-job-search-dev:master Nov 9, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants