[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-16

[github-actions[bot]]

Summary

• Date: 2026-05-16
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The home page has a strong value proposition and the Quick Start is well-structured with clear steps and time estimates. However, several technical terms are used without explanation and the CLI reference page is overwhelming for beginners.

Note: Screenshots could not be captured due to a network firewall in the test environment preventing the Playwright browser from reaching localhost:4321. Analysis was performed via direct HTTP content inspection.

---

🔴 Critical Issues Found

1. Jargon used before definition: frontmatter

Page: Quick Start — Step 4 (Customize Your Workflow)

The word "frontmatter" appears without any definition or link to a glossary:

"Within the .md file, the frontmatter (the configuration block between the --- markers at the top)..."

While there is an inline parenthetical, it comes after multiple prior uses of the term. A beginner will wonder: "What is frontmatter? Is this a GitHub thing? A Markdown thing?"

Recommendation: Add a tooltip, inline definition, or link to the Glossary on first use, e.g.: the [frontmatter](/gh-aw/reference/glossary/#frontmatter) (the YAML config block at the top of the file).

---

2. Two-file system (.md + .lock.yml) is confusing

Page: Quick Start — Step 4

The note box explains:

"Workflows have two files: the .md file you edit and a generated .lock.yml file that GitHub Actions runs. gh aw compile regenerates the .lock.yml file..."

A noob sees two files appear in their repo and doesn't understand why. The relationship between them is non-obvious, and the note appears only in Step 4. When the add-wizard command in Step 2 adds both files, the user already has this question.

Recommendation: Add a brief callout box in Step 2 when files are first added: "Two files will be added: daily-repo-status.md (what you edit) and daily-repo-status.lock.yml (auto-generated — don't edit this)."

---

3. No visual preview of the add-wizard interactive process

Page: Quick Start — Step 2

The guide says add-wizard "takes you through an interactive process" but shows no terminal output, screenshot, or ASCII art of what the wizard actually looks like. A first-time user has no idea what to expect when they run the command.

Recommendation: Add a short terminal recording or a screenshot showing the wizard prompts (engine selection, secret setup, etc.).

---

🟡 Confusing Areas

4. Home page has two equally prominent CTAs

Page: Home

The hero section has two buttons: "Quick Start with CLI" and "Creating Workflows". A true beginner doesn't know which to click. "Creating Workflows" sounds like the natural first step.

Recommendation: Make "Quick Start with CLI" visually more prominent (it already has variant: primary) and add a subtle sub-label: "Start here if you're new". Or add a short sentence: "New here? Start with the Quick Start."

---

5. gh auth login --scopes repo,workflow — "scopes" unexplained

Page: Quick Start — Prerequisites

The prerequisite says:

"run gh auth login --scopes repo,workflow if needed"

A beginner doesn't know what "scopes" are. They may have already run gh auth login and be confused why they need to add scopes.

Recommendation: Add a brief explanation: "(scopes are the permissions granted to the CLI — repo and workflow are needed to push workflow files)"

---

6. add-wizard reference format is cryptic

Page: Quick Start — Step 2

"add-wizard accepts workflow references in <owner>/<repo>/<workflow-name> format"

The <owner>/<repo>/<workflow-name> format is common to developers but a beginner might ask: "Is githubnext/agentics/daily-repo-status a real thing I need to change?"

Recommendation: Add: "In this tutorial, use the command exactly as shown — githubnext/agentics/daily-repo-status is an official sample workflow."

---

7. CLI page is overwhelming for beginners

Page: CLI Commands (/gh-aw/setup/cli/)

The page lists 30+ commands in a flat list. A beginner landing here feels immediately overwhelmed. The "Most Common Commands" table at the top is helpful, but even that table uses unexplained terms (init vs add-wizard — what's the difference?).

Recommendation:

• Add a "Beginner path" callout at the top: "If you're new, you only need gh aw add-wizard and gh aw compile to get started."
• Expand the "Most Common Commands" table to include a one-line use-case example for each command.

---

8. Search not functional in dev mode

Page: All pages

All pages show the message: "Search is only available in production builds. Try building and previewing the site to test it out locally." This appears in the search bar. While expected during development, any staging/preview deployment would show this, confusing visitors.

Recommendation: Ensure search is enabled in deployed preview builds, or hide the search bar entirely when it's non-functional rather than showing a confusing placeholder message.

---

🟢 What Worked Well

• ✅ Time estimate on Quick Start ("⏱️ Estimated time: 10 minutes") — sets expectations immediately
• ✅ Prerequisites are explicit — lists every required item with links
• ✅ Tip callouts for common problems (authentication issues, secret setup)
• ✅ Links to FAQ and Common Issues at end of Step 2 — exactly where users get stuck
• ✅ "By the Numbers" table on home page gives concrete, scannable info
• ✅ Security architecture prominently featured — builds trust
• ✅ Step-by-step numbered structure — easy to follow
• ✅ The video placeholder in Quick Start — good idea even if the content isn't always accessible

---

Recommendations

Quick Wins (High impact, low effort)

1. Add a link/tooltip for "frontmatter" on first use → links to glossary
2. Add a callout in Step 2 explaining the two-file system before the user is surprised
3. Add a note that githubnext/agentics/daily-repo-status is a sample and doesn't need to be changed
4. Add "Start here if you're new" label to the primary CTA on home page

Medium-term improvements

5. Add a terminal screenshot or GIF showing the add-wizard interactive flow
6. Add a "Beginner path" callout to the CLI Commands page
7. Expand "Most Common Commands" table with use-case examples

Longer-term improvements

8. Add a "Concepts" or "Glossary" page linked prominently from Quick Start covering: frontmatter, lock file, safe outputs, engine, trigger
9. Consider a split CLI page: "Getting Started Commands" vs "Full Reference"

---

References: §25953152721

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · ● 30.3M · ◷

• expires on May 17, 2026, 5:02 AM UTC