[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-15

[github-actions[bot]]

Summary

• Date of test: 2026-05-15
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page pitch is compelling, but as a brand-new user I found several areas where unclear terminology and assumed knowledge would make getting started confusing.

---

🔴 Critical Issues Found

1. "Frontmatter" jargon in Step 4 with no definition

In Step 4 – Customize your workflow, the docs say:

"Within the .md file, the frontmatter (the configuration block between the --- markers at the top) controls when the workflow runs..."

The word frontmatter is used before it's defined. As a first-time user, I had to re-read twice before the parenthetical explanation helped. A tooltip, callout, or an early glossary link would help.

2. Token secret setup instructions are fragmented

The setup process for COPILOT_GITHUB_TOKEN requires you to:

1. Create a fine-grained PAT on GitHub
2. Set Account permissions → Copilot Requests → Read
3. Add as a repository secret via CLI

This is distributed across multiple collapsible notes without a single clear walkthrough. New users easily miss the "Account permissions" distinction vs. repository permissions. A short numbered checklist or a dedicated "Setting up secrets" callout would reduce confusion.

3. Missing "what is a .lock.yml file?" explanation

Step 4 mentions:

"the .md file you edit and a generated .lock.yml file that GitHub Actions runs"

There's no explanation of why there are two files or what the .lock.yml does conceptually. As a beginner, I initially thought I needed to hand-edit the .lock.yml too.

---

🟡 Confusing Areas

1. Home page hero — unclear what "compile" means

The home page tagline says the tool "transforms natural language markdown files into GitHub Actions". The word "compile" appears in the nav but is unexplained on the home page. Beginners unfamiliar with build tools won't know what this implies.

2. Prerequisites list has mixed urgency

The prerequisites list mixes easy items (OS requirement) with hard items (AI account setup, which may require billing). New users might start installing and only discover they need a paid AI API key mid-process. Consider marking prerequisites that require external signup/payment with a 💳 or ⚠️ indicator.

3. add-wizard vs add command distinction unclear

The Quick Start uses gh aw add-wizard. The CLI Commands page also shows gh aw add. A beginner doesn't know when to use which — the difference (interactive vs non-interactive) is only explained on the CLI Commands page, not in the Quick Start.

4. CLI Commands page "Most Common Commands" table

The table is great, but new users don't know which command to run first. The table would benefit from a "Start here →" marker pointing to gh aw init or gh aw add-wizard for first-time users.

5. OS requirement: "Windows with WSL" — no WSL setup link

The prerequisites list mentions "Windows with WSL" but doesn't link to WSL setup instructions. Windows users unfamiliar with WSL will be stuck.

---

🟢 What Worked Well

• Clear time estimate ("⏱️ Estimated time: 10 minutes") on the Quick Start — sets good expectations
• Video at the top of Quick Start is excellent for visual learners
• Most Common Commands table on CLI page — scanning this immediately gives a mental model
• AI engine choice is well documented — all four options (Copilot, Claude, Codex, Gemini) are listed upfront
• Tip callouts for authentication issues are helpful and well-placed
• Navigation structure is logical: Quick Start → Create → Docs flow makes sense

---

Recommendations

Quick wins 🎯

1. Add a brief glossary callout or tooltip for "frontmatter" on first use
2. Link "Windows with WSL" to the Microsoft WSL setup guide
3. Add "→ Start here if you're new" marker to add-wizard in the CLI Commands table
4. Add a one-sentence explanation of the .md + .lock.yml two-file system earlier in Step 2

Longer-term 🏗️

1. Create a "Setting up secrets" dedicated page that walks through each AI provider step-by-step with screenshots
2. Add a "What is GitHub Agentic Workflows?" one-pager with a 60-second conceptual overview before the Quick Start
3. Consider a "Prerequisites checker" section (e.g., gh aw doctor command mention) so users can verify their setup is ready before proceeding

---

Screenshots

📎 home.png — Home page

📎 quick-start.png — Quick Start page (full page)

📎 quick-start-prerequisites.png — Prerequisites section (viewport)

📎 cli.png — CLI Commands page (full page)

---

References:

• §25901179938

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · ● 35.8M · ◷

• expires on May 16, 2026, 5:15 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-16T05:15:42.978Z.

Closed by Workflow