[experiments] Daily Experiment Report — 2026-05-16

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-05-16

11 experiments analysed across 11 workflows. 0 reached statistical significance. All recommendations: EXTEND — most experiments are still in early data-collection phase. No experiments have reached their minimum sample size.

---

output_format · daily-issues-report

Variants: collapsible vs inline · Window: last 30 runs · Analysed: 10 runs with assignments
min_samples: 30 per variant

H0: no change in discussion engagement score. H1: inline format produces ≥20% higher reactions+replies by making charts and recommendations immediately visible

Experiment : output_format
Workflow   : daily-issues-report
Hypothesis : H0: no change in engagement. H1: inline format ≥20% higher reactions+replies
Window     : last 30 runs  |  Analysed: 10 runs with assignments
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
| collapsible      |   4  |   0.0%   |    323.4       | [275.3 , 371.4]    |  (ref)    |  4/30 (13%)   |
| inline           |   6  |   0.0%   |    390.3       | [243.3 , 537.3]    |   N/A     |  6/30 (20%)   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
Note: p-value N/A — zero variance in success rates (all failures); no discriminating signal yet.

Guardrails:
  empty_output_rate ==0 : UNKNOWN (no success data in sampled runs)

Recommendation: EXTEND
Rationale     : Need 30 samples per variant; currently at 4 collapsible / 6 inline (all failed)

Recommendation: EXTEND — All 10 analysed runs failed (workflow-level failures unrelated to variant). Need 30 per variant.

---

prompt_style · daily-astrostylelite-markdown-spellcheck

Variants: concise vs detailed · Window: last 30 runs · Analysed: 11 runs with assignments
min_samples: 30 per variant

Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck
Hypothesis : Concise prompt reduces token consumption ≥20% without degrading fix precision
Window     : last 30 runs  |  Analysed: 11 runs with assignments
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
| concise          |   4  | 100.0%   |    447.6       | [358.4 , 536.7]    |  (ref)    |  4/30 (13%)   |
| detailed         |   7  |  85.7%   |    394.7       | [286.2 , 503.2]    |  0.4279   |  7/30 (23%)   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001
p-value is two-tailed vs. control variant (concise).

Guardrails:
  empty_output_rate <0.10 : UNKNOWN (insufficient data)
  run_success_rate >=0.90 : concise=PASS(1.00), detailed=FAIL(0.86) — provisional, n<30

Recommendation: EXTEND
Rationale     : p=0.43 (not significant); concise n=4, detailed n=7 — far from min_samples=30

Recommendation: EXTEND — 11 runs total; p=0.43 not significant. Run for several more weeks.

---

output_format · deep-report

Variants: full_briefing vs executive_brief vs annotated_brief · Window: last 30 runs · Analysed: 8 runs with assignments
min_samples: 15 per variant

H0: no change in discussion engagement or token cost. H1: executive_brief reduces token usage by ≥20%.

Experiment : output_format
Workflow   : deep-report
Hypothesis : H0: no change in engagement. H1: executive_brief reduces tokens ≥20%
Window     : last 30 runs  |  Analysed: 8 runs with assignments
min_samples: 15 per variant

+------------------+------+----------+----------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
| full_briefing    |   4  | 100.0%   |    820.1       | [624.3 ,1015.8]    |  (ref)    |  4/15 (27%)   |
| executive_brief  |   2  | 100.0%   |    761.9       | [N/A   , N/A  ]    |   N/A     |  2/15 (13%)   |
| annotated_brief  |   2  |  50.0%   |   1016.7       | [763.5 ,1269.8]    |  0.1213   |  2/15 (13%)   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Guardrails:
  empty_output_rate ==0      : UNKNOWN
  issue_creation_success_rate >=0.8 : UNKNOWN

Recommendation: EXTEND
Rationale     : executive_brief n=2, annotated_brief n=2 — below min_samples=15; p=0.12 for annotated_brief

Recommendation: EXTEND — 3-way factorial; executive_brief and annotated_brief each have only 2 runs.

---

View Remaining Experiments (early-stage, <5 runs/variant)

prompt_style · daily-news

Variants: detailed vs concise · 5 total runs · min_samples: 30

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| detailed         |   3  |   0.0%   |    300.8       |  (ref)    |  3/30 (10%)   |
| concise          |   2  |   0.0%   |    335.8       |   N/A     |  2/30 (7%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — all sampled runs failed; need 30/variant

reasoning_depth · daily-fact

Variants: single_pass vs multi_candidate · 3 total runs (single_pass only) · min_samples: 30

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| single_pass      |   3  |   0.0%   |    233.5       |  (ref)    |  3/30 (10%)   |
| multi_candidate  |   0  |   N/A    |    N/A         |   N/A     |  0/30 (0%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — multi_candidate not yet assigned; imbalanced assignment

prompt_style · issue-arborist

Variants: concise vs detailed · 4 total runs · min_samples: 30 · (workflow disabled)

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| concise          |   2  |   0.0%   |    185.4       |  (ref)    |  2/30 (7%)    |
| detailed         |   2  |   0.0%   |    164.7       |   N/A     |  2/30 (7%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — workflow disabled; data collection paused

reasoning_depth · daily-security-red-team

Variants: single_pass vs iterative · 4 total runs · min_samples: 30

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| single_pass      |   2  | 100.0%   |    358.1       |  (ref)    |  2/30 (7%)    |
| iterative        |   2  | 100.0%   |    302.3       |   N/A     |  2/30 (7%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — insufficient data; 2 per variant vs min_samples=30

prompt_style · ci-coach

Variants: detailed vs concise · 1 run assigned (concise) · min_samples: 20 · Started: 2026-05-15

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| detailed         |   0  |   N/A    |    N/A         |  (ref)    |  0/20 (0%)    |
| concise          |   1  | 100.0%   |   1059.2       |   N/A     |  1/20 (5%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — experiment started 2026-05-15; only 1 run

output_format · daily-compiler-quality

Variants: detailed vs concise · 1 run assigned (detailed) · min_samples: 20 · Started: 2026-05-16

+------------------+------+----------+----------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   |  p-value  | min_samples   |
+------------------+------+----------+----------------+-----------+---------------+
| detailed         |   1  | 100.0%   |    463.8       |  (ref)    |  1/20 (5%)    |
| concise          |   0  |   N/A    |    N/A         |   N/A     |  0/20 (0%)    |
+------------------+------+----------+----------------+-----------+---------------+
Recommendation: EXTEND — experiment started today (2026-05-16); only 1 run

output_format · daily-code-metrics

Variants: full_detail vs executive_summary · 0 runs assigned · min_samples: 20 · Started: 2026-05-16

Recommendation: EXTEND — experiment started today; no runs assigned yet

detail_level · daily-architecture-diagram

Variants: brief vs comprehensive · 0 runs assigned · min_samples: 10 · Start: 2026-05-19

Recommendation: EXTEND — experiment start date in the future (2026-05-19)

---

📊 Summary

Experiment	Workflow	Control	Best variant	p-value	Guardrails	Recommendation
output_format	daily-issues-report	collapsible	inline	N/A	UNKNOWN	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck	concise	concise	0.43	provisional	EXTEND
output_format	deep-report	full_briefing	executive_brief	N/A	UNKNOWN	EXTEND
prompt_style	daily-news	detailed	—	N/A	UNKNOWN	EXTEND
reasoning_depth	daily-fact	single_pass	—	N/A	UNKNOWN	EXTEND
prompt_style	issue-arborist	concise	—	N/A	UNKNOWN	EXTEND
reasoning_depth	daily-security-red-team	single_pass	single_pass	N/A	—	EXTEND
prompt_style	ci-coach	detailed	—	N/A	—	EXTEND
output_format	daily-compiler-quality	detailed	—	N/A	—	EXTEND
output_format	daily-code-metrics	full_detail	—	N/A	—	EXTEND
detail_level	daily-architecture-diagram	brief	—	N/A	—	EXTEND

Analysis window: last 30 runs per workflow · Significance threshold: p < 0.05 (two-tailed)
Run: §25957307134

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · ● 28.6M · ◷

• expires on May 19, 2026, 8:40 AM UTC