Enhance TDG test suite with role-based cluster formation and WASM deployment validation

[kingdonb]

Problem Statement

Our current TDG (Test-Driven Generation) test suite validates upstream integration but doesn't comprehensively test the cluster formation scenarios and WebAssembly deployment patterns that are our project's core value proposition.

Current Test Coverage

Existing: tests/custom-image/03-upstream-integration.sh

• ✅ Upstream Makefile integration
• ✅ ARM64 asset structure
• ✅ Build configurability
• ✅ Asset validation

Missing Critical Test Scenarios

1. Role-Based Cluster Formation

• Compute-only cluster: Multiple nodes with only Spin extension reach Ready state
• Mixed cluster: Compute nodes + one gateway node form functional cluster
• Node Ready validation: Nodes don't hang waiting for unconfigured extensions

2. WebAssembly Deployment Validation

• Spin runtime: Can actually execute WASM workloads on ARM64
• Multi-arch confidence: WASM provides deployment certainty without complex Docker multi-arch pipelines
• Performance baseline: ARM64 WASM execution performance vs traditional containers

3. Tailscale Subnet Router Testing

• Gateway functionality: Single gateway node provides subnet routing
• Traffic flow: Service/pod CIDR accessible through Tailscale network
• Conflict prevention: Multiple Tailscale configurations don't break cluster

4. CozyStack Learning Path Validation

• Build machinery: Verify CozyStack's build system works for custom Talos variants
• Documentation accuracy: Ensure our "learn Talos through CozyStack" approach actually works
• Upstream compatibility: Changes don't break when CozyStack updates

Proposed Test Enhancements

1. Cluster Formation Tests

tests/
├── cluster-formation/
│   ├── 01-compute-only-cluster.sh   # Spin-only nodes form cluster
│   ├── 02-mixed-role-cluster.sh     # Compute + gateway cluster formation
│   └── 03-node-ready-timing.sh      # Extension loading doesn't block Ready state

2. WebAssembly Integration Tests

tests/
├── wasm-integration/
│   ├── 01-spin-runtime-deploy.sh    # Deploy real WASM app to ARM64
│   ├── 02-multi-arch-confidence.sh  # Validate WASM deployment simplicity
│   └── 03-performance-baseline.sh   # ARM64 WASM vs container performance

3. Network Architecture Tests

tests/
├── network-architecture/
│   ├── 01-subnet-router-config.sh   # Tailscale gateway configuration
│   ├── 02-traffic-flow-test.sh      # Service CIDR accessibility
│   └── 03-single-gateway-only.sh    # Prevent multiple subnet routers

Success Criteria

• Role separation validated: Compute and gateway nodes have distinct, working configurations
• Cluster formation confirmed: Mixed-role clusters reach operational state
• WASM deployment proven: Real WebAssembly applications deploy and execute on ARM64
• Learning path verified: CozyStack → Talos Linux education approach validated
• Network routing working: Tailscale subnet router provides proper connectivity

Implementation Priority

Phase 1: Core Functionality (Blocking Production)

• Compute-only cluster formation
• Basic WASM deployment on ARM64
• Gateway node subnet router validation

Phase 2: Architecture Validation (CozySummit Demo)

• Mixed cluster deployment scenarios
• Performance comparisons (WASM vs containers)
• Documentation accuracy verification

Phase 3: Reliability & Edge Cases

• Gateway failover behavior
• Extension loading error handling
• Upstream compatibility regression detection

Technical Context

• Architecture Goal: Simplified home lab setup vs traditional VM/ingress/CNI complexity
• Learning Focus: CozyStack build machinery for Talos Linux education
• Production Target: ARM64 clusters ready for CozySummit Virtual 2025
• TDG Methodology: Tests define production requirements, implementation satisfies tests

This comprehensive test coverage ensures our role-based ARM64 architecture actually works for real cluster deployments, not just image building.