Skip to content

development.md

Latest commit

 

History

History
224 lines (160 loc) · 7.52 KB

development.md

File metadata and controls

224 lines (160 loc) · 7.52 KB

Development Guide

Requirements

  • Go 1.24+
  • Docker (for building container images)
  • A kubeconfig pointing at a cluster (for manual testing)
  • frp binaries (for integration tests)

Building

# Build the operator binary
make build

# Build the Docker image
make docker-build IMG=fly-tunnel-operator:dev

The binary is output to bin/manager.

Running locally

The operator can run outside the cluster using your local kubeconfig:

export FLY_API_TOKEN=<your-token>
export FLY_ORG=<your-org-slug>
export FLY_REGION=ord

go run . --namespace <target-namespace>

Or equivalently:

make run

Important: --namespace flag

The --namespace flag (or OPERATOR_NAMESPACE env var) controls where the operator creates frpc Deployments, ConfigMaps, and the leader election Lease. When running locally you should always set this explicitly:

# Create the namespace first
kubectl create namespace my-dev-ns

# Run with an explicit namespace
go run . --namespace my-dev-ns

If omitted, it defaults to fly-tunnel-operator-system. The Helm chart handles this automatically by setting --namespace={{ .Release.Namespace }} in the Deployment spec, so the operator always targets the Helm release namespace.

By default the operator watches Services with loadBalancerClass: fly-tunnel-operator.dev/lb. Override with --load-balancer-class.

Testing

Unit tests

Unit tests use a fake Fly.io API server (internal/fakefly/server.go) and controller-runtime's fake client. No network or cluster access required.

make test

Or run specific packages:

go test ./internal/frp/ -v          # frp config generation
go test ./internal/flyio/ -v        # Fly.io API client
go test ./internal/tunnel/ -v       # tunnel lifecycle manager

Controller integration tests

Controller tests use envtest which runs a real kube-apiserver and etcd locally. The envtest binaries are auto-discovered from ~/.local/share/kubebuilder-envtest/.

go test ./internal/controller/ -v

If you don't have envtest binaries installed, use setup-envtest:

go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
setup-envtest use 1.31.x

Then set the KUBEBUILDER_ASSETS environment variable to the path printed by setup-envtest, or let the test suite auto-discover them.

frp integration tests

Integration tests verify that generated frpc/frps configs work with real frp binaries. They are gated behind a build tag and skipped if the binaries are not found.

# Download frp binaries
curl -L https://github.com/fatedier/frp/releases/download/v0.61.1/frp_0.61.1_linux_amd64.tar.gz | tar xz -C /tmp

# Run integration tests
go test -tags integration ./internal/frp/ -v

The tests look for frps/frpc in these locations (in order):

  1. FRP_BIN_DIR environment variable
  2. PATH
  3. /tmp/frp_0.61.1_linux_amd64/
  4. /usr/local/bin/
  5. /usr/bin/

The integration test suite includes:

Test What it verifies
TestIntegration_SinglePortTunnel Full TCP tunnel with echo server, data flows end-to-end
TestIntegration_MultiPortTunnel Multi-port tunnel (HTTP + HTTPS), independent backends
TestIntegration_ConfigParseValid frpc verify accepts a 3-port generated config
TestIntegration_ServerConfigParseValid frps verify accepts the server config
TestIntegration_LargePortRange 20-port config generates correctly and parses
TestIntegration_UDPProxy UDP protocol type is emitted and parseable

Running all tests

# Unit + controller integration tests
make test

# Including frp integration tests
go test -tags integration ./... -v

Project structure

internal/
├── controller/
│   ├── service_controller.go       # Reconciler: watches Services, drives provisioning
│   ├── service_controller_test.go  # envtest integration tests (6 tests)
│   └── suite_test.go               # envtest setup (shared manager, fake fly server)
├── tunnel/
│   ├── manager.go                  # Provision / Update / Teardown orchestration
│   └── manager_test.go             # Unit tests with fakes (5 tests)
├── flyio/
│   ├── client.go                   # Fly.io Machines REST API + GraphQL client
│   └── client_test.go              # Unit tests with httptest server (15 tests)
├── frp/
│   ├── config.go                   # TOML config generation for frpc/frps
│   ├── config_test.go              # Unit tests (3 tests)
│   └── config_integration_test.go  # Integration tests with real frp binaries (6 tests)
└── fakefly/
    └── server.go                   # Fake Fly.io API (REST + GraphQL) for testing

Key design decisions

No CRDs

The operator works entirely with core Service objects. It watches Service type: LoadBalancer with a specific loadBalancerClass and stores all tunnel state in annotations on the Service itself. This avoids CRD installation and version management.

Finalizer-based cleanup

A finalizer (fly-tunnel-operator.dev/finalizer) is added to every managed Service. On deletion, the operator tears down the Fly.io Machine, releases the IPv4, and deletes the in-cluster frpc Deployment + ConfigMap before removing the finalizer and allowing the Service to be garbage collected.

One Machine per Service

Each LoadBalancer Service gets its own Fly.io Machine running frps and its own dedicated IPv4. This provides isolation and makes per-service region/size overrides straightforward.

frpc runs in-cluster

The frpc client runs as a Deployment inside the cluster. Its config is mounted from a ConfigMap that the operator regenerates on port changes. The frpc connects outbound to the Fly.io Machine's public IP, so no inbound firewall rules are needed on the cluster.

Service annotations

All tunnel state is stored directly on the Service as annotations — no external database or CRD state:

Annotation Description
fly-tunnel-operator.dev/fly-app Fly.io App name created for this Service
fly-tunnel-operator.dev/machine-id Fly.io Machine ID
fly-tunnel-operator.dev/frpc-deployment Name of the in-cluster frpc Deployment
fly-tunnel-operator.dev/ip-id Fly.io IP address allocation ID
fly-tunnel-operator.dev/public-ip Allocated public IPv4 address
fly-tunnel-operator.dev/fly-region (user-set) Override Fly.io region
fly-tunnel-operator.dev/fly-machine-size (user-set) Override machine size

Helm chart

The Helm chart is in charts/fly-tunnel-operator/. To render templates without installing:

make helm-template

To install:

make helm-install

To customize values:

helm install fly-tunnel-operator charts/fly-tunnel-operator \
  --namespace fly-tunnel-operator-system \
  --create-namespace \
  -f my-values.yaml

Makefile targets

Target Description
make build Build the operator binary to bin/manager
make run Run the operator locally via go run
make test Run all unit and envtest integration tests
make lint Run golangci-lint
make fmt Format Go source files
make vet Run go vet
make docker-build Build Docker image
make docker-push Push Docker image
make helm-install Install Helm chart
make helm-uninstall Uninstall Helm chart
make helm-template Render Helm templates to stdout
make clean Remove build artifacts