e2e smoke test and Go-vs-Python parity checklist

1. Purpose

This directory is the evidence gate for cutting over from the Python cde-plugin/bin/runcode to the Go runcode binary. It contains two things:

• An opt-in live smoke test (smoke_test.go, build-tagged e2e) that drives a real workspace through a full lifecycle and asserts key behaviors.
• This parity checklist (sections 5-7) that tells the operator how to verify the Go binary matches the Python binary's observable contract before flipping production traffic.

The cutover itself is out of scope here. Pass this checklist first.

2. WARNING

Running the smoke test creates and then deletes a real, billed workspace.

• With RUNCODE_API_BASE unset the test runs against production. Point it at a non-prod backend by setting RUNCODE_API_BASE.
• The test sets RUNCODE_CACHE_HOME to an isolated temp dir for the duration of the run. It will NOT touch or corrupt your real ~/.cache/runcode session or stored token.
• The delete step runs via t.Cleanup, so the workspace is removed even if an intermediate step fails. A very early panic (e.g., build failure) could leave it orphaned; check the dashboard if the run crashes hard.

3. How to run

RUNCODE_TOKEN=<real-token> go test -tags e2e ./e2e/ -v

Run from the runcode-cli/ directory.

Required: RUNCODE_TOKEN must be set to a valid API token. Without it the test skips immediately, which is why the default go test ./... never runs it.

Optional: RUNCODE_API_BASE overrides the API endpoint (default: prod). Set it to target a staging or local backend.

The forward probe sub-step needs python3 on the workspace. If it is absent the sub-step is soft-skipped (logged, not failed). Everything else still runs.

4. What the smoke test covers

Steps run in order; delete is guaranteed via t.Cleanup:

1. create - create --size tiny --json - waits for SSH-ready, parses workspace_id + asserts created=true and attached=true.
2. exec - exec -- echo <marker> - asserts the marker string appears in stdout.
3. write + get round-trip - writes a binary-safe file (includes non-ASCII byte) to the remote workdir, reads it back with get --out, asserts byte-for-byte equality.
4. forward + probe - starts python3 -m http.server on the workspace at port 8099, opens a local forward, HTTP-probes it with retries; cancels the forward on exit (soft-skipped if python3 absent).
5. delete (cleanup) - delete <id> --yes --json - asserts deleted=true.

5. Parity checklist

Method: run the Go binary and the Python cde-plugin/bin/runcode with identical arguments. Where the command supports --json, add it to both invocations. After each run, diff three things:

• error code strings (the "code" field in JSON error output)
• process exit codes
• the set of JSON keys in the top-level response object

All three must match on every command except the intentional differences listed in section 6. After the final fix wave, the error code strings and exit codes match Python everywhere (soft errors exit 1, operational errors exit 2), and list --json matches Python's bare-array-minus-provider shape.

Table A - offline / error-path parity (no workspace needed; run these first)

#	Command	Go invocation	Python invocation	Diff (codes/exit/keys)	Result
1	--version	runcode --version	cde-plugin/bin/runcode --version	Version STRING differs by build; assert both print a version line and exit 0
2	login / no-token shape	runcode logout && runcode list --json (no token present)	same	both should produce no_token error code, same exit
3	list --json	runcode list --json	cde-plugin/bin/runcode list --json	bare array; per-element keys match (Go lists all by default — see §6.5; Python may show fewer rows but the same shape)
4	status --json	runcode status --json	cde-plugin/bin/runcode status --json	keys/codes/exit
5	current --json (no attachment)	runcode current --json	cde-plugin/bin/runcode current --json	both report "none"/not_found shape
6	exec with no attached workspace	runcode exec -- echo hi	cde-plugin/bin/runcode exec -- echo hi	both not_found, exit non-zero
7	delete without --yes	runcode delete somews	cde-plugin/bin/runcode delete somews	both confirm_required, exit non-zero, ZERO destructive API calls
8	get with no attachment	runcode get /nope	cde-plugin/bin/runcode get /nope	both not_found
9	port-forward bad port	runcode port-forward notaport	cde-plugin/bin/runcode forward notaport	both bad_request (Go forward alias also works)
10	doctor --json	runcode doctor --json	cde-plugin/bin/runcode doctor --json	intentional difference - see section 6; do NOT count as failure
11	clean	runcode clean	cde-plugin/bin/runcode clean	both preserve the token; compare behavior

Table B - live-lifecycle parity (needs a workspace; same args, --json)

Run both binaries against the same workspace (or sequential workspaces of the same size). Diff codes/exit/keys for each.

#	Command	Go invocation	Python invocation	Diff (codes/exit/keys)	Result
1	create	runcode create --size tiny --json	cde-plugin/bin/runcode create --size tiny --json	keys/codes/exit
2	connect	runcode connect <id> --json	cde-plugin/bin/runcode connect <id> --json	keys/codes/exit
3	exec	runcode exec -- echo hi	cde-plugin/bin/runcode exec -- echo hi	keys/codes/exit
4	ssh (was run)	runcode ssh <id> -- ls	cde-plugin/bin/runcode run <id> -- ls	keys/codes/exit (Go run alias also works)
5	context	runcode context --json	cde-plugin/bin/runcode context --json	keys/codes/exit
6	write	runcode write test.txt --file /tmp/x	cde-plugin/bin/runcode write test.txt --file /tmp/x	keys/codes/exit
7	put	runcode put /tmp/x test.txt	cde-plugin/bin/runcode put /tmp/x test.txt	keys/codes/exit
8	get	runcode get test.txt --out /tmp/y	cde-plugin/bin/runcode get test.txt --out /tmp/y	keys/codes/exit
9	port-forward (was forward)	runcode port-forward 8080 --local 9090	cde-plugin/bin/runcode forward 8080 --local 9090	keys/codes/exit
10	port-forward --cancel	runcode port-forward 8080 --local 9090 --cancel	cde-plugin/bin/runcode forward 8080 --local 9090 --cancel	keys/codes/exit
11	disconnect	runcode disconnect --json	cde-plugin/bin/runcode disconnect --json	keys/codes/exit
12	stop	runcode stop <id> --json	cde-plugin/bin/runcode stop <id> --json	keys/codes/exit
13	delete --yes	runcode delete <id> --yes --json	cde-plugin/bin/runcode delete <id> --yes --json	keys/codes/exit

6. Intentional differences

These are the owner-approved, deliberate deviations from the Python cde-plugin/bin/runcode. They are NOT regressions; every other observable behavior must match. The error code strings and process exit codes match Python on every command (soft/user errors exit 1, operational errors exit 2).

1. doctor --json check-name set. The Go doctor checks token, api-base, cache, platform, status-line, auth; Python checks for a Python interpreter, ssh, and ssh-keygen on PATH. The Go CLI speaks SSH natively (golang.org/x/crypto/ssh) and never shells out, so those binaries are irrelevant; the cache-writability probe replaces them. The checks[].name set therefore differs by design.
2. Human / progress text goes to stderr (Python prints it to stdout). --json output is on stdout in both, so machine consumers see identical streams — parity is preserved for the path that matters.
3. status / current / disconnect accept --json (Python's argparse for those rejects the flag). A superset, so any Python invocation still works.
4. exec with no attached workspace returns error code not_found (Python returns a generic error). More specific, exit code unchanged.
5. list lists ALL workspaces by default and accepts --all as a no-op alias (Python defaults to SSH-connectable-only and uses --all to widen). Scripts passing --all keep working; list --json otherwise emits the same bare array (each element minus the internal provider field, plus a computed connectable).
6. Loopback HTTP allowance is narrower: only 127.0.0.1 / ::1 / localhost are accepted over plain http, vs Python's whole 127.0.0.0/8. Stricter, never looser.
7. (Known limitation, not yet fixed) context / write / put / get do not enforce a per-call timeout (Python caps reads at 30s and writes at 120s). An unreachable box is still bounded by the 20s SSH handshake, so calls cannot hang indefinitely.
8. Command surface aligned to Coder/Gitpod conventions (the Go CLI is the replacement for the Python cde-plugin, so it is free to improve once cut over). All Python names still work as hidden aliases, so existing scripts and muscle memory are unaffected:
   • ssh <ws> [-- cmd] is the human verb (bare → interactive shell, -- cmd → one-shot). It absorbs Python's run and shell, both kept as hidden aliases. The agent-facing exec (targets the attached workspace, no positional) is unchanged.
   • port-forward <port> renames Python's forward (kept as a hidden alias). port-forward is unambiguous and matches coder port-forward.
   • start <ws> (NEW, no Python equivalent) powers a stopped box on and waits for running without attaching — the inverse of stop; --no-wait returns immediately. (connect --start remains the power-on-AND-attach combo.)
   • open <ws> (NEW, no Python equivalent) opens the workspace browser IDE, or prints the URL with --print (and auto-falls-back to printing on a headless box). It reads the cached session's web_url when one is fresh, else mints to learn the current URL. URL is validated to absolute http/https before being handed to the OS opener (security: never a file:///option-like string to xdg-open/open/rundll32, always via argv, never a shell).

7. Results log

Fill this in after the operator run:

Date:            ____________________
Operator:        ____________________
Backend:         prod | ____________________  (RUNCODE_API_BASE)
Workspace id:    ____________________
Region/size:     ____________________
Go binary ver:   ____________________

Smoke lifecycle:
  create ........ PASS / FAIL  notes:
  exec .......... PASS / FAIL  notes:
  write+get ..... PASS / FAIL  notes:
  forward+probe . PASS / FAIL / SKIPPED(python3 absent)  notes:
  delete ........ PASS / FAIL  notes:

Parity (Go vs Python --json): codes / exit / keys
  Table A offline:  PASS / FAIL  notes:
  Table B live:     PASS / FAIL  notes:

Overall: PASS / FAIL