Skip to content

Troubleshooting

First step: open the admin Allocator Logs page or run lablink logs, then run the built-in checks and read their output:

lablink doctor          # operator-side prerequisites, for the provider you configured
lablink status          # allocator health and client inventory
lablink client doctor   # on a BYO client box: registration, container, log shipper

lablink doctor branches on your provider — Docker checks under manual, and OpenTofu, AWS credentials, S3 and AMI checks under aws. Most problems below are named in its output.

Run tofu output commands below from the initialized infrastructure directory: lablink-infrastructure/ in a template checkout, or ~/.lablink/deploy/<deployment_name>/<environment>/ for the CLI.

Which port?

Inside the AWS host, the allocator container listens on 5000. Caddy proxies host port 80 to it for none, letsencrypt, and cloudflare; acm uses an ALB. With domain-based HTTPS, port 5000 may be bound to loopback, so test through the configured public URL from outside. A manual-provider compose stack exposes host port 80.

Docker

Permission denied connecting to the Docker daemon 
Got permission denied while trying to connect to the Docker daemon socket

Add yourself to the docker group:

sudo usermod -aG docker $USER
newgrp docker
docker ps
Cannot connect to the Docker daemon 
Cannot connect to the Docker daemon at unix:///var/run/docker.sock

Linux — start the service:

sudo systemctl start docker
sudo systemctl enable docker

macOS / Windows — start Docker Desktop.

SSH access

Permission denied (publickey)

In order, the three things that cause this:

  1. Key permissions too open — SSH silently refuses a world-readable key. 
    chmod 600 ~/lablink-key.pem
ls -l ~/lablink-key.pem      # -rw-------
  2. Wrong key — the deployment key opens the allocator, while client VMs use the allocator-generated keypair. For the allocator, re-extract it from OpenTofu state: 
    tofu output -raw private_key_pem > ~/lablink-key.pem
chmod 600 ~/lablink-key.pem
  3. Wrong user — it's ubuntu on the Ubuntu AMIs LabLink uses, not ec2-user.
SSH connection timed out

Timeout means the packets aren't arriving at all — a closed port or a stopped instance, not an auth problem.

aws ec2 describe-instances --instance-ids <id> \
  --query 'Reservations[0].Instances[0].State.Name'
tofu output ec2_public_ip

The template already allows inbound SSH on port 22; check the instance state, effective security group, and VPC network ACL before changing rules.

How do I get the client-VM SSH key?

Client VMs use a different key from the allocator. On an AWS deployment, first SSH to the allocator with its deployment key. In the production allocator container, the client key is the lablink_private_key_pem OpenTofu output. Save it to a private file on the allocator host:

sudo docker ps --format '{{.ID}} {{.Image}}' # find the allocator container ID
sudo docker exec <allocator-container-id> sh -lc \
  'cd /app/.venv/lib/python*/site-packages/lablink_allocator_service/terraform && tofu output -raw lablink_private_key_pem' \
  > ~/lablink-client-key.pem
chmod 600 ~/lablink-client-key.pem

The client-VM public IP comes from the allocator's client inventory or the client OpenTofu outputs, not ec2_public_ip in the deployment state.

Deploying (AWS provider)

OpenTofu init fails: bucket does not exist

lablink setup creates the S3 state bucket and DynamoDB lock table and writes the bucket name into your config. If either is missing, re-run:

lablink setup

The CLI bucket name is lablink-tf-state-<account-id>.

Error acquiring the state lock

A previous OpenTofu run died without releasing its DynamoDB lock.

First make sure nothing is actually still running:

aws dynamodb scan --table-name lock-table --region us-west-2
ps aux | grep tofu

Entries with an Info field are real locks; the rest are digests. Copy the exact LockID — it does not always carry an -md5 suffix — and delete it:

aws dynamodb delete-item \
    --table-name lock-table \
    --key '{"LockID":{"S":"<exact-lock-id-from-scan>"}}' \
    --region us-west-2

On Windows PowerShell, put the key in a key.json file and pass --key file://key.json.

Common lock paths:

  • Template infrastructure: <bucket>/<env>/terraform.tfstate
  • CLI infrastructure: lablink-tf-state-<account-id>/<deployment_name>/<env>/terraform.tfstate
  • Client VMs: <bucket>/<deployment_name>/<env>/client/terraform.tfstate

If you get AccessDeniedException: not authorized to perform: dynamodb:GetItem, the allocator's IAM role is missing dynamodb:GetItem, PutItem and DeleteItem on table/lock-table. Add them and redeploy.

Prevention: let OpenTofu runs finish. Don't terminate instances from the console mid-apply, and use the destroy workflow rather than deleting resources by hand.

Resource already exists

Inspect the resource and state before retrying. For a separate deployment, choose a different deployment_name or environment in the config and workflow inputs. See Template Deployment.

Resources won't destroy cleanly

Inspect the failed OpenTofu destroy output and check whether client VMs still exist. The template destroy workflow removes client VMs first. For orphaned resources after a failed destroy, preview the template cleanup script or run lablink cleanup --dry-run on the CLI path before deleting anything.

GitHub Actions: could not assume role with OIDC 
aws iam list-open-id-connect-providers
aws iam get-role --role-name github-actions-lablink \
  --query 'Role.AssumeRolePolicyDocument'

The trust policy must name your repository (repo:<owner>/<repo>:*), and the role ARN in the workflow must match your account ID.

GitHub Actions: workflow won't trigger

Check the branch name and path filters against the workflow's on: block, confirm Actions are enabled for the repository, and validate the YAML:

yamllint .github/workflows/*.yml

Reaching the allocator

Browser shows connection refused

Work outward from the container:

ssh -i ~/lablink-key.pem ubuntu@<ip>
sudo docker ps                    # is it running?
sudo docker logs -f <container>   # did it crash on startup?
curl localhost:5000               # does it answer locally?

The template's security group already allows 80, 443, 5000, and 22, but HTTPS modes may bind 5000 only to loopback. Check Caddy, the public URL, and the effective security group rather than opening another port.

Browser cannot reach the HTTP site (no SSL provider)

Symptoms: "This site can't be reached", ERR_CONNECTION_REFUSED on a deployment using ssl.provider: "none".

Your browser cached an HSTS policy from a previous HTTPS deployment on the same hostname, so it silently upgrades to HTTPS — and port 443 is closed.

Quickest workarounds: use a private/incognito window, or hit the IP directly.

To clear it properly:

  1. Open chrome://net-internals/#hsts (edge:// on Edge)
  2. Under Delete domain security policies, enter your domain
  3. Click Delete, then reload with an explicit http://
  1. Close all Firefox windows
  2. Delete SiteSecurityServiceState.txt from your profile directory (~/Library/Application Support/Firefox/Profiles/ on macOS, %APPDATA%\Mozilla\Firefox\Profiles\ on Windows, ~/.mozilla/firefox/ on Linux)
  3. Restart and reload with an explicit http://
  1. Quit Safari
  2. rm ~/Library/Cookies/HSTS.plist
  3. Restart and reload with an explicit http://

Expected behaviour with ssl.provider: "none" is a "Not Secure" badge — no certificate is issued and traffic is unencrypted. For HTTPS, set ssl.provider to letsencrypt, cloudflare or acm. See Configuration → SSL Options.

HTTP works but HTTPS doesn't

Caddy cannot get a certificate until DNS resolves to the allocator, so check that first:

nslookup <your-domain> 8.8.8.8
sudo systemctl status caddy
sudo journalctl -u caddy -f

In the Caddy log, certificate obtained successfully means it worked; challenge failed means DNS isn't ready; timeout means ports 80/443 aren't reachable. Caddy retries every 2 minutes on its own, so a few minutes of patience often fixes it.

Verify the Caddyfile proxies to the container's real port:

cat /etc/caddy/Caddyfile
# <your-domain> {
#     reverse_proxy localhost:5000
# }
sudo systemctl restart caddy

See Configuration for the Let's Encrypt duplicate-certificate limit during repeated test deployments.

Domain doesn't resolve to the allocator 
# Does the record exist?
aws route53 list-resource-record-sets --hosted-zone-id <zone-id> \
  --query "ResourceRecordSets[?Name=='<your-domain>.']"

# Does it match the actual IP?
tofu output ec2_public_ip
dig <your-domain> +short

If the record is missing, re-run tofu apply. If it exists but isn't propagating, give it 5–15 minutes and flush your local cache (sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder on macOS, sudo systemd-resolve --flush-caches on Linux, ipconfig /flushdns on Windows).

DNS record landed in the wrong hosted zone

OpenTofu's zone lookup matched the parent zone (e.g. example.com instead of lablink.example.com). Pin it explicitly:

dns:
  enabled: true
  terraform_managed: true
  domain: "test.lablink.example.com"
  zone_id: "Z0123456789ABCDEFGHIJ"   # force the correct zone

Delete the stray record from the wrong zone, then re-apply.

Verifying a deployment end to end

The template repo ships a script that checks DNS, HTTP, and SSL together. It lives in scripts/, not lablink-infrastructure/:

./scripts/verify-deployment.sh prod          # reads config.yaml + tofu outputs
./scripts/verify-deployment.sh <domain> <ip> # or pass them explicitly
./scripts/verify-deployment.sh --ci prod     # no ANSI colors, for CI logs

Green checks are pass, yellow warnings usually mean "not ready yet — wait and retry", red is a real failure. The three common ones are a DNS timeout (wait longer), HTTP not responding (check the allocator container logs), and SSL not ready (check the Caddy logs).

NS delegation not working

nslookup returns your registrar's nameservers instead of AWS, so records in Route53 are never consulted.

dig NS lablink.example.com          # should list four awsdns nameservers
aws route53 get-hosted-zone --id <zone-id> --query 'DelegationSet.NameServers'

Add those four as NS records for the lablink subdomain at whoever hosts the parent domain, TTL 300, then re-check after 5–15 minutes.

If you have both a parent and child zone in Route53 fighting each other, keep the one LabLink manages and delete the duplicate — aws route53 list-hosted-zones --query 'HostedZones[*].[Name,Id]' --output table shows them all.

Client VMs (AWS provider)

Clicking Create New VM Instance does nothing 
sudo docker logs -f <allocator-container>
sudo docker exec <allocator-container> aws sts get-caller-identity
sudo docker exec <allocator-container> tofu version

If get-caller-identity fails, the allocator's IAM instance profile isn't attached or lacks EC2 permissions (RunInstances, DescribeInstances) and VPC permissions (CreateSecurityGroup).

To run OpenTofu by hand inside the container:

sudo docker exec -it <allocator-container> bash
cd /app/.venv/lib/python*/site-packages/lablink_allocator_service/terraform
tofu init && tofu plan
VM created but never appears in the allocator

Clients register themselves — the allocator does not insert rows when OpenTofu finishes. Each client posts to /api/v1/clients/register, and the allocator stores its hostname alongside an argon2 hash of a freshly minted client_secret. A VM that never registers is a VM that never reached that endpoint.

# What the allocator has
sudo docker exec <allocator-container> psql -U lablink -d lablink_db \
  -c "SELECT hostname, status, inuse FROM vms;"

# What the allocator saw
sudo docker logs <allocator-container> | grep -E "clients/register|heartbeat"

# What the client tried (use the allocator-generated client key)
ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip>
sudo docker logs <client-container>

# Can the client reach the allocator at all?
curl -i http://<allocator-ip>/api/health  # IP-only mode; use the configured HTTPS domain otherwise

If /api/health fails from the client, stop debugging the client — it's the security group or the allocator's listening port.

Don't INSERT rows by hand

A row created with just (hostname, inuse) has a null client_secret_hash, so that client can never authenticate for heartbeats, status or logs. It will look present in the admin panel and be permanently broken. Let the client register, or re-provision it.

Failed VMs aren't being rebooted

The allocator runs an AutoRebootService that sweeps every 60 seconds for VMs that are in error, running but GPU-Unhealthy, stuck initializing over 25 minutes, stuck rebooting over 10 minutes, or running but silent for 3 minutes with no heartbeat.

Assigned VMs get a warm sudo reboot that keeps their container state. Unassigned VMs get a cold reboot (docker rm -f, cloud-init clean, then reboot) that reprovisions from scratch. EC2 stop/start is used only if SSH fails. There are at most 3 attempts with a 300-second cooldown; an exhausted VM becomes error and its seat is released.

sudo docker logs <allocator-container> | grep -i reboot

sudo docker exec <allocator-container> psql -U lablink -d lablink_db -c \
  "SELECT hostname, status, reboot_count, last_reboot_time FROM vms WHERE reboot_count >= 3;"

There is no admin reset endpoint for an exhausted VM. Review its logs and replace it if needed; an out-of-band EC2 reboot alone does not reset the allocator's attempt counter.

BYO boxes are never rebooted

Only providers that can recover hosts participate. The manual provider can't, so bring-your-own machines are left alone.

CUDA not available on a client VM 
ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip>
nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
sudo docker inspect lablink-client --format '{{.HostConfig.Runtime}}'

The client container is launched with --runtime=nvidia; a Docker default-runtime setting is not required. If nvidia-smi fails on the host, use an AMI with NVIDIA drivers. Confirm the instance type has a GPU.

Bring-your-own clients (manual provider)

Start here: lablink client doctor

Run it on the client box. It checks three things — that ~/.lablink/client.env exists with this box's credentials, that the lablink-client container is running, and that the log shipper is forwarding to the allocator.

Most failures are fixed by re-running lablink client register. See Bring-Your-Own Clients.

The client registers but participants can't reach the desktop

Almost always the wrong manual.connectivity mode for your network:

  • lan_direct puts the participant's browser onto the client's LAN IP. It cannot serve off-LAN participants, no matter what exposure mode you pair it with.
  • mesh_overlay joins the client to a Tailscale tailnet and proxies the byte path through the allocator's nginx.
  • reverse_tunnel has the client dial out and hold one connection open — for networks that won't carry Tailscale or boxes that can't accept inbound connections.

See Pick a connectivity mode.

A reverse-tunnel client logs Invalid status code: 524 every ~2 minutes

Expected churn when manual.connectivity=reverse_tunnel is paired with manual.participant_exposure=cloudflare_tunnel. The wstunnel client keeps one "await" connection open for the server to hand new tunnels to, and the server holds that HTTP upgrade pending until it's needed. Cloudflare's proxy gives an origin 120 seconds to answer, then kills the pending request with its own 524 status — so the client logs the error and re-dials within a second, forever.

Established desktop sessions ride their own, fully-upgraded connections and are not affected (measured: sessions survive many churn cycles and 400+ seconds of idle). The cost is a ~1-second window every ~2 minutes in which a brand-new participant connection stalls until the client re-attaches, plus this log noise. There is no fix on the free Cloudflare plan (the 120s timeout is configurable only for Enterprise), and wstunnel upstream documents CDN buffering as incompatible with its protocol — treat the message as noise unless it stops being periodic.

Sessions on a laptop client drop when the lid closes (or randomly)

A bring-your-own client on a laptop dies with the machine: Windows sleep/Modern Standby suspends the WSL2 VM, which freezes Docker, the tunnel, and KasmVNC in one stroke — every participant session drops and the allocator eventually marks the client unhealthy. Wi-Fi power saving can do the same without sleep.

Before serving real participants from a laptop, disable sleep while plugged in (Windows: powercfg /change standby-timeout-ac 0) and turn off Wi-Fi power management. The viewer auto-reconnects when the machine comes back, but anything running in the session meanwhile was frozen.

A mesh-overlay client looks healthy but is unreachable

Re-registering a mesh_overlay client mints a new Tailscale node. MagicDNS suffixes the name (-1, -2, …) because the old node is offline but still holds the original name, so the allocator's stored hostname points at a machine that no longer answers — while the client's own logs look perfectly fine.

Delete the stale machine in the Tailscale admin console, then re-register. Use lablink client reset-overlay when you deliberately want a fresh node identity; note it requires the lablink-client container to be gone first, since Docker won't remove an attached volume.

Allocator compose stack won't come up 
docker compose ps
docker compose logs -f allocator
curl -i http://localhost/api/health

lablink doctor under the manual provider checks the Docker side for you.

Custom client images

Symptoms specific to running your own research image instead of the default — see Adapting for Your Software.

The client container never starts on a fresh VM 
ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip>
sudo docker ps -a                    # did it exit, or was it never created?
sudo docker logs <client-container>

An image that exits immediately is almost always a missing dependency or a broken entrypoint — reproduce it locally with the same command before blaming the deployment:

docker run --rm -it your-image:tag bash

If the container was never created, the failure is earlier than your image: read sudo cat /var/log/cloud-init-output.log for the pull, and confirm the tag in machine.image exists and is publicly readable (private registries need credentials baked into the AMI).

machine.repository isn't cloned onto the desktop

Check the URL, that the repository is public (LabLink clones over HTTPS with no credentials), and that the VM has outbound network access. The clone runs at startup, so its output is in the client container logs.

Your software isn't found inside the session

The desktop session doesn't inherit the Dockerfile's build-time shell. Set PATH (and any activation, e.g. a virtualenv) via ENV in the image rather than in ~/.bashrc or a RUN line, then verify from inside the running container:

sudo docker exec <client-container> which your-software

Inside the allocator

Cannot connect to PostgreSQL

psycopg2.OperationalError: could not connect to server. The container's start.sh starts Postgres and blocks on pg_isready before Flask boots, so a refused connection normally means the cluster died rather than that it was never started.

sudo docker exec <container> pg_isready -U lablink
sudo docker exec <container> psql -U lablink -d lablink_db -c "SELECT 1;"
sudo docker exec <container> tail -f /var/log/postgresql/postgresql-*-main.log

To restart the cluster in place:

sudo docker exec -it <container> pg_ctlcluster 17 main restart
Running out of database connections

The allocator reports its own connection usage — check /admin (a line under the title) or hit the endpoint directly:

curl -u admin:<password> https://<allocator-domain>/api/health/connections

At the critical level, new connections can be refused, which fails VM registration and admin actions. Raise Postgres max_connections and LABLINK_DB_POOL_MAX_SIZE together, and check for connections stuck idle in transaction.

Container runs but Flask doesn't start 
sudo docker logs <container>          # port in use, import error, bad config?
sudo docker exec <container> cat /config/config.yaml
sudo netstat -tulpn | grep 5000
sudo docker restart <container>

A config that fails schema validation stops the app before it serves anything — the log names the offending key.

Diagnostic commands

# Host health
df -h; free -h; top
sudo docker ps -a
sudo docker stats
sudo netstat -tulpn

# Allocator on an AWS host (manual-provider host port is 80)
curl localhost:5000/api/health
sudo docker logs -f <allocator-container>

# Database
sudo docker exec <container> pg_isready -U lablink
sudo docker exec <container> psql -U lablink -d lablink_db -c "SELECT hostname, status, inuse FROM vms;"

# AWS reachability from inside the container
sudo docker exec <container> aws sts get-caller-identity

Before you ask for help

  • lablink doctor run, output read
  • Container running (docker ps) and answering at the host port for your provider
  • Public URL reaches port 80 or 443 as configured; SSH reaches port 22
  • SSH key is mode 600
  • AWS credentials resolve (aws sts get-caller-identity)
  • OpenTofu state not locked, S3 bucket exists
  • Config passes validation — no unknown keys

Still stuck? Search existing issues, then open a new one with the lablink doctor output, the relevant container logs, what you expected, and what you tried.