Lewati ke isi

Deploy

Purpose

Step-by-step manual deploy of API, RAG, and admin from a fresh merge to main. Pass 1 of HUPH does not have CI/CD — deploys are performed by an engineer on the production host.

Prerequisites

  • SSH access to the production host
  • sudo access (for nginx + systemd commands)
  • Clean local working tree on main
  • docker-compose available on the host
  • You know which services your PR touched (API? RAG? admin?)

Before you deploy

  1. Confirm tests pass on main locally:
npm run test -w apps/api
cd apps/rag && pytest
cd apps/admin && npx tsc --noEmit && npm run build
  1. Confirm merge is on main:
git checkout main && git pull origin main
git log --oneline -5

  1. Alert the channel if this deploy changes behavior visible to operators (new feature, migration, env var addition).

  2. Take a snapshot of current state for rollback:

docker ps --format '{{.Names}}\t{{.Status}}' > /tmp/pre-deploy-state.txt
git rev-parse HEAD > /tmp/pre-deploy-sha.txt

Deploy API

cd /opt/huph
git pull origin main
docker-compose build huph-api
docker-compose up -d huph-api

Verify:

docker-compose ps huph-api   # should show "Up (healthy)"
curl -s http://localhost:3101/health   # should return {"status":"ok"}
docker-compose logs huph-api --tail 30 | grep -i error   # should be empty

Deploy RAG

cd /opt/huph
docker-compose build huph-rag
docker-compose up -d huph-rag

Important: RAG takes ~60–90 seconds to load models on startup. During that window, /health returns 503 — this is expected.

Verify after ~90 seconds:

curl -s http://localhost:3102/health   # should return {"status":"ok"}
docker-compose logs huph-rag --tail 50 | grep "Uvicorn running"

Deploy admin

The admin runs via systemd (not docker) in production, on port 47293.

cd /opt/huph
git pull origin main   # if not already done
cd apps/admin
npm install   # if package.json changed
npm run build
sudo systemctl restart huph-admin

Verify:

sudo systemctl status huph-admin --no-pager | head -20
curl -s http://localhost:47293/ | grep -q "<title>" && echo "OK"

Admin is served via nginx at admin.huph.val.id → 127.0.0.1:47293.

Deploy docs (docs.huph.val.id)

Documentation is a static site built with MkDocs Material. Build locally, nginx serves /opt/huph/site/ directly.

cd /opt/huph
source docs-env/bin/activate

# Build with strict validation
mkdocs build --strict
# Expected: "Documentation built in X.XX seconds"

# (Optional) inspect the new output
ls site/index.html site/id/index.html site/guide/operators/getting-started/

Nginx does not need to be restarted because the site is served from disk directly — the next request picks up the new files. However, if you have any Cloudflare caching or want to force Cloudflare to re-pull static assets, purge the docs.huph.val.id zone.

First-time deploy requires bootstrap of the Let's Encrypt cert via certbot. The repo vhost file references fullchain.pem / privkey.pem at /etc/letsencrypt/live/docs.huph.val.id/ — those don't exist until certbot issues them.

Bootstrap sequence:

# 1. Write an HTTP-only stub so certbot webroot challenge can reach nginx
sudo tee /etc/nginx/sites-available/docs.huph.val.id > /dev/null <<'EOF'
server {
    listen 80;
    listen [::]:80;
    server_name docs.huph.val.id;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 503;
    }
}
EOF
sudo ln -sfn /etc/nginx/sites-available/docs.huph.val.id /etc/nginx/sites-enabled/docs.huph.val.id
sudo nginx -t && sudo nginx -s reload

# 2. Request the Let's Encrypt cert
sudo certbot certonly --webroot -w /var/www/certbot -d docs.huph.val.id

# 3. Replace the stub with the full HTTP+HTTPS vhost from the repo
sudo cp /opt/huph/docker/nginx/docs.huph.val.id.conf /etc/nginx/sites-available/docs.huph.val.id
sudo nginx -t && sudo nginx -s reload

# 4. Smoke test
curl -sI https://docs.huph.val.id/ | head -3

Subsequent deploys (after the cert is live) only need steps 3–4. Cert renewals are handled automatically by the certbot.timer systemd unit (or crontab, depending on setup) because the vhost keeps the .well-known/acme-challenge/ location block in the HTTP server.

Nginx reload (if configs changed)

sudo nginx -t                    # test config first
sudo nginx -s reload              # graceful reload

If you changed docker/nginx/*.conf, copy to system nginx path first (follow the existing deploy.sh pattern or manual copy).

Database migrations

If the PR includes a new migration script under scripts/:

cd /opt/huph
docker exec -i huph-postgres psql -U huph -d huph < scripts/migrate-<name>.sql

Verify:

docker exec huph-postgres psql -U huph -d huph -c "\d <table_you_changed>"

Migrations are additive only (no down migrations currently tracked). Plan schema changes carefully — if you break a column, you break forward.

Post-deploy smoke tests

Run these after deploying any service:

# Health checks
curl -s http://localhost:3101/health
curl -s http://localhost:3102/health
curl -sI http://localhost:47293/ | head -1

# API v1 surface
curl -s http://localhost:3101/api/v1/health/realtime | jq

# Lead capture smoke (if API changed)
# See: docs/dev/operations/lead-capture-smoke.en.md

# Realtime smoke (if realtime changed)
# See: docs/dev/operations/realtime-smoke.en.md

Open the admin dashboard in a browser and:

  1. Log in
  2. Open Conversations — should load without errors
  3. Open Leads v2 — should load with 30s polling
  4. Check the realtime indicator in the header — should show "Connected"

If anything fails, rollback immediately (see rollback.en.md).

Zero-downtime caveats

  • API container restart causes a brief gap (~2–5 s) where requests fail. Acceptable for small deploys; not acceptable during peak WA traffic. Consider time-of-day.
  • RAG container restart causes ~90 s of 503 from /health while models reload. During this window, any chat request that hits RAG will fail. Plan accordingly.
  • Admin systemctl restart is roughly seamless (Next.js restarts in <3 s, nginx buffers briefly).
  • No blue-green deploy currently — live restart is the only option.

What to do after a successful deploy

  1. Update the internal deploy log (Slack/wiki/whatever the team uses)
  2. Monitor Phoenix + Langfuse for anomaly spikes for ~30 minutes
  3. Watch the API container logs for unexpected errors:
docker-compose logs -f huph-api --tail 100
  1. Delete the pre-deploy snapshot files if stable:
rm /tmp/pre-deploy-state.txt /tmp/pre-deploy-sha.txt

Gotchas

  1. .env changes don't propagate without container restart. The container reads env at boot. After changing .env, do docker-compose up -d --no-deps <service> to recreate with new env.
  2. Systemd admin service name. The unit file is huph-admin.service. If you see Admin or admin in a tab, verify first — wrong name means the restart silently does nothing.
  3. npm run build for admin is required before systemctl restart. Without it, you restart the old build.
  4. Dify stack is not deployed via this flow. Dify lives in a separate docker-compose.dify.yml. Its deploy is manual and rare. Don't accidentally docker-compose down from the main compose file — that only covers the main HUPH stack, not Dify.
  5. deploy.sh is an older helper script. It still works but was written before the current docker-compose workflow. Prefer the manual commands above.

See also