Serving Sites with scorpiox-server

You want to put a site or an API on a box. The traditional answer is a stack: a web server that only proxies, an application runtime it hands work to, an auth middleware package, a deployment pipeline, and a reverse-proxy config so TLS terminates somewhere sensible. Each layer is a product with its own config language, and every new site means wiring all of them together again.

SCORPIOX CODE ships a different answer: scorpiox-server, a single native binary that is the web server and the application runtime and the auth gate and the deployment mechanism. The routing is your file layout. A git push deploys. JWT validation is a config key. And every page you build serves two audiences at once — HTML for the human in the browser, JSON for the machine in the loop.

This is the same server that powers the product's own public website, and the same one SCORPIO BOT supervises as its API tier. It runs sites in production every day.

Docs for SCORPIOX CODE @ 77c49df.

The whole idea in one line: drop scorpiox-server next to a folder of Python scripts or native executables, and the file names become the routes — with a mandatory HTML + JSON contract on every page, git-push deployment, built-in JWT authentication, an MCP mode that turns your scripts into agent tools, and a reverse-tunnel mesh for sites that live behind NAT — all in one zero-dependency binary.

The mental model: routing is the file layout

There is no route table to maintain. Give the server a script directory and a URL prefix, and every executable or .py file in that directory becomes a route whose name is the file name:

/var/site/
  index.py        ->  GET  /                     (the front page)
  install.py      ->  GET  /install
  api-reference.py -> GET  /api-reference
  _fallback.py    ->  *    (everything else)

Hit /{prefix}{name} and the server finds a handler named name: a native executable first (a file with the execute bit set on Unix, name.exe on Windows), then a name.py script run by python3. With multiple directories configured, they are searched in order and the first match wins — useful for a shared library of scripts shadowed by site-specific overrides. Name conflicts between directories are detected and reported at startup.

A few conventions make the whole system work:

• index is the default page. A request to the bare prefix (with or without the trailing slash) runs index or index.py.
• _fallback is the catch-all. When no script matches, the server runs _fallback or _fallback.py — your 404 page, your dynamic router, whatever you want. The product's own website uses a single _fallback.py that routes /tools/{name} pages and renders 404s.
• The route namespace is flat. Names may contain letters, digits, -, _, and . — but not /. There are no subdirectories in routing. Scripts that need internal paths (like a docs tree) read PATH_INFO and route inside themselves.
• Traversal is blocked twice. Names containing .. are rejected, and every resolved file is verified to actually live inside its script directory before anything executes.

If no script matches and there is no fallback, the request gets a clean 404 with a JSON error body.

Start it

scorpiox-server                     # default/configured port (8080)
scorpiox-server -p 3000             # specific port
scorpiox-server -e SERVER_SCRIPT_DIR=/var/site -e SERVER_ROUTE_PREFIX=/
scorpiox-server -r https://git.example.com/org/site.git   # git deploy mode
scorpiox-server -h                  # full help

-e KEY=VAL overrides any configuration key for this process — repeatable, and values for sensitive keys (secrets, tokens) are masked in the startup log. Configuration otherwise comes from the same scorpiox-env.txt cascade the rest of SCORPIOX CODE uses (see Configuration and Profiles), so a server and your agent share one configuration story.

On startup the server prints its banner: port, PID, request/response limits, the route prefix, and each script directory with an accessibility check. Missing directories are warned about; a server with no accessible directories refuses to start rather than serve 404s.

The dual contract: HTML for humans, JSON for machines

This is a requirement of sites built on the server, not a nice-to-have: every page must answer a browser and a machine from the same URL. Humans get HTML. Automation, agents, monitors, and other tools get JSON they can consume without scraping. The HTML view is what a person reads; the JSON view is what makes the same page actionable in a loop — one URL, both audiences, no scraping layer in between.

The product's own site does exactly this. The landing page serves rendered HTML to a browser and a plain-text installer script to PowerShell (user-agent sniffing at the CGI layer). The /version endpoint answers JSON. A Swagger UI route serves interactive HTML at /swagger and the same API surface as a machine-readable OpenAPI document at /swagger/v1/swagger.json. SCORPIO BOT's API takes it further with an explicit ?format=json switch on every route, so a human in the dashboard and a script in a loop use identical URLs.

Your scripts implement the contract however suits the page — content negotiation on the Accept header, a ?format=json query parameter, a parallel /api/... route — but build it in from the first commit. Retrofitting "make this page machine-readable" onto an HTML-only site is the mistake this contract exists to prevent. An agent that can read your site is an agent that can act on it.

What a script receives: the CGI environment

Scripts run in a full CGI environment. The server sets everything a standard CGI program expects, plus its own additions:

The query-parameter-to-environment mapping is the workhorse: a script reads os.environ.get('platform') and never parses a query string. The product's own install router does exactly this — one ?platform= parameter, read straight out of the environment.

POST bodies are handled in one of two ways:

• Buffered (bodies up to 512 KB): the body is written to a temp file and exported as POST_BODY_FILE, and also piped to the script's stdin.
• Streamed (larger bodies, or chunked requests): the body streams straight from the socket to the script's stdin with no temp file and no buffering, with SX_STREAMING=1 set. A 1 MB upload round-trips through this path byte-for-byte — the mesh test suite verifies exactly that with a SHA256 equality check.

The script's working directory is its own directory. stdout and stderr are merged. The script's stdout becomes the HTTP response.

What a script sends back

Script output is CGI: optional headers, a blank line, then the body.

Content-Type: text/html; charset=utf-8
Status: 200 OK
<html>...

• Status: sets the HTTP status code (values outside 100–599 fall back to 200).
• Content-Type: sets the response type (default text/html).
• Everything else — Location:, Set-Cookie:, your own headers — is forwarded to the client. Redirects and cookies are pure script territory.
• If the output has no recognizable CGI header block, the whole output is served as text/html with status 200. A script that just prints HTML works with zero ceremony.

Responses also carry permissive CORS headers on every reply (Access-Control-Allow-Origin: *, all methods, all headers, credentials allowed), so browser-based clients on other origins work without extra configuration. Tighten this at a TLS-terminating proxy if your deployment needs stricter origins.

Streaming (SSE): if a script declares Content-Type: text/event-stream, the server switches from buffering to streaming — each data: chunk is forwarded the moment the script flushes it, on all platforms. Server-sent events, live logs, progress feeds, token-by-token output: write a normal script that flushes, and the client gets it live. SSE streams are subject to the idle timeout (below), not a total-duration cap — a stream that keeps producing can run as long as it keeps producing.

Limits and lifecycle

Three keys govern resource behavior, all with defaults that suit most deployments:

The timeout is idle-based, not wall-clock: the deadline resets on every chunk the script produces. A report that streams progress for an hour is fine; a script that hangs silently for 300 seconds is killed (SIGTERM, then SIGKILL after a 100 ms grace) and whatever output was captured is returned. Bump SERVER_SCRIPT_TIMEOUT for scripts with long silent phases — SCORPIO BOT raises it to 900 seconds for exactly that reason.

Concurrency is a process per request on Unix (fork) and a thread per request on Windows. That is the entire concurrency story — no worker pool to size, no event loop to starve. The listen backlog is 256.

Built-in endpoints

Two health/utility endpoints ship in the binary itself, plus optional favicon serving:

OPTIONS requests get a real answer too: 204 No Content by default, unless a script named options (or the fallback) exists, in which case the request is routed to it — CORS preflight handling you can customize in script.

Git deploy mode: push to deploy

This is the deployment story, and it needs no pipeline:

scorpiox-server -r https://git.example.com/org/site.git
scorpiox-server -r https://git.example.com/org/site.git -b staging --poll 30

What happens:

1. The server clones the repository (--depth 1) into its cache directory and serves the clone as the script directory.
2. In git mode the route prefix defaults to / — a site repository is the site, no prefix ceremony.
3. A background poll thread compares the local HEAD with the remote branch head (git ls-remote) every SERVER_GIT_POLL_INTERVAL seconds (default 10). On a change it pulls (fetch + reset --hard FETCH_HEAD) and the new code is live on the next request.

Configuration for the mode:

Operational details worth knowing: an existing clone is pulled rather than re-cloned on restart; a failed pull logs a warning and keeps serving the previously working code rather than taking the site down; a failed initial clone aborts startup. Git operations never touch a shell — they are spawned as argument vectors directly, eliminating the command-injection class of bug. And the obvious security note: deploying whatever the branch contains is exactly as trustworthy as whoever can push to that branch. Treat write access to the deploy branch as production access.

The loop is deliberately simple — no webhooks to receive, no runners to register. git push is the deploy button, and the site follows the branch within one poll interval. For most sites that is the entire CI/CD story.

Authentication: built-in JWT

Set one key and every route can require authentication:

SERVER_JWT_SECRET=<hmac-secret>        # or a file path: /etc/scorpiox/jwt.key
SERVER_JWT_PROTECT=/admin,/api/private
SERVER_JWT_LOGIN_URL=https://login.example.com
SERVER_JWT_COOKIE=sx_token

The server validates HMAC-SHA256 JWTs itself — no library, no middleware package. Tokens arrive as an Authorization: Bearer header or, if SERVER_JWT_COOKIE is set, from the named cookie (the header is checked first). Verification is constant-time, expired tokens (exp) are rejected, and optional iss/aud checks pin tokens to your issuer and audience. An empty SERVER_JWT_SECRET disables validation entirely — scripts then decide their own authentication, which is a legitimate mode for sites that handle auth internally.

Protecting routes: SERVER_JWT_PROTECT is a comma-separated list of route prefixes that require a valid token. Unauthenticated requests are rejected based on what the client is: a browser (its Accept header includes text/html) gets a 302 redirect to SERVER_JWT_LOGIN_URL; anything else gets 401 with a JSON error body. One mechanism, both audiences.

Public exemptions: SERVER_JWT_PUBLIC exempts routes from protection so a site can protect / and still serve assets or a health check. Matching is by whole path segment — /assets covers /assets and /assets/x, but not /assetsX — and a valid token presented on a public route is still validated and exported to the script. Protection and exemption compose instead of fighting.

Identity reaches your scripts. Every request exports what it knows about the caller as environment variables:

So SERVER_JWT_CLAIMS is your authorization seam: mint tokens with a permissions claim, map it to an environment variable, and have scripts check it. Roles, tiers, scopes — the server handles verification and transport; your script decides what the identity may do.

Network exposure controls

Two more layers ship in the binary:

• IP allowlist: SERVER_IP_WHITELIST=203.0.113.7,10.0.0.0/8 restricts script routes and built-in endpoints to listed IPs or CIDR ranges. Empty means allow all. Behind a reverse proxy, the check consults X-Forwarded-For first, then X-Real-IP, then the socket peer — the first available proxy header wins, with no fallback to the peer address when a proxy header is present. Non-matching clients get 403 Forbidden. Note the boundary: mesh endpoints (/ws/join, /mesh/nodes, /node/<id>/) are intercepted at the accept loop before the allowlist runs, which is exactly why the mesh has its own JWT/PSK authentication layer — the allowlist protects your routes, the mesh auth protects the tunnel.
• TLS: the server speaks plain HTTP and expects TLS termination in front of it — Caddy, nginx, IIS, a cloud load balancer, whatever your stack already runs. This is the same pattern the product's own infrastructure uses, and it keeps the binary free of certificate management. Terminate TLS at the front, point it at scorpiox-server, done.

The two compose into a sensible default posture: TLS terminator in front, JWT on the sensitive prefixes, IP allowlist when the audience is known. Everything is config keys — no policy files, no middleware wiring.

MCP mode: your scripts become agent tools

With --mcp, the served folder gains a second interface: every script becomes an MCP tool callable by any MCP client — including other SCORPIO CODE instances and the llama.cpp Web UI.

scorpiox-server --mcp -p 8888
scorpiox-server -r https://git.example.com/org/tools.git --mcp
scorpiox-server --mcp --name myserver

The endpoint is POST /mcp, speaking MCP JSON-RPC 2.0 over Streamable HTTP: initialize (protocol version 2024-11-05, your serverInfo name from --name or MCP_SERVER_NAME), tools/list, tools/call, and proper JSON-RPC errors for unknown methods. A tool call forks the script, passes the arguments as JSON on stdin, and returns stdout as the tool result — with isError flagged when the exit code is non-zero.

Scripts can self-describe with a --schema flag:

# deploy_report --schema prints:
#
# Build a deployment report for an environment.
# environment:string:staging or production:required
# since:string:ISO date, e.g. 2026-09-01:optional

Line one is the tool description; each following line is a parameter as name:type:description:required|optional. At startup the server probes each script with this flag (5-second timeout), builds real JSON schemas from what answers, and registers up to 128 tools. Tool names must be alphanumeric plus _ and -, names starting with _ are skipped (internal helpers stay internal), and MCP_TOOL_EXCLUDE=helper-*,setup,*.bak filters out anything that should never be advertised as a tool.

The same deployment modes apply: MCP mode over a git-deployed repo means your tool folder gets updates by git push, same as a website. For the client side of MCP — connecting to other servers, OAuth flows, allow/deny lists — see Native MCP 2.0 and OAuth 2.1.

One operational caution, stated plainly: a tool call executes a script with your privileges, and /mcp answers whatever client can reach the port. Combine SERVER_IP_WHITELIST, JWT protection via your TLS terminator's auth layer, and MCP_TOOL_EXCLUDE so nothing dangerous gets advertised by accident.

The mesh: sites behind NAT, without open ports

scorpiox-server also speaks an inverted transport — a reverse-tunnel mesh that solves the oldest deployment problem there is: how do you serve a site on a machine that has no open inbound ports?

A hub accepts worker connections and routes public traffic to them:

scorpiox-server --hub -p 8080

A worker dials out to the hub and serves traffic through that connection:

scorpiox-server --connect wss://mesh.example.com/ws/join --id build-node

Run the worker with no -p flag and it opens zero inbound ports — it only dials out. (Give it a port and it also listens locally, which is how a machine can be both a mesh worker and a normally-reachable server.) The hub receives GET /node/<id>/<path>, forwards it down the worker's existing tunnel, the worker executes it through its normal routing (same scripts, same CGI environment, same JWT identity variables), and the response streams back — including POST bodies and SSE. A worker behind home NAT, a container, or a corporate firewall becomes a reachable site as long as it can make outbound connections.

Hub endpoints, all intercepted alongside its normal local routes:

Connection hygiene is handled: 30-second keepalive pings, automatic reconnection with exponential backoff (1 s doubling to a 30 s ceiling, auth failures jumping straight to the ceiling), and a dead worker answers 502 node offline while its registration — and its owner — are retained for when it returns.

Three authentication modes, selected automatically by configuration:

JWT and PSK combine — the token provides identity, the key is an extra gate. Client-supplied X-Mesh-* headers are stripped at the hub and replaced with the real authenticated identity (X-Mesh-Node, X-Mesh-User, X-Mesh-Email), so a script behind a worker sees trustworthy caller identity it cannot be spoofed into believing.

Workers find their token in order: --token, SERVER_MESH_TOKEN, the SX_TOKEN environment variable, then the ~/.scorpiox/auth.json file written by scorpiox-bot login — so a machine that has already logged into SCORPIO+ needs no token plumbing at all. wss:// is supported with certificate verification against the system CA store (or SERVER_MESH_CA_FILE); --insecure skips verification, loudly. The hub itself always speaks plaintext with TLS terminated in front of it — the product's own hub at mesh.scorpiox.net runs exactly this way behind Caddy.

The cleanest way to run all of this: scorpiox-bot --connect supervises a mesh worker for you — wiring the token, the node id, the parent-death shutdown so an orphaned tunnel can never outlive its supervisor — and shows up in the SCORPIO BOT fleet dashboard. See Remote Agent Control and Fleet Management with SCORPIOX BOT.

The shared-library form factor

The same source also builds scorpiox-server-dll, a shared library exposing web_server_start, web_server_stop, web_server_status, and web_server_get_config for embedding in another application via P/Invoke or FFI. Point it at a JSON config (root_dir, route_prefix, port) and your .NET (or anything-with-FFI) application hosts a full script-serving web server in-process — no side process, no port juggling between apps.

This is worth knowing about even if you never use it, because it explains a design property: the HTTP handling is a component, not a monolith. The embedded form runs its own accept thread and dispatches requests through the exact same pipeline the standalone binary uses — the same routing, CGI environment, JWT gate, and limits. The code that serves code.scorpiox.net can live inside your process.

Against the mainstream stacks

Here is what an operator would otherwise assemble, and what each piece costs:

The honest framing: those stacks are excellent at the things they were built for, and a large organization with dedicated platform teams should keep them. What scorpiox-server changes is the floor. The floor for serving a script-backed site with authentication, deployment, and machine-readable responses drops from five coordinated products and their wiring to one binary, one config file, and one git remote. And when a site outgrows the floor — when you genuinely need nginx's raw proxy throughput or IIS's Windows integration — the TLS-terminator pattern means the mainstream stack slots in front of scorpiox-server without displacing it.

The operator checklist

Bringing a site up, end to end:

# 1. Write the site — scripts in a folder
mkdir -p /var/site && cd /var/site
cat > index.py <<'EOF'
#!/usr/bin/env python3
print("Content-Type: text/html")
print()
print("<h1>It works</h1>")
EOF
# 2. Configure
cat > /etc/scorpiox/scorpiox-env.txt <<'EOF'
SERVER_PORT=8080
SERVER_SCRIPT_DIR=/var/site
SERVER_ROUTE_PREFIX=/
SERVER_JWT_SECRET=/etc/scorpiox/jwt.key
SERVER_JWT_PROTECT=/admin
SERVER_JWT_LOGIN_URL=https://login.example.com
SERVER_IP_WHITELIST=203.0.113.0/24
EOF
# 3. Run — or deploy by git and let it follow the branch
scorpiox-server -r https://git.example.com/org/site.git -b main
# 4. Terminate TLS in front, point /api/ping at your monitor

Every key above is documented in the shipped scorpiox-env.txt and read through the standard cascade — machine-wide, user, project, and profile tiers all work, so a staging server and a production server can be two profiles of the same file. See Configuration and Profiles.

Gotchas

• The default route prefix is not /. The compiled-in default is /api/platform/websites/ — a legacy integration path. Sites almost always want SERVER_ROUTE_PREFIX=/. Git deploy mode defaults to / for exactly this reason; bare mode does not. If your routes 404 and the banner shows a long prefix, this is why.
• /api/ping, /api/otp, and /favicon.ico are reserved. A script named api will not shadow /api/ping — the built-in endpoints are matched before script routing. Plan around them.
• Query parameters become environment variables with their literal names. A query parameter named PATH_INFO or HTTP_COOKIE will shadow the CGI variable of the same name — query-derived values are set last and win. Use distinctive parameter names so a request like ?content_length=1 cannot poison what your script reads.
• The idle timeout is not a total timeout. A script that prints one dot every 299 seconds runs forever. If you need wall-clock limits, enforce them in the script.
• SERVER_JWT_PROTECT and SERVER_JWT_PUBLIC match differently. PROTECT is a raw prefix match — /admin also protects /adminX. PUBLIC is whole-segment — /assets exempts /assets/anything but not /assetsX. A trailing slash in either value is trimmed, so /assets/ behaves the same as /assets. If you protect /admin, protect against lookalike paths too (use /admin/ and a route that never starts with the same letters).
• A valid token on a public route is still validated and exported. Public means not required, not ignored. Scripts can rely on X_USER_ID on public routes when a token happens to be present — and should check X_AUTHENTICATED before trusting it.
• Git deploy mode serves the branch, not a release artifact. A bad push goes live within one poll interval. Protect the branch, or use a deploy branch you push to deliberately.
• The response cap is a buffer cap, not a streaming cap. SSE streams bypass the response cap (they stream), but a buffered script that would emit 300 MB gets truncated at SERVER_MAX_RESPONSE_MB. Raise the cap or switch the script to streaming output.
• /mcp has no JWT gate of its own. MCP tool calls run scripts with your privileges. Protect the port with SERVER_IP_WHITELIST or terminate auth at your TLS proxy before exposing it beyond a trusted network.
• Mesh workers need outbound connectivity only — and that is also their failure mode. A worker that can no longer reach the hub disconnects; the hub returns 502 node offline for its routes until it reconnects. Design clients to tolerate transient 502s on /node/ routes.
• The hub is plaintext by design. Do not "fix" this by putting certificates on the hub — put a TLS terminator (Caddy, nginx, a load balancer) in front, exactly like the product's own mesh does.

Related

• Native MCP 2.0 and OAuth 2.1 — the client side of MCP, and more on --schema self-description.
• Remote Agent Control and Fleet Management with SCORPIOX BOT — the fleet layer that supervises scorpiox-server as its API tier, and the ?format=json machine contract in practice.
• Configuration and Profiles — the cascade every SERVER_* key is read through.
• Privacy Architecture and Zero Data Collection — what the server writes to disk, and what never leaves your machine.