ansible-roles-sglang

Deploys the SGLang OpenAI-compatible inference server as a GPU-enabled, rootless Podman service. Its defaults map the runtime settings from SGLang's upstream docker/compose.yaml: host networking and IPC, privileged GPU access, Hugging Face cache persistence, the server health check, and the specified ulimits.

Host configuration

Podman and NVIDIA CDI must be configured on the managed host. A recent NVIDIA Container Toolkit creates the CDI specification automatically; otherwise, generate it with:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list

As in the llamacpp role, sglang_nvidia_start_service installs a small host oneshot unit by default so NVIDIA device nodes exist before rootless services start after boot. Set it to false if the host already guarantees that.

The default host network and privileged mode follow upstream because they are required for RDMA. The server consequently listens directly on 0.0.0.0:30000; put authentication and TLS in a reverse proxy or override sglang_host before deploying it on an untrusted network.

Role configuration

---
sglang_user: llm
sglang_model: meta-llama/Llama-3.1-8B-Instruct
sglang_hf_token: "{{ vault_sglang_hf_token }}"
sglang_gpus: all

# Each item is one command argument. This example uses two GPUs.
sglang_extra_args:
  - --tp
  - "2"

The Hugging Face cache persists under ~/.config/sglang/cache/huggingface/ for the service user. Large model data is downloaded directly by SGLang rather than copied through Ansible.

For ModelScope, add its cache bind mount and enable it through the deployment environment:

---
sglang_extra_volumes:
  - "/srv/modelscope:/root/.cache/modelscope:Z"
sglang_deploy_env:
  SGLANG_USE_MODELSCOPE: "true"

Deployment and removal

Add sglang to deploy/actual/group_vars/all/services.yml, regenerate the generated playbook, and put the target host in the sglang inventory group. Then deploy:

./run.sh services actual
./run.sh deploy actual --tags sglang --limit somehost

Stop the generated user service without removing its data:

systemctl --user stop container-sglang.service

Remove the container and systemd unit while retaining the persistent cache:

./run.sh deploy actual --tags sglang \
  --extra-vars "deployment_state=absent" \
  --limit somehost