In this tutorial we will deploy a LLM Chat-GPT like service on a GPU node on Jetstream. For experimentation purposes we are using the smaller and cheapest GPU node available on Jetstream, the g3.small which has a virtual GPU with 8GB of memory, and deploy the meta-llama/Llama-3.2-1B-Instruct model which is a 1.3B parameter model.
However the same instructions can be used to deploy any other model available on the Hugging Face model hub.
This tutorial is based on work by Tijmen de Haan, the author of Cosmosage.
Choose a model
Jetstream has GPU nodes with 4 NVIDIA A100 GPUs, a user can create a Virtual Machine with 1 entire GPU or a fraction of it.
The most important requirement is the GPU memory available to load the model parameters, Jetstream provides:
| Instance Type | GPU Memory (GB) |
|---|---|
| g3.small | 8 |
| g3.medium | 10 |
| g3.large | 20 |
| g3.xl | 40 |
So g3.xl is the largest available and gets an entire A100 GPU with 40GB of memory.
Therefore we need to make sure that the model we want to deploy fits in the available memory.
Llama 3.2 1B model uses about 6.5GB of memory, so it fits in the g3.small instance.
We also need to make sure the model has been fine-tuned for responding to text prompts, generally those models are marked as Instruct.
Create a Jetstream instance
Login to Exosphere, request a Ubuntu 24 g3.small instance, name it chat and ssh into it using either the SSH key or the passphrase generated by Exosphere.
Install miniforge
We will use Miniforge to create 2 separate Python environments, one for the Hugging Face model serving and one for the web interface.
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).shConfigure vllm to serve the model
Create the environment:
conda create -y -n vllm python=3.11
conda activate vllm
pip install vllmAs we are using a Llama model, we need specific authorization, you can login to Hugging Face and request access to the model on the model page.
Next we can serve the model:
huggingface-cli login
vllm serve "meta-llama/Llama-3.2-1B-Instruct" --max-model-len=8192If this starts with no error, we can kill it with Ctrl-C and create a service for it.
Create a file /etc/systemd/system/vllm.service with the following content:
[Unit]
Description=VLLM model serving
After=network.target
[Service]
User=exouser
Group=exouser
WorkingDirectory=/home/exouser
# Activating the conda environment and starting the service
ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate vllm && vllm serve 'meta-llama/Llama-3.2-1B-Instruct' --max-model-len=8192 --enforce-eager"
Restart=always
Environment=PATH=/home/exouser/miniforge3/envs/llm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Install]
WantedBy=multi-user.targetThen enable and start the service:
sudo systemctl enable vllm
sudo systemctl start vllmIn case of errors:
- Check the logs with
sudo journalctl -u vllm - Check the status with
sudo systemctl status vllm
You can also check the GPU usage with nvidia-smi:
Thu Oct 31 16:51:09 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.06 Driver Version: 535.183.06 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 GRID A100X-8C On | 00000000:04:00.0 Off | 0 |
| N/A N/A P0 N/A / N/A | 6400MiB / 8192MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 53332 C ...miniforge3/envs/vllm/bin/python3.11 6399MiB |
+---------------------------------------------------------------------------------------+Configure the chat interface
The chat interface is provided by Open Web UI.
Create the environment:
conda create -y -n open-webui python=3.11
conda activate open-webui
pip install open-webui
open-webui serveIf this starts with no error, we can kill it with Ctrl-C and create a service for it.
Create a file /etc/systemd/system/webui.service with the following content:
[Unit]
Description=Open Web UI serving
After=network.target
[Service]
User=exouser
Group=exouser
WorkingDirectory=/home/exouser
# Activating the conda environment and starting the service
ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate open-webui && open-webui serve"
Restart=always
Environment=PATH=/home/exouser/miniforge3/envs/open-webui/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Install]
WantedBy=multi-user.targetThen enable and start the service:
sudo systemctl enable webui
sudo systemctl start webuiConfigure web server for HTTPS
Finally we can use Caddy to serve the web interface with HTTPS.
Follow the instructions to Install Caddy
Modify the Caddyfile to serve the web interface:
sudo vim /etc/caddy/Caddyfileto:
chat.xxx000000.projects.jetstream-cloud.org {
reverse_proxy localhost:8080
}Where xxx000000 is the allocation code of your Jetstream instance.
Then reload Caddy:
sudo systemctl reload caddyConnect the model and test the chat interface
Point your browser to https://chat.xxx000000.projects.jetstream-cloud.org and you should see the chat interface.
Create an account, click on the profile icon on the top right and enter the “Admin panel” section, open “Settings” then “Connections”.
Under “OpenAI API” enter the URL http://localhost:8000/v1 and leave the API key empty.
Click on the “Verify connection” button, then to “Save” on the bottom.
Finally you can start chatting with the model!
Use a larger model using quantization
The weights of LLMs can be quantized to a lower precision to reduce the GPU memory required to run them, often larger models with quantization outperform smaller models with no quantization. Hugging Face has several quantized models, the most popular are GGUF models, but vllm has just experimental support for that format, so better search explicitely for a model “quantized for vllm”, for example hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4.
sudo systemctl stop vllm
vllm serve "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4" --max_model_len 4096 --gpu_memory_utilization 1 --enforce-eagerModify the systemd service replacing the relevant line with:
ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate vllm && vllm serve 'hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4' --max-model-len=4096 --gpu_memory_utilization 1 --enforce-eager"Then restart the service:
sudo systemctl daemon-reload
sudo systemctl start vllmCheck nvidia-smi, memory consumption should be about 7.5 GB.
Unfortunately we needed to also decrease max-model-len to fit in such a small GPU, so the model will only support 4096 tokens, so it would be best to deploy this model on a slightly larger Virtual Machine and increase the number of tokens.