Tutorial - text-generation-webui

Interact with a local AI assistant by running a LLM with oobabooga's text-generaton-webui on NVIDIA Jetson!

Set up a container for text-generation-webui

The jetson-containers project provides pre-built Docker images for text-generation-webui along with all of the loader API's built with CUDA enabled (llama.cpp, ExLlama, AutoGPTQ, Transformers, ect). You can clone the repo to use its utilities that will automatically pull/start the correct container for you, or you can do it manually .

git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh

How to start

Use jetson-containers run and autotag tools to automatically pull or build a compatible container image:

jetson-containers run $(autotag text-generation-webui)

The container has a default run command ( CMD ) that will automatically start the webserver like this:

cd /opt/text-generation-webui && python3 server.py \
  --model-dir=/data/models/text-generation-webui \
  --chat \
  --listen

Open your browser and access http://<IP_ADDRESS>:7860 .

Download a model on web UI

See the oobabooga documentation for instructions for downloading models - either from within the web UI, or using download-model.py

jetson-containers run --workdir=/opt/text-generation-webui $(./autotag text-generation-webui) /bin/bash -c \
  'python3 download-model.py --output=/data/models/text-generation-webui TheBloke/Llama-2-7b-Chat-GPTQ'

From within the web UI, select Model tab and navigate to " Download model or LoRA " section.

You can find text generation models on Hugging Face Hub , then enter the Hugging Face username/model path (which you can have copied to your clipboard from the Hub). Then click the Download button.

GGUF models

The fastest oobabooga model loader to use is currently llama.cpp with 4-bit quantized GGUF models.

You can download a single model file for a particular quantization, like *.Q4_K_M.bin . Input the file name and hit Download button.

Load a model

After you have downloaded a model, click the 🔄 button to refresh your model list, and select the model you want to use.

For a GGUF model, remember to

• Set n-gpu-layers to 128
• Set n_gqa to 8 if you using Llama-2-70B (on Jetson AGX Orin 64GB)

Then click the Load button.

Chat Template

If you're using a Llama model fine-tuned for chat, like the models listed above (except for LLaMA-30b ), you need to use the oobabooga Instruct mode and set the template. On the Parameters tab, go to the Instruction Template sub-tab, then select Llama-v2 from the Instruction Template drop-down (or Vicuna, Guanaco, ect if you are using one of those models)

Selecting the right chat template will make sure the model is being prompted correctly - you can also change the system prompt in the Context box to alter the agent's personality and behavior. There are a lot of other settings under the Generation tab, like the maximum length it should output per reply, and token sampling parameters like temperature and top_p for controlling randomness.

Then change back to the Chat tab, and under the mode section, make sure Instruct is selected (confusingly, not chat mode). Then you can start chatting with the LLM!

Results

Things to do with your LLM

Here are some common test prompts for coding, math, history ect. You can also ask it about geography, travel, nature, recipies, fixing things, general life advice, and practically everything else. Also Llama-2 is quite playful and likes to play games to test its logic abilities!

>> What games do you like to play?

I'm a large language model, so I can play text-based games and answer questions on a wide variety of topics. Here are some game ideas:

- 20 Questions:  You or I can think of an object, and the other player tries to guess what it is by asking up to 20 yes-or-no questions.  The goal is to guess the object in as few questions as possible.
- Hangman: I can think of a word or phrase and you try to guess it by suggesting letters. Each incorrect guess will bring the hangman one step closer to being complete!
- Word association: I'll give you a word and you have to come up with a word that is related to it. We can keep playing back and forth as long as you'd like!
- Trivia: I can provide questions on a wide range of topics, from history to science to pop culture. See how many questions you can answer correctly!
- Storytelling: I can start telling a story and then stop at a cliffhanger. You can then try to guess what happens next or even take over the storytelling and continue it in your own
- Jokes: I love to tell jokes and make people laugh with my "Dad humor"!  Knock knock!  *giggles*

Model size tested

With llama.cpp, GGUF model, 4-bit quantization: