Getting Started with Yi-1.5-34B-Chat-16K

May 12, 2024 • 4 minutes to read

On May 20th, Yi released Yi-1.5-9B-Chat-16K and Yi-1.5-34B-Chat-16K, two advanced chat models developed by Yi on Hugging Face. Both models are part of the Yi-1.5 series, which is an improvement over its predecessor, enhancing abilities in areas like coding, math, reasoning, and instruction-following, while maintaining strong language understanding and commonsense reasoning skills.

Compared with the Yi-1.5-Chat, the Yi-1.5-9B-Chat-16k has a much longer context window, which means the model can hold longer background information and more complex instructions in the prompt. It can also remember much longer conversations. The Yi 9B model is known for its great performance among similarly sized models, and the 34B model matches or exceeds larger models in many benchmarks. These models are designed for high-quality conversation generation and are versatile across various applications. You can explore these models further on their respective Hugging Face pages for the 9B model and the 34B model.

In this article, taking Yi-1.5-9B-Chat-16K as an example, we will cover

  • How to run Yi-1.5-9B-Chat-16K on your own device
  • How to create an OpenAI-compatible API service for Yi-1.5-9B-Chat-16K

You can also try out Yi-1.5-34B-Chat-16K following these steps, just by changing the model name from “Yi-1.5-9B-Chat-16K” to “Yi-1.5-34B-Chat-16K”.

We will use LlamaEdge (the Rust + Wasm stack) to develop and deploy applications for this model. There is no complex Python packages or C++ toolchains to install! See why we choose this tech stack.

Run Yi-1.5-9B-Chat-16K on your own device

Step 1: Install WasmEdge via the following command line.

curl -sSf | bash -s -- --plugin wasi_nn-ggml wasmedge_rustls

Step 2: Download theYi-1.5-9B-Chat-16K model GGUF file. Since the size of the model is 6.3 GB so it could take a while to download.

curl -LO

Step 3: Download a cross-platform portable Wasm file for the chat app. The application allows you to chat with the model on the command line. The Rust source code for the app is here.

curl -LO

That's it. You can chat with the model in the terminal by entering the following command.

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-1.5-9B-Chat-16K-Q5_K_M.gguf \
  llama-chat.wasm \
  --prompt-template chatml \
  --ctx-size `16384`

The portable Wasm app automatically takes advantage of the hardware accelerators (eg GPUs) I have on the device. Here is a trick question I asked it.

I have 5 apples today. I ate 3 apples last week. How many apples do I have now?

If you had 5 apples today and ate 3 apples last week, then according to the information provided, you still have 5 apples now. The action of eating apples last week doesn't affect the number of apples you currently have today.

The Yi-1.5-9B-Chat-16K model has great logical reasoning capability.

Create an OpenAI-compatible API service for Yi-1.5-9B-Chat-16K

An OpenAI-compatible web API allows the model to work with a large ecosystem of LLM tools and agent frameworks such as, LangChain and LlamaIndex.

Download an API server app. It is also a cross-platform portable Wasm app that can run on many CPU and GPU devices.

curl -LO

Then, download the chatbot web UI to interact with the model with a chatbot UI.

curl -LO
tar xzf chatbot-ui.tar.gz
rm chatbot-ui.tar.gz

Next, use the following command lines to start an API server for the model. Then, open your browser to http://localhost:8080 to start the chat!

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-1.5-9B-Chat-16K-Q5_K_M.gguf \
  llama-api-server.wasm \
  --prompt-template chatml \
  --ctx-size 16384 \
  --model-name Yi-1.5-9B-Chat-16K

From another terminal, you can interact with the API server using curl.

url -X POST http://localhost:8080/v1/chat/completions \
  -H 'accept:application/json' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user", "content": "write a hello world in Rust"}], "model":"Yi-1.5-9B-Chat-16k"}'

That’s all. WasmEdge is easiest, fastest, and safest way to run LLM applications. Give it a try!

Talk to us!

Join the WasmEdge discord to ask questions and share insights.

Any questions getting this model running? Please go to second-state/LlamaEdge to raise an issue or book a demo with us to enjoy your own LLMs across devices!

LLMYiAI inferenceRustWebAssembly
A high-performance, extensible, and hardware optimized WebAssembly Virtual Machine for automotive, cloud, AI, and blockchain applications