How to Extend WebAssembly with Host Functions

By DarumaDocker，WasmEdge Contributor in charge of the development of WasmEdge-bindgen

WebAssembly was developed for the browser. It gradually gain popularity on the server-side, but a significant disadvantage is its incomplete functionality and capability. The WASI proposal was initiated to solve these problems. But the forming and implementation of a standard is usually slow.

What if you want to use a function urgently? The answer is to use the Host Function to customize your WebAssembly Runtime.

What is a Host Function

As the name suggests, a Host Function is a function defined in the Host program. For Wasm, the Host Function can be used as an import segment to be registered in a module, and then it can be called when Wasm is running.

Wasm has limited capability, but those can't be achieved with Wasm itself can be resolved with Host Function, which expanded Wasm functionality to a large extent.

WasmEdge‘s other extensions apart from standards are majorly based on Host Function, for example, WasmEdge‘s Tensorflow API is implemented with Host Function and thus achieving the goal of running AI inference with the native speed.

Networking socket is implemented with host function as well. Thus we can run asynchronous HTTP client and server in WasmEdge which compensate for the WebAssembly's disadvantage in networking.

Another example. Fastly uses Host Function to add HTTP Request and Key-value store APIs to Wasm which added the extension functions.

How to write a simple Host Function

Let's start with a simple example and see how to write a host function in a Go program.

Let's write a simple rust program. As usual,Cargo.toml is essential.

`Cargo.toml`

[package]
name = "rust_host_func"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib", "rlib"]

[dependencies]

Look at what Rust code looks like.

`lib.rs`

extern "C" {
	fn add(a: i32, b: i32) -> i32;
}

#[no_mangle]
pub unsafe extern fn run() -> i32 {
	add(1, 2)
}

The add function in the above program is declared in extern "C" . This is a Host Function. We use the following command to compile this Rust program to wasm:

cargo build --target wasm32-wasi --release

Then we use wasm2wat to check wasm file's import segment:

wasm2wat target/wasm32-wasi/release/rust_host_func.wasm | grep import

The export is as follows:

  (import "env" "add" (func $add (type 0)))

We can see that the add function is put in the import segment of the module with the default name env.

Next, let's look at how to use WasmEdge-go SDK to execute this wasm program.

`hostfunc.go`

package main

import (
	"fmt"
	"os"

	"github.com/second-state/WasmEdge-go/wasmedge"
)

func add(_ interface{}, _ *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// Add together the two parameters passed from wasm
	return []interface{}{params[0].(int32) + params[1].(int32)}, wasmedge.Result_Success
}

func main() {
	vm := wasmedge.NewVM()
	
	// Use the default name env to build the import objective
	obj := wasmedge.NewImportObject("env")

	// Build Host Function's parameter and return value type 
	funcAddType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		})
	hostAdd := wasmedge.NewFunction(funcAddType, add, nil, 0)
	
	// Add Host Function to import segment object
	// Note that the first parameter `add`  is the name of the external function defined in rust
	obj.AddFunction("add", hostAdd)

	// Register the import segment object
	vm.RegisterImport(obj)

	// Load, validate and instantiate the wasm program
	vm.LoadWasmFile(os.Args[1])
	vm.Validate()
	vm.Instantiate()

	// Execute the function exported by wasm and get the return value
	r, _ := vm.Execute("run")
	fmt.Printf("%d", r[0].(int32))

	obj.Release()
	vm.Release()
}

Compile and execute:

go build
./hostfunc rust_host_func.wasm

The program outputs 3 .

In this way, we have completed a very simple example of defining Function in Host and calling it in wasm.

Let's try to do something more interesting with the Host Function.

Passing complex types

Restricted by the data type in Wasm, Host Function can only pass a few basic types of data such as int32, which greatly limits the use cases of Host Functions. Is there any way for us to pass complex data types such as string data? The answer is yes, of course! Let's see how to do it with an example.

Here in this example, we need to count how many times google appears in the source code of the https://www.google.com web page. The example's source code is here.

Again, let's see the Rust code first. Surely there should beCargo.toml, but I skip it here.

`lib.rs`

extern "C" {
	fn fetch(url_pointer: *const u8, url_length: i32) -> i32;
	fn write_mem(pointer: *const u8);
}

#[no_mangle]
pub unsafe extern fn run() -> i32 {
	let url = "https://www.google.com";
	let pointer = url.as_bytes().as_ptr();

	// call host function to fetch the source code, return the result length
	let res_len = fetch(pointer, url.len() as i32) as usize;

	// malloc memory
	let mut buffer = Vec::with_capacity(res_len);
	let pointer = buffer.as_mut_ptr();

	// call host function to write source code to the memory
	write_mem(pointer);

	// find occurrences from source code
	buffer.set_len(res_len);
	let str = std::str::from_utf8(&buffer).unwrap();
	str.matches("google").count() as i32
}

In the code, we add 2 Host Functions:

fetch for sending HTTP requests to get the Webpage source code
write_mem for writing the webpage source code to wasm's memory

You may have seen that to pass a string in the Host Function, it is actually achieved by passing the memory pointer and length where the string is located. fetch receives two parameters, the pointer and byte length of the stringhttps://www.google.com .

After fetching the source code, fetch returns the byte length of the source code as the return value. After Rust allocates memory of this length, it passes the memory pointer to write_mem, and the host writes the source code to this memory, thereby achieving the purpose of returning a string.

The compiling process is the same as above and we will skip that. Next, let's see how to use WasmEdge-go SDK to execute the Wasm program.

`hostfun.go`

package main

import (
	"fmt"
	"io"
	"os"
	"net/http"

	"github.com/second-state/WasmEdge-go/wasmedge"
)

type host struct {
	fetchResult []byte
}

// do the http fetch
func fetch(url string) []byte {
	resp, err := http.Get(string(url))
	if err != nil {
		return nil
	}
	defer resp.Body.Close()
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil
	}

	return body
}

// Host function for fetching
func (h *host) fetch(_ interface{}, mem *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// get url from memory
	pointer := params[0].(int32)
	size := params[1].(int32)
	data, _ := mem.GetData(uint(pointer), uint(size))
	url := make([]byte, size)

	copy(url, data)

	respBody := fetch(string(url))

	if respBody == nil {
		return nil, wasmedge.Result_Fail
	}

	// store the source code
	h.fetchResult = respBody

	return []interface{}{len(respBody)}, wasmedge.Result_Success
}

// Host function for writting memory
func (h *host) writeMem(_ interface{}, mem *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// write source code to memory
	pointer := params[0].(int32)
	mem.SetData(h.fetchResult, uint(pointer), uint(len(h.fetchResult)))

	return nil, wasmedge.Result_Success
}

func main() {
	conf := wasmedge.NewConfigure(wasmedge.WASI)
	vm := wasmedge.NewVMWithConfig(conf)
	obj := wasmedge.NewImportObject("env")

	h := host{}
	// Add host functions into the import object
	funcFetchType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		})

	hostFetch := wasmedge.NewFunction(funcFetchType, h.fetch, nil, 0)
	obj.AddFunction("fetch", hostFetch)

	funcWriteType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{})
	hostWrite := wasmedge.NewFunction(funcWriteType, h.writeMem, nil, 0)
	obj.AddFunction("write_mem", hostWrite)

	vm.RegisterImport(obj)

	vm.LoadWasmFile(os.Args[1])
	vm.Validate()
	vm.Instantiate()

	r, _ := vm.Execute("run")
	fmt.Printf("There are %d 'google' in source code of google.com\n", r[0])

	obj.Release()
	vm.Release()
	conf.Release()
}

With an understanding of Rust code, this Go code is actually easy to understand. The key is to access Wasm memory:

mem.GetData(uint(pointer), uint(size)) Get the Webpage URL in the Wasm
mem.SetData(h.fetchResult, uint(pointer), uint(len(h.fetchResult))) write the webpage source code into the wasm memory

This example's compiling and execution steps are the same as the example before. The end result is:

There are 79 'google' in source code of google.com

Summary

With the above two examples, now you have a basic idea about how Host Function works.

Wasm has many limitations and developer's coding experience with it is not optimal，but as we enhance the dev tools and libs, there will be infinite future use cases for Wasm.

Please stay tuned for the WasmEdge Runtime updates. Thanks!

Join us in the WebAssembly revolution!

👉 Slack Channel: #wasmedge on CNCF slack channel

👉 Disocrd: Join WasmEdge org

👉 Mailing list: Send an email to WasmEdge@googlegroups.com

👉 Twitter: @realwasmedge

👉 Be a contributor: checkout our wish list to start contributing!