What kind of bird?

Starlark (formerly known as Skylark) is a Python-like language originally developed for the Bazel assembly system, which later branched out through interpreters for Go and Rust.

The language favors its use as a configuration tool, however, thanks to a well-written Go interpreter and a detailed description of the specification, it can be used as a programming language embedded in an application – for example, when you want to let the user interact with the application logic object, but do not want to constantly spawn entities under almost the same key due to almost the same parameters.

I could not find sufficiently complete tutorials during my work, so I had the idea to write a small article on this topic. In the article, we will walk you through working with Starlark in Go, from the simplest script launch to adding a new built-in type.

Starting small

To start work, it is enough to do two things – write the source script and execute it:

# hello.star
print("Hello from Starlark script")

// main.go
package main

import "go.starlark.net/starlark"

func main() {
	_, err := starlark.ExecFile(&starlark.Thread{}, "hello.star", nil, nil)
	if err != nil {
		panic(err)
	}
}

ExecFile loads the source file (or source code if the function’s third argument is not nil), parses and executes the code in the specified thread, after which it returns a dictionary with objects of the global scope of the script – variables, functions. The global scope is frozen, which means that any attempt to change the global definitions will cause a runtime error.

We launch it, we get:

$ go run main.go
Hello from Starlark script

A good start, however, there is little benefit from this – the point is that the script should be considered not as a separate program, but as a module, the functions of which we are going to call as necessary.

Let’s try another way:

# hello.star
def hello():
    print("Hello from Starlark script")

Re-running the code on Go and… nothing happens (as expected) because we only declared the function and didn’t call it. So, you need to call it from the main program:

// main.go
package main

import "go.starlark.net/starlark"

func main() {
    // выделим поток, в котором будем загружать и выполнять скрипт
    thread := &starlark.Thread{}
	globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
	if err != nil {
		panic(err)
	}

	_, err = starlark.Call(thread, globals["hello"], nil, nil)
	if err != nil {
		panic(err)
	}
}

Result:

$ go run main.go
Hello from Starlark script

globals this is the same dictionary with global variables and functions mentioned earlier. Through Call we call our function hello() by name, getting it from the dictionary. Positional and named arguments can be passed as the third and fourth arguments to the function.

We transmit and receive values

“Out of the box” Starlark has eleven built-in types (and a few more available modules, but not about them now):

• None, analog nilwhich is used when the lack of meaning is to be expressed

• Boolean, logical True or False

• Integer type combines signed and unsigned integers

• Float, a floating point number

• String, a string in UTF-8

• List, list, variable sequence of values

• Tuple, a tuple like a list, only immutable (but the values ​​contained in the tuple can be changed)

• Dictionary, a key-value dictionary, only hashable types are supported as keys.

• Set uses a hash table under the hood, so the requirement for values ​​is the same as for keys in a dictionary; A Go-specific type that requires a special flag to be set to use

• Function, functions defined in Starlark code

• Built-in function, a separate type for functions (or methods, more on that later) implemented in Go

On the Go side, all types must implement the Value interface, except for type-specific interfaces like Callable for functions – you’ll need this information when writing your types.

So, let’s try to pass something to our function:

# hello.star
def hello(message):
    print(message)

// main.go
package main

import "go.starlark.net/starlark"

func main() {
	thread := &starlark.Thread{}
	globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
	if err != nil {
		panic(err)
	}

	// здесь готовим позиционные аргументы для вызываемой функции
	args := starlark.Tuple{
		starlark.String("Hello from Golang"),
	}
	_, err = starlark.Call(thread, globals["hello"], args, nil)
	if err != nil {
		panic(err)
	}
}

Result:

$ go run main.go
Hello from Golang

In this way, any reasonable number of arguments can be passed to the called function. It is worth noting that an attempt to transfer more or less arguments is a script execution time error – if three arguments are specified in the signature, three are transferred.

Let’s try to get something from our function. Let’s write the addition of numbers as the simplest example:

# hello.star
def sum(x, y):
    return x + y

// main.go
package main

import "go.starlark.net/starlark"

func main() {
	thread := &starlark.Thread{}
	globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
	if err != nil {
		panic(err)
	}

    // здесь готовим позиционные аргументы для вызываемой функции
	args := starlark.Tuple{
		starlark.MakeInt(42),
		starlark.MakeInt(451),
	}
	result, err := starlark.Call(thread, globals["sum"], args, nil)
	if err != nil {
		panic(err)
	}
	print(result.String()) // распечатаем результат
}

Let’s start:

$ go run main.go
493

In variable reslut is stored some the result of the called function. Now it turned out to be a translation Value in a line, but in real use you need to cast the interface to the required type. As an example:

func toGoValue(starValue starlark.Value) (any, error) {
	switch v := starValue.(type) {
	case starlark.String:
		return string(v), nil
	case starlark.Bool:
		return bool(v), nil
	case starlark.Int: // int, uint both here
		if value, ok := v.Int64(); ok {
			return value, nil
		}

		if value, ok := v.Uint64(); ok {
			return value, nil
		}

		return nil, errors.New("unknown starlark Int representation")
	case starlark.Float:
		return float64(v), nil
	case *starlark.List:
		slice := []any{}
		iter := v.Iterate()
        defer iter.Done()
		var starValue starlark.Value
		for iter.Next(&starValue) {
			goValue, err := toGoValue(starValue)
			if err != nil {
				return nil, err
			}
			slice = append(slice, goValue)
		}
		return slice, nil
	case *starlark.Dict:
		datamap := make(map[string]any, v.Len())
		for _, starKey := range v.Keys() {
			goKey, ok := starKey.(starlark.String)
			if !ok { // datamap key must be a string
				return nil, fmt.Errorf("datamap key must be a string, got %v", starKey.String())
			}

			// since the search is based on a known key, 
			// it is expected that the value will always be found
			starValue, _, _ := v.Get(starKey)
			goValue, err := toGoValue(starValue)
			if err != nil {
				return nil, err
			}
			
			datamap[goKey.String()] = goValue
		}
		return datamap, nil
	default:
		return nil, fmt.Errorf("%v is not representable as datamap value", starValue.Type())
	}
}

A small note: in the above code, it is worth paying attention to the work with the iterator – when it is no longer needed, you need to explicitly call Done().

We add a new type

The Starlark interpreter allows you to extend the language by adding new types – it is enough to implement the Value interface.

Let’s imagine that we have a type, let it be a user, a synthetic example:

type User struct {
	name string
	mail *mail.Address
}

// конструктор пригодится позже
func NewUser(name, address string) (*User, error) {
	mail, err := mail.ParseAddress(address)
	if err != nil {
		return nil, err
	}

	if len(name) == 0 {
		return nil, errors.New("name required")
	}

	return &User{name: name, mail: mail}, nil
}

func (u *User) Rename(newName string) {
	u.name = newName
}

func (u *User) ChangeMail(newMail string) error {
	mail, err := mail.ParseAddress(newMail)
	if err != nil {
		return err
	}
	u.mail = mail
	return nil
}

func (u *User) Name() string {
	return u.name
}

func (u *User) Mail() string {
	return u.mail.String()
}

and we want to make it available in Starlark. This will require a wrapper type with the appropriate methods:

var _ starlark.Value = &StarlarkUser{}

type StarlarkUser struct {
	user *User
}

func (e *StarlarkUser) String() string {
	return fmt.Sprintf("name: %v, mail: %v", e.user.Name(), e.user.Mail())
}

func (e *StarlarkUser) Type() string {
	return "user"
}

// для упрощения, не будем заморачиваться с реализацией методов ниже
func (e *StarlarkUser) Freeze() {}

func (e *StarlarkUser) Truth() starlark.Bool {
	return len(e.user.Name()) > 0 && len(e.user.Mail()) > 0
}

func (e *StarlarkUser) Hash() (uint32, error) {
	return 0, errors.New("not hashable")
}

MethodsString() , Type() and Truth() required to use the type in the language’s built-in functions str(), type() and bool() (in addition, the second carries information about the type), Hash() is used to hash a value for use in a hash map in dictionaries and sets, a Freeze() it is necessary to freeze the object (as you can see, the guarantee of the immutability of the object after freezing the global scope lies entirely in the implementation of the type).

This type can already be used somehow, let’s try:

# user.star
def user_info(user):
    print(type(user)) # напечатает тип
    print(user)       # напечатает строковое представление объекта

args := starlark.Tuple{
	&StarlarkUser{
		&User{
			name: "John", 
			mail: &mail.Address{
				Name:    "John",
				Address: "[email protected]",
			},
		},
	},
}
_, err = starlark.Call(thread, globals["user_info"], args, nil)
if err != nil {
	panic(err)
}

Result:

user
name: John, mail: "John" <[email protected]>

We have a full-fledged type, however, it is not possible to perform any operations with it – to implement them, you need to support the corresponding interfaces, for example, HasUnary, HasBinary, etc., which we will not do in the framework of this article, otherwise there is already a lot of text, and the creation of built-in functions and methods is still ahead.

We are adding a new function

This is where the constructor will come in handy NewUser(). Built-in functions are implemented via Builtin:

// тело функции
func newUser(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
	var name, mail string
	if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 2, &name, &mail); err != nil {
		return starlark.None, err
	}

	user, err := NewUser(name, mail)
	if err != nil {
		return starlark.None, err
	}

	return &StarlarkUser{user: user}, nil
}

func main() {
	thread := &starlark.Thread{}
    // собираем наши встраиваемые функции, которые затем передаются в ExecFile()
	builtins := starlark.StringDict{
		"newUser": starlark.NewBuiltin("newUser", newUser),
	}
	
	globals, err := starlark.ExecFile(thread, "user.star", nil, builtins)
	if err != nil {
		panic(err)
	}

	_, err = starlark.Call(thread, globals["user_info"], nil, nil)
	if err != nil {
		panic(err)
	}
}

Several new things appeared. First, inline functions must match the type func(thread *starlark.Thread, fn *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error).

Secondly, built-in functions, or rather, all conditional objects, are collected in a dictionary and transferred to ExecFile()to make them available to Starlark code.

Thirdly, you can use UnpackPositionalArgs to unpack the arguments – it will check the number and types of arguments passed.

We are trying to call:

# user.star
def user_info():
    user = newUser("John", "[email protected]")

    print(type(user))
    print(user)

$ go run main.go
user
name: John, mail: <[email protected]>

Working! It is worth noting that not only functions can be passed in this way, but also any other objects whose type implements Value – for example, you can pass a pre-compiled set of constants that can be useful in embedded code.

We add methods

Adding “methods” to custom objects is implemented this way, via BuitlinAt the same time, a type that has methods must implement the HasAttrs interface.

But first, let’s prepare our methods:

func userName(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
	if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 0); err != nil {
		return starlark.None, err
	}

    // получаем ресивер, приводим к нужному типу и работаем уже с ним
	name := b.Receiver().(*StarlarkUser).user.Name()

	return starlark.String(name), nil
}

func userRename(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
	var name string
	if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
		return starlark.None, err
	}

	b.Receiver().(*StarlarkUser).user.Rename(name)

	return starlark.None, nil
}

The mechanics of the methods differ from functions in that they are transferred Builtin the receiver is pulled out, brought to the desired type, after which the necessary manipulations are performed on it.

Looking at the implementation of built-in types in the interpreter library, we collect our methods into a dictionary:

var userMethods = map[string]*starlark.Builtin{
	"name":   starlark.NewBuiltin("name", userName),
	"rename": starlark.NewBuiltin("rename", userRename),
}

And we implement HasAttrs:

func (e *StarlarkUser) Attr(name string) (starlark.Value, error) {
	b, ok := userMethods[name]
	if !ok {
		return nil, nil // нет такого метода
	}
	return b.BindReceiver(e), nil
}

func (e *StarlarkUser) AttrNames() []string {
	names := make([]string, 0, len(userMethods))
	for name := range userMethods {
		names = append(names, name)
	}
	sort.Strings(names)
	return names
}

BindReceiver() creates a new one Builtinwhich carries the value passed to which we access the method.

Let’s try:

# user.star
def user_info():
    user = newUser("John", "[email protected]")

    user.rename("Jim")
    print(user.name())

$ go run main.go
Jim

This is how we managed to add methods to our custom type in a not so clever way.

Bonus: Modules, built-in and custom

Starlark has several built-in modules that can be useful, here are some of them:

• math – a module with mathematical functions and a pair of constants

• time – functions and types for working with time

There is a special load function for loading modules, but you can’t use it just like that – loading modules is performed by the Load() function in the thread, and by default it is not there, so you need to implement it. Let’s extend our initial flow:

import (
	"go.starlark.net/lib/math"
	"go.starlark.net/lib/time"
)

thread := &starlark.Thread{
	Load: func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
		switch module {
		case "math.star":
			return starlark.StringDict{
				"math": math.Module,
			}, nil
		case "time.star":
			return starlark.StringDict{
				"time": time.Module,
			}, nil
		default:
			return nil, fmt.Errorf("no such module: %v", module)
		}
	},
}

The function will be called for each load() in the code. Let’s try to display the current time:

# modules.star
load("time.star", "time")

print(time.now()) # выведет текущее время

The second (and subsequent) arguments load() define imported literals, while starting with _ are not imported. In the case of modules implemented on Go, a structure is imported, through the fields of which we refer to functions and constants.

You can write modules on Starlark, and for convenience, use the starlarkstruct extension so that working with our custom module does not differ from working with built-in ones:

builtins := starlark.StringDict{
    "newUser": starlark.NewBuiltin("newUser", newUser),
    "struct":  starlark.NewBuiltin("struct", starlarkstruct.Make),
}

We will support uploading files to the module upload function:

thread := &starlark.Thread{
	Load: func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
		switch module {
		case "math.star":
			return starlark.StringDict{
				"math": math.Module,
			}, nil
		case "time.star":
			return starlark.StringDict{
				"time": time.Module,
			}, nil
		default: // внешний модуль, загружаем из файла
			script, err := os.ReadFile(module)
			if err != nil {
				return nil, err
			}

			entries, err := starlark.ExecFile(thread, module, script, builtins)
			if err != nil {
				return nil, err
			}

			return entries, nil
		}
	},
}

Let’s define the module:

# myModule.star
def hello():
    print("hello from module")

# создаем структуру с экспортируемыми литералами
myModule = struct(
    hello = hello
)

And we will use it in the main code:

load("myModule.star", "myModule")

myModule.hello() # выведет hello from module

Due to the fact that we have defined a structure of the same name with literals in the module, the module is easily imported and used.

Conclusion

In my case, embedding Starlark allowed the app to give users a tool for flexible configuration and adding custom logic to some of the event-driven steps. I will be glad if this material turns out to be useful to you, and Starlark may take a worthy place in your code.