Serialization of data in Golang with Protobuf

Serialization of data in Golang with Protobuf

Hello, Habre!

Protobuf, or Protocol Buffers, is a binary serialization format developed by Google for efficient data exchange between services. It’s like JSON, only more compact, faster and more typed. If JSON was your first foray into the world of serialization, then Protobuf is the one you want to get serious with.

If you want your applications to communicate between services with lightning speed and minimal latency, then Protobuf is your good choice.

Installation

Step 1: Install Protobuf Compiler

First of all, you need to download the Protobuf Compiler. Check out the Protobuf page on GitHub and select the version that works for your OS.

Once downloaded, extract the contents and add the path to the executable protoc file to your system PATH variable. In Unix it is something like export PATH=$PATH:/path/to/protoc.

Step 2: Install Protobuf Plugin

Run it go get -u google.golang.org/protobuf/cmd/protoc-gen-go to install the protoc plugin for Go. This plugin is required to convert your .proto files to go files.

Enter protoc --version in the console. If you see the version, you have successfully installed the compiler!

Structure of .proto files

First of all, .proto file is a data schema for Protobuf. This is where you describe the data structure you want to serialize/deserialize.

It all starts with specifying the syntax version. Example, syntax = "proto3";. This tells the compiler what rules to use when parsing the content. Then comes the announcement of the package package mypackage;. This helps avoid name conflicts and organize the code.

The heart of the .proto file is this announcement of messages. Each message is a data structure that you want to serialize.

Messages are defined using syntax message MyMessage {}. Inside the curly braces, you describe the data fields. Fields can be standard data types such as int32, float, double, string or bool. There may also be user types or other messages.

Protobuf has three types of rules for fields: singular for singular values, repeated for arrays of values ​​and map for associative arrays Each field is given a unique number. These numbers are used in binary representation and are very important for data compatibility

Start field numbers with 1 and be careful not to jump through tens and hundreds – this may be useful for future versions of your protocol. If a field is no longer needed, mark it as reserved.

Let’s say you want to serialize information about a technical review that includes sections, headings, and a table of contents:

syntax = "proto"

package techreview;

// Объявляем перечисление для различных типов контента
enum ContentType {
  UNKNOWN = 0;
  INTRODUCTION = 1;
  TECHNICAL_OVERVIEW = 2;
  BEST_PRACTICES = 3;
  CONCLUSION = 4;
}

// Определяем структуру для раздела обзора
message Section {
  string title = 1; // Заголовок раздела
  string content = 2; // Содержание раздела
  ContentType type = 3; // Тип содержимого (введение, обзор и т.д.)
}

// Определяем основную структуру для всего технического обзора
message TechnicalReview {
  repeated Section sections = 1; // Массив разделов обзора
}

Enums, Maps

We create an Enum in .proto:

enum Status {
  UNKNOWN = 0;
  RUNNING = 1;
  STOPPED = 2;
}

After compilation .proto file, we get a well-defined type Statuswhich can be used directly in the code.

var currentStatus Status = Status_RUNNING

Maps allows you to create structured collections of key-value pairs.

Map declaration in .proto:

message Environment {
  map<string, string> variables = 1;
}

After compilation .proto file, we get a map that can be easily used.

env := Environment{
    Variables: map[string]string{"HOME": "/home/user", "PATH": "/usr/bin"},
}

Goodbye array of pairs or complex structures for simple tasks.

Oneof: when one field is not enough

oneof is a tool in Protobuf for working with different structures in the same field.

In the .proto file:

message Command {
  oneof command_type {
    string text = 1;
    int32 number = 2;
  }
}

After compilation .protoyou can manage these fields as normal structures in Go.

cmd := Command{
    CommandType: &Command_Text{Text: "Hello, Protobuf!"},
}

How to automatically generate code from .proto files

Suppose we have the following .proto file that describes the data structure for the user:

syntax = "proto ";

package user;

message User {
  string name = 1;
  int32 age = 2;
  string email = 3;
} 

You can use the command to generate the code protoc:

protoc --go_out=. user.proto

After executing this command, a file will appear in the same directory user.pb.gocontaining Go code for data structures.

How you can use the generated code in Go:

package main

import (
    "fmt"
    "log"

    "github.com/golang/protobuf/proto"
    "path/to/generated/user"  // Импорт сгенерированного кода
)

func main() {
    newUser := &user.User{
        Name:  "Alice",
        Age:   30,
        Email: "[email protected]",
    }

    // сериализация данных
    data, err := proto.Marshal(newUser)
    if err != nil {
        log.Fatal("Marshaling error: ", err)
    }

    // десериализация данных
    newUser2 := &user.User{}
    err = proto.Unmarshal(data, newUser2)
    if err != nil {
        log.Fatal("Unmarshaling error: ", err)
    }

    fmt.Println(newUser2.GetName(), newUser2.GetAge(), newUser2.GetEmail())
}

You can use plugins to add additional functionality. Example, protoc-gen-go-grpc to create a gRPC server and client.

gRPC in protobuf

Let’s install the necessary packages:

go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go

First, let’s define our gRPC service and message in .proto files:

syntax = "proto3";

package example;

// Определение сервиса
service Greeter {
  // Определение метода сервиса
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// Определение сообщения, используемого для запроса
message HelloRequest {
  string name = 1;
}

// Определение сообщения, используемого для ответа
message HelloReply {
  string message = 1;
}

The next step is to generate Golang code from ours .proto file:

protoc --go_out=plugins=grpc:. *.proto

We implement the server:

package main

import (
    "context"
    "log"
    "net"

    "google.golang.org/grpc"
    pb "path/to/your/protos/example"
)

// server is used to implement example.GreeterServer.
type server struct {
    pb.UnimplementedGreeterServer
}

func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
    log.Printf("Received: %v", in.GetName())
    return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterGreeterServer(s, &server{})
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

And finally, we implement a client to communicate with our server:

package main

import (
    "context"
    "log"
    "os"
    "time"

    "google.golang.org/grpc"
    pb "path/to/your/protos/example"
)

const (
    address     = "localhost:50051"
    defaultName = "world"
)

func main() {
    // Установка соединения с сервером
    conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewGreeterClient(conn)

    // Имя пользователя для приветствия
    name := defaultName
    if len(os.Args) > 1 {
        name = os.Args[1]
    }

    // Контекст для отмены запроса по истечении таймаута
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()

    // Вызов метода SayHello на сервере
    r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
    if err != nil {
        log.Fatalf("could not greet: %v", err)
    }
    log.Printf("Greeting: %s", r.GetMessage())
}

Microservices

Let’s say we have a microservice to manage users:

syntax = "proto3";

package user;

// Сервис для работы с пользователями
service UserService {
  // Запрос на получение информации о пользователе
  rpc GetUser (UserRequest) returns (UserResponse) {}
}

// Запрос на получение информации о пользователе
message UserRequest {
  int64 id = 1;
}

// Ответ с информацией о пользователе
message UserResponse {
  int64 id = 1;
  string name = 2;
  string email = 3;
}

Let’s say we generated Golang code from our file. Let’s write the server:

package main

import (
    "context"
    "log"
    "net"

    "google.golang.org/grpc"
    pb "path/to/your/protos/user"
)

type server struct {
    pb.UnimplementedUserServiceServer
}

func (s *server) GetUser(ctx context.Context, in *pb.UserRequest) (*pb.UserResponse, error) {
    log.Printf("Received: %v", in.GetId())
    // здест к примеру логика получения пользователя из базы данных или другого сервиса
    return &pb.UserResponse{Id: in.GetId(), Name: "John", Email: "[email protected]"}, nil
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterUserServiceServer(s, &server{})
    log.Printf("Server listening at %v", lis.Addr())
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

Let’s create a client that will communicate with our server:

package main

import (
    "context"
    "log"
    "os"
    "time"

    "google.golang.org/grpc"
    pb "path/to/your/protos/user"
)

const (
    address     = "localhost:50051"
)

func main() {
    // установка соединения с сервером
    conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewUserServiceClient(conn)

    // ус тановим контекст с таймаутом
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()

    // запрс на получение пользователя
    r, err := c.GetUser(ctx, &pb.UserRequest{Id: 123})
    if err != nil {
        log.Fatalf("could not greet: %v", err)
    }
    log.Printf("User: %s", r.GetName())
}

Protobuf is a compact, binary format that reduces the size of data and speeds up its processing, and it is also compatible with many APIs.

In conclusion, I would like to recommend a free webinar where you will learn what specific competencies Golang engineers need at different stages of career development. Registration available at the link.

Related posts