Mediastreamer2. Application of Lua machine in filters

Short description

In this additional chapter to a Mediastreamer2 guide, the article explores creating a programmable filter using the Lua engine, allowing for the algorithm of the filter to be changed without recompiling the running code. The filter initializes an instance of the Lua machine upon starting, transforming data through a filter using two global variables, lf_data and lf_data_len. The resulting block of data is put into a long string, followed by a pair of global variables, lf_data_out and lf_data_out_len. The Lua filter is able to change the processing algorithm and the behavior of the media stream transfer scheme without stopping the main application.

Mediastreamer2. Application of Lua machine in filters

This post is an additional chapter to my Mediastreamer2 guide. The updated version of the manual can be freely downloaded here: Yandex disk.

Chapter 8 Using the Lua engine in filters

Previously, we considered filters whose behavior after starting can be controlled only partially – by calling the methods provided in them. In this article, we will create a programmable filter, the behavior of which will be completely determined by the Lua engine built into it, or rather by the script loaded into it. This will allow changing the algorithm of the filter without recompiling the running code.

The program code for this section can be downloaded from Github

Let’s start with practical implementation. To do this, you can remember how to create a new filter, see section 4. In this scheme, the source of the sound signal can be the signal from the line input of the sound card (sound_card_read) or a DTMF signal generator (dtmf_generator). Next, the data enters the input of the Luafilter (lua_filter), which performs their transformation according to the script uploaded to it. The data then goes to the splitter (Tee), which forms two copies of the input stream, which it outputs to two outputs. One of these streams enters the recorder (recorder) and to the sound card for playback (sound_card_write). The registrar saves them to disk in the format raw (wav-file without a header). In this way, we will be able to listen and record the result of the Lua filter.

8.1 Implementation of the filter

The Lua filter will be arranged in the following way. In the filter, upon its initialization, an instance of the Lua machine will be created. Moving data through a filter is shown in Figure 8.1.

Figure 8.1: Moving data inside a Lua filter

The incoming block of data will be passed to the Lua machine via its stack using two global variables:

  • lf_data – a long string containing the bytes of the data block;

  • lf_data_len – integer, string length lf_data in bytes.

Once these two variables are pushed onto the stack, execution is passed to the Lua engine. It will activate the downloaded script and convert the data string received from the stack. The resulting block of data is again put into a long string and put into a global variable and then onto the stack. Another pair of global variables is used for this:

  • lf_data_out – the line in which the bytes of the output data block are located;

  • lf_data_out_len – integer, string length lf_data_out in bytes.

Then the filter pulls the data from the stack and the output block of data is formed, which is exposed to the filter output.

Thus, the use of the Lua filter makes it possible to change the processing algorithm and the behavior of the media stream transfer scheme without recompiling and stopping the main application, i.e. on the fly

In fact, a text string of the appropriate size is used to pass the block to the Lua machine and back. More sophisticated options for data exchange can be implemented using types[User‑Defined Types](https://www.lua.org/pil/28.html), their implementation is beyond the scope of this book.

Since the block of input data is known to consist of 16-bit signed counts, in the line script each count will be encoded with two bytes in the additional code.

8.2 Organization of the script

Initially, the filter does not contain an executable Lua script, it must be loaded there. At the same time, it is advisable to divide the script into two parts. The first part performs the initial initialization. This part of the script is executed once. The second part is intended for multiple, cyclical execution on each beat of the ticker. We will call these parts of the script as:

  • the preamble of the script is performed once upon receipt of the first beat from the ticker.

  • body of the script – this part is executed every ticker cycle (10 ms).

The LUA_FILTER_SET_PREAMBLE and LUA_FILTER_RUN methods are provided to load the preamble and body into the filter, respectively.

8.2.1 Script preamble

This is a one-shot Lua code that links libraries, declares functions (used by the body of the script), and needs to initialize variables. Since the preamble starts with the first beat of the ticker, we need to define its code before the ticker starts. For this, the LUA_FILTER_SET_PREAMBLE method will be used, to which the usual pointer to the filter structure is passed as the first argument, and the preamble text as the second argument. The preamble can be written/rewritten to the filter multiple times and at any time. After each re-recording of the preamble, it is followed by its one-time performance on the nearest measure.

8.2.2 Body of the script

Unlike the preamble, this part of the script is executed every ticker cycle (by default every 10ms). This part of the code can be written/overwritten in the filter at any time. For this, the LUA_FILTER_RUN method will be defined, the arguments are similar to the preamble.

8.2.3 Stopping the script

At any moment, the script can be stopped (paused) using the LUA_FILTER_STOP method. After calling this method, the input blocks of data are passed immediately to the output of the filter, bypassing script processing. You can resume processing by calling the LUA_FILTER_RUN method. Substituting a null pointer or a new text pointer instead of the text of the body of the script.

8.2.4 Additional functions

In order for the script to be able to extract data from the string and put the result of its work back into the string, we need two data access functions get_sample() and append_sample(). The first performs subtraction of the signal count from the line. With the help of the second, you can add a countdown to a long line. Their text is given in Listing 8.1.

Listing 8.1: Data access functions

-- funcs.lua

-- Функция извлекает из строки один 16-битный отсчет.
function get_sample(s, sample_index)
local byte_index = 2*sample_index - 1
local L = string.byte(s, byte_index)
local H = string.byte(s, byte_index + 1)
local v = 256 * H + L
if (H >= 128) then
v = - ((~(v - 1)) & 65535)
end
return v
end

-- Функция добавляет в строку один 16-битный отсчет.
function append_sample(s, sample_value)
local v = math.floor(sample_value + 0.5)
if v < 0 then
v = - v
v = ((~v) + 1) & 65535
end
local H = v // 256
local L = v - H * 256
return  s .. string.char(L, H)
end

Function get_sample() is used to access the signal counts stored in the string. It returns a single 16-bit count extracted from the string s. By the given reference index sample_index() the indices of 2 bytes in which it is stored are determined. Next, a 16-bit number is collected from these two bytes. Since the count is a signed number, these bytes store the number in additional code, we need to convert the number to normal form. First, we determine the state of the 15th bit of the number. If it is equal to 0, then the number is positive and additional transformations are not required. If the bit is set, the number is negative. Then we subtract one from between and do the inversion of each bit and multiply by –1.

Function append_sample() is used to compose a string with output data. It adds to its first argument (string) two bytes that represent the second argument (signal count) in the additional code.

The function file will be called funcs.lua it must be placed in the directory where the filter code will run.

8.2.5 Script example

Let’s create a script that will simply pass the data through itself without changing it.

The preamble is shown in Listing 8.2:

Listing 8.2: Script preamble

-- preambula2.lua
-- Этот скрипт выполняется в Lua-фильтре как преамбула.
-- Подключаем файл с функциями доступа к данным.
require "funcs"
preambula_status = 0
body_status = 0 -- Эта переменная будет инкрементироваться в теле скрипта.
local greetings="Hello world from preambula!\n" -- Приветствие.
print(greetings)
return preambula_status 

In the preamble, we include a file with signal data access functions (funcs.lua).

The body of the script is shown in Listing 8.3:

Listing 8.3: The body of the script

-- body2.lua
-- Этот скрипт выполняемый в Lua-фильтре как тело скрипта.
-- Перекладываем результат работы в выходные переменные.
lf_data_out =""
if lf_data_len == nil then
print("Bad lf_data_len.\n")
end
for i = 1, lf_data_len/2 do
s = get_sample(lf_data, i)
lf_data_out = append_sample(lf_data_out, s)
end
lf_data_out_len = string.len(lf_data_out)
return body_status

Nothing special is done in the body of the script, just the counts from the input line are transferred one by one to the output line. Obviously, it’s between functions get_sample() and append_sample() any count transformation can be arranged. Another option is that the filter does not handle counts, but can control other filters according to the input data.

It should be noted that when writing scripts, it is convenient to put a comment containing the name of the file in the first line of the file, as is done in the examples: then when an error occurs in the diagnostic message, the name of the file in which the error was detected will appear in the first place: and it will become clear to you which part of the script is meant.

-- preambula2.lua
 Filter <LUA_FILTER> Lua error. Lua error description:<[string "-- preambula2.lua ..."]:12: attempt to perform arithmetic on a nil value>.

8.3 Filter source code

The filter header file will look like this:

Listing 8.4: Lua filter header file

#ifndef lua_filter_h
#define lua_filter_h
/* Подключаем заголовочный файл с перечислением фильтров медиастримера. */
#include <mediastreamer2/msfilter.h>
/* Подключаем интерпретатор Lua. */
#include <lua5.3/lua.h>
#include <lua5.3/lauxlib.h>
#include <lua5.3/lualib.h>
/*
Задаем числовой идентификатор нового типа фильтра. Это число не должно
совпадать ни с одним из других типов.  В медиастримере  в файле allfilters.h
есть соответствующее перечисление enum MSFilterId. К сожалению, непонятно
как определить максимальное занятое значение, кроме как заглянуть в этот
файл. Но мы возьмем в качестве id для нашего фильтра заведомо большее
значение: 4001.   Будем полагать, что разработчики добавляя новые фильтры, не
скоро доберутся до этого номера.
*/
#define LUA_FILTER_ID 4001
/* Имя глобальной переменной, в которую функция фильтра помещает блок входных
данных. */
#define LF_DATA_CONST       "lf_data"
/* Имя глобальной переменной, в которую функция фильтра помещает размер блока входных
данных.*/
#define LF_DATA_LEN_CONST   "lf_data_len"
/* Имя глобальной переменной, в которую функция фильтра помещает блок выходных
данных.*/
#define LF_DATA_OUT_CONST   "lf_data_out"
/* Имя глобальной переменной, в которую функция фильтра помещает размер блока выходных
данных.*/
#define LF_DATA_OUT_LEN_CONST "lf_data_out_len"
/* Флаг того, что входная очередь фильтра пуста. */
#define LF_INPUT_EMPTY_CONST "input_empty"
/* Определяем константы фильтра. */
#define LF_DATA             LF_DATA_CONST
#define LF_DATA_LEN         LF_DATA_LEN_CONST
#define LF_DATA_OUT         LF_DATA_OUT_CONST
#define LF_DATA_OUT_LEN     LF_DATA_OUT_LEN_CONST
#define LF_INPUT_EMPTY      LF_INPUT_EMPTY_CONST
/*
Определяем методы нашего фильтра. Вторым параметром макроса должен
порядковый номер метода, число от 0.  Третий параметр это тип аргумента
метода, указатель на который будет передаваться методу при вызове.
*/
#define LUA_FILTER_RUN	      MS_FILTER_METHOD(LUA_FILTER_ID,0,char)
#define LUA_FILTER_STOP         MS_FILTER_METHOD(LUA_FILTER_ID,1,int)
#define LUA_FILTER_SET_PREAMBLE MS_FILTER_METHOD(LUA_FILTER_ID,2,char)
/* Определяем экспортируемую переменную, которая будет
хранить характеристики для данного типа фильтров. */
extern MSFilterDesc lua_filter_desc;
#endif /* lua_filter_h */

Here macros are created with the names of global variables in the context of the Lua machine and the three filter methods mentioned above are declared:

  • LUA_FILTER_RUN;

  • LUA_FILTER_STOP;

  • LUA_FILTER_SET_PREAMBLE

We will consider the source code of the filter only in its important part, i.e. operation of the method control_process() (the complete source code is given in Appendix A). This method executes the Lua machine startup every time the ticker ticks. Its text is shown in Listing 8.5.

Listing 8.5: The control_process() method

static void
control_process(MSFilter *f)
{
ControlData *d = (ControlData *)f->data;
mblk_t *im;
mblk_t *out_im = NULL;
int err = 0;
int i;

if ((!d->stopped) && (!d->preabmle_was_run))
{
	run_preambula(f);
}

while ((im = ms_queue_get(f->inputs[0])) != NULL)
{
	unsigned int disabled_out = 0;
	if ((!d->stopped) && (d->script_code) && (d->preabmle_was_run))
	{
	  bool_t input_empty = ms_queue_empty(f->inputs[0]);
	  lua_pushinteger(d->L, (lua_Integer)input_empty);
	  lua_setglobal(d->L, LF_INPUT_EMPTY);
	  
      /* Кладем блок данных со входа фильтра на стек Lua-машины. */
	  size_t sz = 2 * (size_t)msgdsize(im); /* Размер блока в байтах.*/
	  lua_pushinteger(d->L, (lua_Integer)sz);
	  lua_setglobal(d->L, LF_DATA_LEN);
	  
      lua_pushlstring(d->L, (const char *)im->b_rptr, sz);
	  lua_setglobal(d->L, LF_DATA);

	  /* Удаляем со стека все, что там, возможно, осталось. */
	  int values_on_stack;
	  values_on_stack = lua_gettop(d->L);
	  lua_pop(d->L, values_on_stack);

	  /* Выполняем тело скрипта. */
	  err = luaL_dostring(d->L, d->script_code);

	  /* Обрабатываем результат выполнения. */
	  if (!err)
	  {
		  int script_body_status = lua_tointeger(d->L, lua_gettop(d->L));
		  if (script_body_status < 0)
		  {
			  printf("\nFilter <%s> bad script_body_status: %i.\n", f->desc->name,
					 script_body_status);
		  }

		  /* Извлекаем размер выходного блока данных, возможно он изменился. */
		  lua_getglobal(d->L, LF_DATA_OUT_LEN);
		  size_t real_size = 0;
		  char type_on_top = lua_type(d->L, lua_gettop(d->L));
		   // printf("Type on top: %i\n", type_on_top);
		  if (type_on_top == LUA_TNUMBER)
		  {
			  real_size =
				  (size_t)lua_tointeger(d->L, lua_gettop(d->L));
			  // printf("------- size from lua %lu\n", real_size);
		  }
		  lua_pop(d->L, 1);

		  /* Извлекаем длинную строку с преобразованными данными входного блока
		   данных. И пробрасываем его далее. */
		  lua_getglobal(d->L, LF_DATA_OUT);
		  size_t str_len = 0;
		  if (lua_type(d->L, lua_gettop(d->L)) == LUA_TSTRING)
		  {
			  const char *msg_body = lua_tolstring(d->L, -1, &str_len);
			  if (msg_body && str_len)
			  {
				  size_t msg_len = real_size / 2;

				  out_im = allocb((int)msg_len, 0);
				  memcpy(out_im->b_wptr, msg_body, msg_len);
				  out_im->b_wptr = out_im->b_wptr + msg_len;
			  }
		  }
		  lua_pop(d->L, 1);

		  /* Вычитываем и отбрасываем все, что возможно осталось на стеке. */
		  values_on_stack = lua_gettop(d->L);
		  lua_pop(d->L, values_on_stack);
	  }
	  else
	  {
		  printf("\nFilter <%s> Lua error.\n", f->desc->name);
		  const char *answer = lua_tostring(d->L, lua_gettop(d->L));
		  if (answer)
		  {
			  printf("Lua error description:<%s>.\n", answer);
		  }
	  }
	}
	mblk_t *p = im;
	if (out_im)
		p = out_im;
	
		for (i = 0; i < f->desc->noutputs; i++)
		{
		  if ((!disabled_out) && (f->outputs[i] != NULL))
		  if (p)
			  ms_queue_put(f->outputs[i], dupmsg(p));
		}
	
	freemsg(out_im);
	freemsg(im);
}
}

When the method control_process() receives control, it checks if there is data at the filter input and sets the global variable LF_INPUT_EMPTY so that, if necessary, the script can handle the situation when there is no input data. Then, if there is input data, like any other filter, it starts reading it. Each block is 160 counts or 320 bytes by default, but the block size is defined. The result is placed on the stack of the Lua machine, and from it to the global Lua variable lf_data_len (It). After that, the method puts the data block itself on the stack, and from it into the global variable lf_data (Long line). Next, control is transferred to the Lua engine by calling a function:

luaL_dostring(d->L, d->script_code)

the machine starts executing the script loaded earlier. After the script is executed, the process of transferring the results of the script from the context of the Lua machine to the method will take place.

8.4 Test application

Once the Lua filter is implemented, it’s time to create a test application for it. To check the filter being developed, we will use the scheme shown in Figure 8.2.

Figure 8.2: Scheme for checking the Lua filter

Figure 8.2: Scheme for checking the Lua filter

The algorithm of the program will be as follows. After start-up, when there is already a circuit of filters in the RAM, but the sources of clock signals will not be activated, the filter initialization procedure will begin. It consists in the fact that each filter has its own method. init(). The created filter will not be an exception, but in addition to mandatory actions, init() it will execute preamble Lua code that initializes the Lua machine. When starting the program, we have to pass it the path to the preamble file using the command line switch “scp” The other part, the body of the script, is passed with the key “scb“. A complete list of program keys is given in Listing 8.6.

Listing 8.6: Test program command line keys

--help      List of options.
--version   Version of application.
--scp       Full name of containing preambula of Lua-script file.
--scb       Full name of containing body of Lua-script file.
--gen       Set generator's frequency, Hz.
--rec       Make recording to a file 'record.wav'.

The source code of the test program is given in Appendix B. Earlier, at the beginning of the chapter, a link was given to download this source code and the instructions in the file

README.md

perform assembly.

An example of running a test program:

$ ./lua_filter_demo --scb ../scripts/body2.lua  --scp ../scripts/preambula2.lua --gen 600   

After starting, a 600 Hz tone will be heard in the headphones for 10 seconds. This means that the signal passed through the filter.

8.5 Example of using a filter

As an example, let’s write a script that, starting from the 5000th count (that is, 5/8 of a second), will multiply the input signal by a low-frequency sinusoidal signal (that is, it will modulate by amplitude) for 2 seconds. The signal will then become unmodulated again.

Listing 8.7: Modulator script preamble

-- preambula3.lua
-- Этот скрипт выполняется в Lua-фильтре как преамбула.
-- Подключаем файл с функциями доступа к данным.
require "funcs"
preambula_status = 0
body_status = 0 -- Эта переменная будет инкрементироваться в теле скрипта.
-- Переменные для расчетов.
samples_count = 0
sampling_rate = 8000
low_frequency = 2 -- Модулирующая частота.
phase_step = 2 * math.pi / sampling_rate * low_frequency
return preambula_status 

Modulation will be implemented as follows:

Listing 8.8: The body of the modulator script

-- body3.lua
-- Это скрипт выполняемый в Lua-фильтре как тело скрипта.
-- Скрипт выполняет модуляцию входного сигнала.
lf_data_out =""
if lf_data_len == nil then
print("Bad lf_data_len.\n")
end
for i = 1, lf_data_len/2 do
s = get_sample(lf_data, i)
if (samples_count > 5000)  and (samples_count < 21000) then
output_s = s * math.sin( phase_step * samples_count )
else
output_s = s
end
samples_count = samples_count + 1
lf_data_out = append_sample(lf_data_out, output_s)
end
lf_data_out_len = string.len(lf_data_out)
return body_status

We run the program with the new script:

$ ./lua_filter_demo --scb ../scripts/body3.lua  --scp ../scripts/preambula3.lua --gen 1200 --rec   

after 5 seconds, to stop the program, press the “Enter“. You can also play the file as we did before in 3.8:

$ aplay -t raw --format s16_be --channels 1 ./record.raw

Let’s convert the source file to wav-format:

$ sox -t raw -r 8000 -b 16 -c 1 -L -e signed-integer ./record.raw  ./recording.wav  

To draw signal y gnuplotwe need to convert it to a file with two columns of data. This will be done by the same utility sox in a pair grep:

$ sox ./recording.wav  -r 8000 recording.dat && grep -v «^;» recording.dat > clean_recording.dat

Next, we transfer the resulting file recording.wav use gnuplot:

$ gnuplot -e "set terminal png; set output 'recording.png'; plot 'clean_recording.dat' using 1:2 with lines"

Figure 8.3 shows the result of the script.

Figure 8.3: The result of the Lua filter

In the figure, we see the envelope of the sinusoidal signal after it passes through the filter. The signal amplitude remained unchanged for approximately 5/8 seconds, then the script branch with the modulation algorithm was activated:

output_s = s * math.sin( phase_step * samples_count)

and a modulated signal was present at the output for a second. After that, the script turned off the modulation and the signal began to be transmitted to the output without modulation.

Related posts