We are writing a simple console messenger with end-to-end encryption on top of “Hidden Lake” services

We are writing a simple console messenger with end-to-end encryption on top of “Hidden Lake” services

In my spare time, I write anonymously Hidden Lake, I add new features to it and in parallel fix the bugs I wrote earlier. The entire philosophy of the Hidden Lake network is based on microservice architecture, on the concept of independent (as far as possible) services from each other. One of my articles already gave an example of how we can create our own service in the anonymous HL network, using our technologies and at the same time not using the Go language (the native language of Hidden Lake applications).

But at that time, the Hidden Lake platform was limited in terms of creating non-anonymous but secure applications. In other words, it was possible to write an anonymous and secure messenger (HLM is an example), but one could not remove the anonymity from that set. In part, this is a logical behavior, because HL is somehow an anonymous network. The contradiction here was that when the network is built on mutually independent services, then purely technically and theoretically there should have been an opportunity to get rid of the service representing anonymity. And such a solution was found.

A few words about the services used

Our future messenger with end-to-end encryption will be written in Python with the sole purpose of showing the possibility of using any technology you are comfortable with on top of ready-made implementations.

This messenger will be tied to the use of two HL services: HLT (traffic) and HLE (Encryptor). Briefly, HLT is a relay service and a storage service of generated traffic. Thanks to it, you can restore messages that have already been forgotten by the network. We will need this property in the future. You can read more about HLT here and here.

The HLE service, in turn, is a very simple application, it is only capable of encrypting and decrypting messages read by the HL network, nothing more, nothing less. But the last service was not enough to be able to write exclusively client-safe programs without the quality of anonymity, or more flexible programs, for example, with another proof of anonymity.

Realization

1. Configuration

The first thing we’ll start with is the configuration file. It should have three fields: hlt_host, hle_host and friends. The first two indicate the addresses of the HLT and HLE services, respectively. The HLT service can be on the WAN, while the HLE must be locally on the computer. The bottom line is that HLE refers to the private key, the primary identifier of the network. Its loss inevitably leads to communication compromise.

The last field is a map (dictionary) by type name: public_key. The existence of friends brings us to a type of network friend-to-friendwhen participants of communication set in advance with whom they can communicate and from whom they can receive messages. This feature allows you to get rid of spam from other network members, and also allows you to more easily identify subscribers by their self-exposed name.

So, we get approximately the following configuration file:

hlt_host: localhost:9582
hle_host: localhost:9551
friends: 
  Alice: PubKey{3082020A02820201...3324D10203010001}

The only meaning here unknown to us is the public key itself, the method of obtaining it. We need to get this key from the intended caller, but how should he extract it? There are two ways here:

  1. When starting HLE, if the latter does not detect the presence of a private key, it is its own will generate and will load into its internal state. All that remains is to request the public key for it from HLE API.

    curl -i -X GET -H 'Accept: application/json' http://localhost:9551/api/service/pubkey

  2. You can generate a private / public key pair yourself using the cmd/tools/keygen program from the repository go-peer. In this case, it will be enough to write only the length of the RSA key. The length of the key in the HL network is 4096 bits. After running, two files will be created priv.key and pub.key. The first key is given to the HLE input, and the second key is disclosed to the interlocutor.

    go run . 4096

Now all that remains for us is to implement reading the configuration file in Python. For simplicity, I moved the read configuration fields to global variables. Once read, these global variables are only considered by the program. Therefore, this approach will not lead to a possible race state.

import yaml

HLE_URL = "" # it is being overwritten
HLT_URL = "" # it is being overwritten
FRIENDS = {} # it is being overwritten

def init_load_config(cfg_path):
    global HLE_URL, HLT_URL, FRIENDS

    with open(cfg_path, "r") as stream:
        try:
            config_loaded = yaml.safe_load(stream)
        except yaml.YAMLError as e:
            print("@ failed load config")
            exit(3)
    
    HLT_URL = "http://" + config_loaded["hlt_host"]
    HLE_URL = "http://" + config_loaded["hle_host"]
    FRIENDS = config_loaded["friends"]

...

2. Reading the introduction

The messenger will consist of two parallel procedures: input_task and output_task. The first procedure is reduced to entering data/commands from the user himself. There will be only one team: /friend. It sets a friend to whom we will send messages. The second procedure is reduced to receiving data from the network and outputting them.

...

import multiprocessing

...

def main():
    init_load_config("config.yml")
    parallel_run(input_task, output_task)

def parallel_run(*fns):
    proc = []
    for fn in fns:
        p = multiprocessing.Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

...

if __name__ == "__main__":
    main()

The code itself turned out to be quite simple. There is only one point worth clarifying, namely sys.stdin = open(0). The parallel implementation of starting procedures is reduced to the use of the multiprocessing module, which, in turn, does not push the standard input flow further into the Process class. As a result, it is necessary to explicitly specify this thread (0) in the parallel procedure. Without this part of the code, input will result in errors EOFError: EOF when reading a line. You can read more about it here.

...

import sys

def input_task():
    friend = ""

    sys.stdin = open(0)
    while True:
        msg = input("> ")
        if len(msg) == 0:
            print("@ got null message")
            continue

        if msg.startswith("/friend "):
            try:
                _friend = FRIENDS[msg[len("/friend "):].strip()]
            except KeyError:
                print("@ got invalid friend name")
                continue
            friend = _friend
            continue

        if friend == "":
            print("@ friend is null, use /friend to set")
            continue 

        ...

Next, once we already have the friend’s public key and the exact message we plan to send, we need to call the HLE API first to encrypt the message, and then the HLT API to send the encrypted result to the network.

        ...
      
        resp_hle = requests.post(
            HLE_URL+"/api/message/encrypt", 
            json={"public_key": friend, "hex_data": msg.encode("utf-8").hex()}
        )
        if resp_hle.status_code != 200:
            print("@ got response error from HLE (/api/message/encrypt)")
            continue 
        
        resp_hlt = requests.post(
            HLT_URL+"/api/network/message", 
            data=resp_hle.content
        )
        if resp_hlt.status_code != 200:
            print("@ got response error from HLT (/api/network/message)")
            continue 

  ...

More details about the HLE and HLT APIs can be found in their README here and here.

3. Reading from the network

Reading data from the network is already a more difficult task than ordinary input, but not something supernatural. You’ll just need to call the HLT API more often.

To begin with, we will need to receive from HLT the settings of how much the HLT service itself is able to store on its side of messages (messages_capacity). Because many HLT services are a decentralized network, the settings may differ accordingly. Someone can store thousands of messages, and someone can store a million. After exceeding the specified number, the HLT service starts overwriting old messages with new ones.

...

import requests, json

def output_task():
    # GET SETTING = MESSAGES_CAPACITY
    resp_hlt = requests.get(
        HLT_URL+"/api/config/settings"
    )
    if resp_hlt.status_code != 200:
        print("@ got response error from HLT (/api/config/settings)")
        exit(1)
    
    try:
        messages_capacity = json.loads(resp_hlt.content)["messages_capacity"]
    except ValueError:
        print("@ got response invalid data from HLT (/api/config/settings)")
        exit(2)
    
    ...

After receiving the limit, we need to issue the current one index (In HL terminology) on which the HLT itself is located. A pointer is an index of an array of messages that only increases, but in a ring. In other words, when messages_capacity is reached, the pointer will become zero again. Constant incrementation is described by a simple formula: (i + 1) mod messages_capacity.

After exposure, we will need to constantly check whether the pointer has not moved relative to the position we obtained earlier. If it has shifted, then you need to independently increment the pointer until it is equal to the one obtained. This is necessary in order to first get the message hash from the pointer, and then the message itself. If the pointer does not move, then you should set a time limit before the next request. Such a limit can be set in one second.

import time 

...
    ...
  
    global_pointer = -1
    while True:
        # GET INITIAL POINTER OF MESSAGES
        resp_hlt = requests.get(
            HLT_URL+"/api/storage/pointer"
        )
        if resp_hlt.status_code != 200:
            print("@ got response error from HLT (/api/storage/pointer)")
            time.sleep(1)
            continue 

        try:
            pointer = int(resp_hlt.content)
        except ValueError:
            print("@ got response invalid data from HLT (/api/storage/pointer)")
            time.sleep(1)
            continue

        if global_pointer == -1:
            global_pointer = pointer

        if global_pointer == pointer:
            time.sleep(1)
            continue
    
        # GET ALL MESSAGES FROM CURRENT POINTER TO GOT POINTER
        while global_pointer != pointer:
            global_pointer = (global_pointer + 1) % messages_capacity

            ...

Based on the logic of pointers, we write the final logic – the logic of receiving and decoding the message. First, we receive a hash from HLT by pointer, then we receive a message also from HLT based on the hash, we pass the received message to HLE for decryption.

After decryption, we need to verify the sender’s public key. If the sender is in our friends list, then we print the received message, otherwise we ignore it.

import hashlib 

...
            ...
  
            resp_hlt = requests.get(
                HLT_URL+"/api/storage/hashes?id="+f"{(global_pointer - 1) % messages_capacity}"
            )
            if resp_hlt.status_code != 200:
                break 
            
            resp_hlt = requests.get(
                HLT_URL+"/api/network/message?hash="+resp_hlt.content.decode("utf8")
            )
            if resp_hlt.status_code != 200:
                break 

            # TRY DECRYPT GOT MESSAGE
            resp_hle = requests.post(
                HLE_URL+"/api/message/decrypt", 
                data=resp_hlt.content
            )
            if resp_hle.status_code != 200:
                continue 

            try:
                json_resp = json.loads(resp_hle.content)
            except ValueError:
                print("@ got response invalid data from HLE (/api/message/decrypt)")
                continue
        
            # CHECK GOT PUBLIC KEY IN FRIENDS LIST
            user_id = hashlib.sha256(json_resp["public_key"].encode('utf-8')).hexdigest()
            friend_name = get_friend_name(user_id)
            if friend_name == "":
                continue 

            got_data = bytes.fromhex(json_resp["hex_data"]).decode('utf-8')
            print(f"[{friend_name}]: {got_data}\n> ", end="")

def get_friend_name(user_id):
  for k, v in FRIENDS.items():
      friend_id = hashlib.sha256(v.encode('utf-8')).hexdigest()
      if user_id == friend_id:
          return k
  return ""

...

And that’s all. The final implementation came out in approx 160 lines code

Full source code
import multiprocessing, sys, requests, time, json, hashlib, yaml

HLE_URL = "" # it is being overwritten
HLT_URL = "" # it is being overwritten
FRIENDS = {} # it is being overwritten

def main():
    init_load_config("config.yml")
    parallel_run(input_task, output_task)

def init_load_config(cfg_path):
    global HLE_URL, HLT_URL, FRIENDS

    with open(cfg_path, "r") as stream:
        try:
            config_loaded = yaml.safe_load(stream)
        except yaml.YAMLError as e:
            print("@ failed load config")
            exit(3)
    
    HLT_URL = "http://" + config_loaded["hlt_host"]
    HLE_URL = "http://" + config_loaded["hle_host"]
    FRIENDS = config_loaded["friends"]

def parallel_run(*fns):
    proc = []
    for fn in fns:
        p = multiprocessing.Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

def output_task():
    # GET SETTING = MESSAGES_CAPACITY
    resp_hlt = requests.get(
        HLT_URL+"/api/config/settings"
    )
    if resp_hlt.status_code != 200:
        print("@ got response error from HLT (/api/config/settings)")
        exit(1)
    
    try:
        messages_capacity = json.loads(resp_hlt.content)["messages_capacity"]
    except ValueError:
        print("@ got response invalid data from HLT (/api/config/settings)")
        exit(2)
    
    global_pointer = -1
    while True:
        # GET INITIAL POINTER OF MESSAGES
        resp_hlt = requests.get(
            HLT_URL+"/api/storage/pointer"
        )
        if resp_hlt.status_code != 200:
            print("@ got response error from HLT (/api/storage/pointer)")
            time.sleep(1)
            continue 

        try:
            pointer = int(resp_hlt.content)
        except ValueError:
            print("@ got response invalid data from HLT (/api/storage/pointer)")
            time.sleep(1)
            continue

        if global_pointer == -1:
            global_pointer = pointer

        if global_pointer == pointer:
            time.sleep(1)
            continue
    
        # GET ALL MESSAGES FROM CURRENT POINTER TO GOT POINTER
        while global_pointer != pointer:
            global_pointer = (global_pointer + 1) % messages_capacity

            resp_hlt = requests.get(
                HLT_URL+"/api/storage/hashes?id="+f"{(global_pointer - 1) % messages_capacity}"
            )
            if resp_hlt.status_code != 200:
                break 
            
            resp_hlt = requests.get(
                HLT_URL+"/api/network/message?hash="+resp_hlt.content.decode("utf8")
            )
            if resp_hlt.status_code != 200:
                break 

            # TRY DECRYPT GOT MESSAGE
            resp_hle = requests.post(
                HLE_URL+"/api/message/decrypt", 
                data=resp_hlt.content
            )
            if resp_hle.status_code != 200:
                continue 

            try:
                json_resp = json.loads(resp_hle.content)
            except ValueError:
                print("@ got response invalid data from HLE (/api/message/decrypt)")
                continue
        
            # CHECK GOT PUBLIC KEY IN FRIENDS LIST
            user_id = hashlib.sha256(json_resp["public_key"].encode('utf-8')).hexdigest()
            friend_name = get_friend_name(user_id)
            if friend_name == "":
                continue 

            got_data = bytes.fromhex(json_resp["hex_data"]).decode('utf-8')
            print(f"[{friend_name}]: {got_data}\n> ", end="")

def input_task():
    friend = ""

    sys.stdin = open(0)
    while True:
        msg = input("> ")
        if len(msg) == 0:
            print("@ got null message")
            continue

        if msg.startswith("/friend "):
            try:
                _friend = FRIENDS[msg[len("/friend "):].strip()]
            except KeyError:
                print("@ got invalid friend name")
                continue
            friend = _friend
            continue

        if friend == "":
            print("@ friend is null, use /friend to set")
            continue 

        resp_hle = requests.post(
            HLE_URL+"/api/message/encrypt", 
            json={"public_key": friend, "hex_data": msg.encode("utf-8").hex()}
        )
        if resp_hle.status_code != 200:
            print("@ got response error from HLE (/api/message/encrypt)")
            continue 
        
        resp_hlt = requests.post(
            HLT_URL+"/api/network/message", 
            data=resp_hle.content
        )
        if resp_hlt.status_code != 200:
            print("@ got response error from HLT (/api/network/message)")
            continue 

def get_friend_name(user_id):
    for k, v in FRIENDS.items():
        friend_id = hashlib.sha256(v.encode('utf-8')).hexdigest()
        if user_id == friend_id:
            return k
    return ""

if __name__ == "__main__":
    main()

We launch locally and for sale

Now all we have to do is run the written script. Depending on hlt_host we can run this script as in local environment, and directly on before, By using the Hidden Lake platform. First, let’s try to run all this wonder in a local environment.

In the repository go-peer an example has already been prepared with the automatic start of two nodes, that is, two HLE services with two different config.yml files. At the same time, the HLT service rises alone, but this is not necessary. HLT services can communicate with each other quite flexibly due to the fact that they are also relays of received messages. Therefore, the problem of centralization does not particularly arise here.

To run the example, you must first download the go-peer repository itself, then go to the directory examples/secure_messenger and write the command make. After this action, two HLE services and one HLT will be launched. The example requires the compilation of these two services, which requires the Go compiler. If you do not have a compiler, both programs can be downloaded from the go-peer release version from the link.

$ cd examples/secure_messenger
$ make
... # Логи очистки, компиляции, копирования
[INFO] 2023/12/25 01:41:34 HLE is running...
[INFO] 2023/12/25 01:41:34 HLE is running...
[INFO] 2023/12/25 01:41:34 HLT is running...

Now we create two terminals. From one terminal, go to the directory examples/secure_messenger/node1from under another to the directory examples/secure_messenger/node2. We write the command on the first node /friend Aliceon the second team /friend Bob. After the switch, we can start messaging each other. The switch itself tells only about whom we will write a message to. It has no effect on receiving messages.

# Терминал#1
$ python3 main.py                                                                                INT ✘  16s  
> /friend Alice
> hello
> [Alice]: world
>

# Терминал#2
$ python3 main.py                                                                                INT ✘  13s  
> /friend Bob
> [Bob]: hello
> world
>

Example of a chat with two nodes

After a successful local launch, we can also test the operation of our application already on sale, by using the ready-made Hidden Lake network. To do this, you need to take a list of existing HLT services. They are all in the README.

Existing HLT services on the Hidden Lake network

To switch to another service, we need to change in the configurations config.yml and hle.yml one field at a time. Let’s take the fifth service HLTs, which is used only for storing messages. Change the field in config.yml hlt_host with localhost:6582 on 195.43.4.253:9582. In hle.yml it is necessary that the field network_key corresponded j2BR39JfDf7Bajx3.

Logs of HLE services. WARN – An unsuccessful attempt to decrypt the message. Our application may receive messages that are not addressed to us, so we will not be able to read them.

After these changes, restart HLE services. This can be done with the help of make clean and again make. That’s it, now we also send and receive messages, but already connected to the external HLT service.

Conclusion

Thus, we wrote a messenger with end-to-end encryption on top of Hidden Lake services in the Python programming language. The entire source code of this messenger can be found here. You can run and check its operation with two locally created nodes here.

Related posts