How to extract candles from all shares of Mosbirzhi

How to extract candles from all shares of Mosbirzhi

At the end of last year, I wrote about how you can use Algopack to extract background information about all stocks on the Moss exchange. Gave an example of my first python script that uses the moexalgo library for Algopack and indicated plans to add it to extract all historical data.

A program for downloading historical data from Mosbirzh

The time has come and I want to share with you my first full-fledged, although still console-based, program, with which you can download all historical data (OHLCV candles) with different timeframes (1min, 10, 60, day) on all shares of Mossbirge, legally and using exchanges of the official library.

The program is also written in Python and compiled by the PyInstaller installer, should run on any Windows computer, and can be downloaded from github.

Current statistics program by data mined: stores 996 historical data files with 114.6 million records (candles), the total amount of stored data is 9.15 GB. To obtain these statistics, I added a separate script that can simply be placed next to the program file and run when interest arises. You can download it on Github.

ALGOPACK

The program uses the official moexalgo library to work with Algopack. What was interesting to find out?

  1. Negative moment. Already after compiling the program during testing, Algopak suddenly gave an error when cyclically accessing it, began to understand – the error concerns the “VEON-RX” action. In a telegram, the representative of Mosbirge promised to fix this “jam”. But, unfortunately, after two weeks, the situation has not changed. I had to make an error handler. Perhaps there will be similar features for some other promotion.

  2. A positive point. On the official page of the ALGOPACK Mosbirzhi library, it is stated that the history of Super Candles has been provided since 2020. However, as it turned out, in fact, there is no such restriction, the data start dates differ for all shares and start from 2011. For example, on Sber minutes are available from 2011-12-15 10:00:00 and now 1.8 million records in the amount of 160MB are being downloaded

How to work with the program

The program is easy to use. At the first start of the program, you will be asked to specify the main parameters, which will depend on the further operation of the program.

Scanning and creating a list of shares

At the very beginning, all mosbourge stocks available for download will be scanned, and the earliest dates from which historical data is available will be analyzed. It will take 1-2 minutes.

As a result, a folder will be created in the root of the program “historical_data” which will contain 3 files:

logfile.txt Log file. Contains a report – useful and most important information about the operation of the program. It is advisable to watch after the end of the program, because it always refers to this file. Logs record complete information on all promotions (number of downloaded candles, period, file size), etc.

list_tools.txt The file contains complete background information on all shares of Mosbirzhi. The file is for reference only and does not affect the operation of the program in any way. The decoding of the names of all columns can be viewed here.

list_tools_listlevel.txt The most important file, it contains a list of all stocks, indicating the listing level (sort by this parameter) and the first dates that are actually available for download from Mosbourge at the time of launch. Data mining will take place on the data of this file, so its never delete!

Edit the list of shares and dates

You can not change anything and download all available data for all promotions. You can also delete lines with unnecessary actions from the file ‘list_tools_listlevel.txt’. In this file, in the FIRST_DATE column, you can change the starting dates of receiving historical data for each share separately. Edit this file. In the future, the program will focus on this file.

Setting timeframes

Next, the program will ask about the timeframes you need. The program can download historical data with different timeframes: 1 minute, 10 minutes, 60 minutes, and days.

You need to specify a specific timeframe or select all options. Your answer will be placed in the log file. With all subsequent launches of the program, data will be downloaded and updated according to your initial settings.

Extraction of historical data

After the parameters you set, the data download process begins when the program is first launched. If you have selected all timeframes, then the download takes place in a cycle – first, daily candles for all shares, then watch, 10-minute and minute candles.

Information in the terminal will be constantly updated. The running time of the program can vary significantly from the amount of data being downloaded and range from a few minutes to several hours (10 or more). If your computer goes to sleep, the program will simply turn off, so if you want to download all the data at once, you need to cancel the ability of the computer to go into sleep mode. If there was a failure or you turned off the program yourself, the next time you start it, the data will be downloaded automatically.

Updating historical data

To update historical data, simply run the program. You can do it with any frequency. When restarted, the program will report that the folder ‘historical_data’ already exists before You started the program. In a new session, the program will simply download everything that you did not have time to download earlier, and all historical data that appeared again will be downloaded.

Change program settings

If you are not satisfied with the initial settings made during the first launch of the program and you want to change them – just delete the folder ‘historical_data’ or rename it, for example ‘historical_data_old’. The running program will again ask for all the parameters as it did when it was first started.

In the near future, I plan to move from the console view of the program to a full-fledged graphical interface, as well as expand the range of information received.

This will be the subject of the next post, stay tuned here or Algotrading on Python. To be continued..

Video on the topic:

Related posts