On February 13, 2024, “Sber” reported that its neural network GigaChat passed the examination for the referral of training “medical business”, which is necessary to obtain the qualification of “physician-physician” in the Russian Federation, just like any student who has completed 6 courses of a medical university according to the federal state educational standard.
The neural network was tested and received a score of 4, scoring 82% at a pass threshold of 70%. The exam was accepted by the committee of professors of therapy, surgery, obstetrics and gynecology of the Institute of Medical Education of the NMIC named after V.A. Almazova.
The standard ticket for the oral exam contained three situational tasks from therapy, surgery, obstetrics and gynecology, as well as 3-5 questions related to them (“specify the expected diagnosis”, “draw up a treatment plan”, “prescribe additional examinations”, etc.). GigaChat was also tested by answering a 100-question questionnaire.
“The GigaChat neural network model is developing very quickly, mastering new fields of knowledge. Together with our partners across the country, we will continue to develop digital solutions and technologies for medicine and health. Today, I would like to acknowledge the role and thank the Almazov Center employees, who provide control over model training and its validation. The first stage – taking the doctor’s exam – was passed. Ahead are new tasks and the discovery of new applied solutions based on existing capabilities. In the future, the model can become the basis for creating a doctor’s and patient’s assistant, provide new conditions for care and knowledge about one’s health for each person, and become an essential aid for the clinician in his daily practice. The use of large language models and their followers will become one of the key technologies for the development of human-centered health care,” Serhii Zhdanov, director of the “Sbera” Health Industry Center, explained.
“The project of teaching the large language model GigaChat medical knowledge at the level of a graduate of a medical university became a great challenge for the Almazov Center. Several hundred teachers and researchers are participating in the project. Residents and students actively joined the work. We are satisfied with the current results and the training of the model will continue. Already now, together with Sber, we have planned a whole line of applied solutions for medical institutions, patients and doctors based on GigaChat, the development of which will begin this year,” said the Director General of the FSBU “National Medical Research Center named after V.A. Almazov” of the Ministry of Health of Russia, President of the Russian Cardiology Society Yevhen Shlyakhto.
The developers of the neural network explained that taking the exam using the GigaChat model is the result of the effective joint work of specialists of the “Sbera” Health Industry Center, NMIT named after V. A. Almazov and the GigaChat team. Training the neural network algorithm with medical knowledge took six months. As part of the preparation for the medical examination, a dataset of 42 GB of specialized information was used, including educational and methodical materials recommended for teaching students at Russian medical universities, monographs, methodical manuals, scientific articles, and depersonalized medical data. At the same time, the model is not a doctor, the recommendations received from her must be approved with the doctor.
On February 10, 2023, the media reported that ChatGPT passed the United States Medical Licensing Examination (USMLE) for the right to practice as a doctor in the United States. The chatbot successfully answered 88.9% of the questions and tasks of the test out of 350 medical tasks with a passing threshold of 60%. As part of the scientific study, two practitioners scored the ChatGPT responses, and discrepancies in test results were reviewed separately by a third expert. USMLE is an exam that must be passed in order to have the right to work as a doctor in the USA and a number of other countries (Israel, New Zealand, Qatar). This test is considered one of the most difficult medical exams in the world, and passing it confirms the high level of knowledge and competence of a novice doctor, for example, for students of higher courses in pediatrics or general practice specialists from other countries. A test of ChatGPT found that the neural network provided “at least one important discovery” during diagnosis that was “novel, non-obvious and clinically valid” within its correct answers.