Approaches to formation of the digital learning estimation system in the Russian Federation

The article is devoted to consideration of mechanisms of evaluation of the MOOC in the global context and to the development of the Russian e-learning market. The criteria and mechanism for assessing the quality of online courses by consumers in the development contest of the Russian electronic digital environment are proposed. The approbation of the proposed mechanism was carried out and conclusions were made on the significance of the quality assessment of MOOC with their implication in the educational process.


Introduction
The transition to a new technological order and the increase in the flow of information in the social space are actualizing the development of new forms of education that would meet these challenges. Since it is the system of vocational education that requires the development of new forms of development of the educational environment allowing implementation of the principle of life-long learning, which meets the modern requirements of the global labor market. The trend towards the development of digital technologies has already significantly changed the educational space, in particular the higher school, democratizing the process of obtaining education and allowing to speed up the process of updating knowledge and improving the skills of people who already have an education. The understanding of the need for the development of e-learning to be one of the most important ways to overcome the global challenges faced by the education system was reflected in a number of international strategic planning documents, in particular the Icheon Declaration adopted at the World Education Forum in May 2015 and outlined the development priorities for the education systems of countries -participants of UNESCO [1]. The most current form of implementing e-learning in higher and additional professional education is Massive Open Online Course (MOOC) [2,3].

International experience
The appeal to e-learning as one of the necessary elements of a modern vocational education system to address the global challenges of the labor market has revealed a number of problems related to the need to assess the quality of such technologies, as well as the conditions and results of developing such programs [4,5]. In this regard, a number of international organizations have proposed their own mechanisms for assessing the quality of e-learning, for example Quality Matters TM [6] and Open ECB Check-Quality [7], which provide a system of criteria and indicators for the evaluation of online courses, and the creation of an expert community for their implementation in e-learning. It should be noted that the listed formats are primarily focused on peer review and suggest some mechanism for certifying courses based on the results of the evaluation.

Russian experience
In the Russian Federation, there is currently no common approach to assessing the quality of e-learning. However, there is already a significant reserve associated with its implementation. The effectiveness of the implementation of the MEP in the university's educational environment and the educational programs implemented by it are demonstrated, in particular, by the Higher School of Economics [8] and other successfully developing Russian universities [9].
The priority project "Modern digital educational environment" (the Project passport approved by the Protocol of the meeting of the Presidium of the Council under the President of the Russian Federation for Strategic Development and Priority Projects of October 25, 2016 No. 9) aims to create conditions for a systemic improvement of quality and empowerment by 2018 continuous education for all categories of citizens through the development of the Russian digital educational space and the increase in the number of trained educational organizations, having mastered x online courses up to 11 million people by the end of 2025. As one of the results of the project, it is planned to "create a system for assessing the quality of online courses and online resources of general education, combining automatic and peer review, which provides the formation of a rating." As we see, the declared system assumes the creation of a unified system for assessing courses on the territory of the Russian Federation.

The proposed evaluation mechanism
Simultaneously with the expert review, a number of Russian organizations consider it appropriate to develop mechanisms for assessing the quality of e-learning and the MOOC by consumers that could be integrated into the national system for assessing the quality of e-learning. To this end, the All-Russian Public Organization "For Quality Education" in cooperation with the Commission for the Quality of Education of the Council of the Russian Ministry of Education and Science for Youth Affairs and the "Internet + Education" Competence Center, autonomous noncommercial organization "Institute for Internet Development" proposed a project to consolidate the interests of consumers of elearning educational services.
In the framework of this activity, focus groups were held in 10 Russian universities, the result of their work was a single document -"The Consumer Quality Standard for Education Implemented with the Use of E-Learning", organized on the basis of the "Standards and Recommendations for the Quality Assurance of Higher Education in the European Space (ESG)" [10]. It was divided into structural blocks; we omit the descriptive aspect and give the contents of the main sections.
1) Assessment of the educational content quality This section contains indicators and criteria related to the content of the course. In the section the main requirements of the consumer to the content of the course are considered.

2) Evaluation of the results of e-learning and acknowledgment of e-learning by educational organizations and employers
The section contains the criteria and indicators of the consumer to the system for assessing the results of completed courses, as well as to the procedures for verification, identification and setting off the results of courses in higher education organizations, considering them by employers.
3) Assessment of architectural requirements The section describes the requirements for the design and interface of the courses, as well as the methods of information transfer. Criteria for assessing the quality by consumers were focused primarily on the availability of materials, digestibility and the convenience of the interface and self-evaluation of learning outcomes. These criteria do not duplicate and do not substitute for professional and expert evaluation, but supplement it, allowing to characterize the needs of consumers, which will allow the manufacturers of the courses to adjust their production in their interests.

Approbation of the proposed model
As a result, a mechanism was proposed to complement the system of peer review that was being developed in the Russian Federation by criteria and mechanisms that would allow to see the view of the consumer of online courses and to provide a mechanism for adequate feedback for the platform creators. As an important factor in the effectiveness of its implementation, the mass appeal of users of the MOOC should appear, which will allow to rank the courses effectively. In order to approve the proposed mechanisms within the framework of the special project of the contest of best e-learning practices EdCrunch Award (within the framework of international forum EdCrunch-2017 held annually by National University of Science and Technology "MISiS", I. Rybakov Charitable Foundation and the Ministry of Education and Science of the Russian Federation), a special nomination "Best course from the consumer's point of view (student evaluation)".
More than 100 MOOC consumers and more than 40 courses offered at leading Russian platforms and meeting the basic criteria of the MOOC took part in the approbation. The participation of platforms was not of a declarative nature, but, on the contrary, it was performed by selected students anonymously after passing the course and without informing its creators. This made possible to make the assessment more objective and to guarantee the impartiality of consumers. The high status of the competition was emphasized by the fact that the award was given to the winners by the Deputy Prime Minister of the Government of the Russian Federation A. Dvorkovich and the Deputy Minister of Education and Science of the Russian Federation L. Ogorodova.
The approbation of the consumer assessment of MOOC within the framework of the EdCrunch Award allowed to draw several conclusions.
Consumers show the greatest interest in the most practical courses or soft skills courses; theoretical courses have attracted the least interest. In particular, the most relevant were MOOCs devoted to data analysis in Google Analytics and the implementation of own business idea in the framework of technological entrepreneurship. This allows making a cautious conclusion that consumers are eager to train via e-learning exactly the competencies they often lack in formal education. The approbation of the proposed mechanism was very conditional in the presented sample, but it demonstrates the obvious productivity of attracting consumers to the assessment of the MOOC.
Global challenges, expressed in the need for the comprehensive development of human capital and the continuous improvement of the labor resources skills in connection with the changing technological order, can be overcome with regard to e-learning only if it is neatly implemented and a multi-level system for assessing its quality. In the Russian Federation favorable conditions were created for the development of e-learning and its implementation in university educational programs. The quality assurance system for such training includes an automated assessment of the course data and an expert evaluation. Its supplementation with the consumer appraisal, in our opinion, would make the system more flexible and enhance its ability to take into account the aspirations of students and their needs.