Review on the Evaluation Frameworks of Non-academic Impact of Scientific Research

Assessing the academic impact of scientific research had made great achievement, but nonacademic impact had been neglected for long time in China. This paper used documentary research to analyze representative evaluation frameworks and then summarize the corresponding relation between different evaluation frameworks and their guiding theories. This paper believed that people need to analyze the different conditions when using them because of different frameworks with different background. At last this paper advocated the non-academic impact of research should be paid attention to knowledge consumers from the wider social class, not only to decision makers. It was also well worth studying the timing of non-academic impact of research.


Introduction
The research findings can be recognized and used that was the only way to realize the value of scientific research.They can be recognized by academic peers which we called peer review (academic impact), or by the public by the way of social applications which we called non-academic impact.The former had been concerned, while the latter had been neglected for a long time.
What did non-academic impact mean?In short it meant all kinds of impact beyond the academic circle, including political, cultural, economic, environmental and social impact, and so on.There were many similar names such as "societal impact of research", "usefulness of research", "public values", "knowledge transfer", "societal relevance", and so on.We defined it as social contributions beyond academic contributions.It was social utility in essence.That was to say that research played a role in promoting or hindering social development.From that aspect, we thought that its impact may be good or bad.From the literature and practice cases we can see that the non-academic impact of scientific research sometimes referring to the policy impact--the transformation of knowledge into policy.

The typical evaluation framework of non-academic impact and its application
It was most difficult to measure and evaluate nonacademic impact of scientific research.There have been many models and frameworks to try to solve this problem, but so far there had been no absolute answer.
We would introduce and analyze several representative evaluation frameworks below.

The payback framework
U.S. non-profit research group Rand Corporation used payback framework to evaluate political and social impact of FoW (the Future of Work Programme) being funded by ESRC (Economic and Social Research Council).This framework divided the impact of scientific research into five parts: knowledge contribution, impact on future research, policy impact, social and economic impact.And also divided nonacademic impact of scientific research into five parts: policy impact, potential policy impact, practice impact, potential practice impact and the impact on researchers' career promotion.It thought knowledge playing a role in changing potential users' inner cognition was sometimes soon and sometimes slow.Therefore it was very important to identify its time span.• Policy environment played a decisive role in nonacademic impact.
• The media played a supporting role in nonacademic impact.
• Lack of application record mechanism led to track the non-academic impact more difficult.
• In general, research projects rarely caused radical policy change, usually playing a role in waving of public opinion or leading policy fine-tuning.
Payback framework used the forward-tracking method to evaluate the non-academic impact, and had some limitations: • It was not convincing to evaluate the overall impact of a plan which included many projects by sampling.Because different projects had different ways to play their role in policy and society, there was a unified and same pattern.
• The framework was a linear evaluation model which focused on outcome evaluation, but the process of research impact was much more complex than it in reality.
• It ignored the environment of research, the evaluation conclusions may be effective in some condition, but in another condition was not.
This framework provided a set of practical and operable indicators for evaluating non-academic impact of scientific research, and was widely used in social science.

ESRC framework
This framework was used to track and assess the nonacademic impact of an AIDS project funded by ESRC.It divided assessment elements into three types: research outputs, diffusion channels and impacts.

Research outputs
The codified knowledge being used to record new skills, new methodologies and new tools, or conceptual descriptions.

Diffusion channels
The channels by which research was transferred, such as the media, publication, networks.

Impacts
The ways in which non-academic users used the research outputs, for example helping users directly or indirectly solve the production or life problems, change the users' awareness and judgment.
This framework was not linear.It took into account the sequential and simultaneous impacts.

Lavis et al. examined and analyzed eight cases from
Ontario and Saskatchewan in Canada (four health research programs in each province).They found the key informants for each program according to clues provided by program managers and users.They asked these informants to describe whether health research played a role in decision making, and what were the processes and conditions.They built a framework of impact assessment by investigation.This framework mainly evaluated the impact of health research on decision making, not the societal changes.Users were identified and the impact measures were divided into three categories: • Producer (researcher)-push measures • User (decision maker, the public, patients, clinician, manager, development officer, and so on)pull measures • Exchange measures, joint effort by both groups to solve the problem.
Within each of these categories can be grouped according to the process, the intermediate outputs and the full outputs.
Lavis et al. advocated the receptor mechanism to drive the communication between producers and users.The decision makers need to establish accountability mechanisms for the practical application of projects, not only focus on the citation.If people only evaluated the immediate and direct impacts and ignored the potential and indirect impacts, the important value of research may be covered up, missing the best opportunity to use research results.

CHASS framework
This framework was developed by Council for the Humanities, Arts & Social Science (CHASS) for evaluating research projects funded by CHASS, including the quality, impact and final goals.It consisted of four parts: Part one: sources of funding, mainly referring to the project sponsor, including private and public funding.
Part two: research process.These problems need to be answered as followed: who it was done by? where it was done?
Part three: the outputs of the research, including the publication channels such as the media, the media or the internet.
Part four: the outcomes of research, mainly societal benefits.
CHASS was developed on the basis of Payback Model.It took two-communities theory as a guide and thought that there was a significant difference in the values, the preference of knowledge dissemination between researchers and decision makers.So the evaluation was done according to the source of the project, the research process and the outputs, which directly or indirectly reflected and decided the communication between the two groups.

IDRC framework
Lindquist believed that people should have a realistic anticipation about the potential impact of the projects funded by International Development Research Center (IDRC).The public institutions supported the policy consultation not only for the direct and obvious impacts but also for promoting the negotiation and dialogue among related parties and sharing some good ideas.In fact most ideas and even innovations were not directly transferred into policies, and a few may even be eliminated for various kinds of reasons.Therefore it need classify and identify all kinds of impacts.Lindquist divided the policy influence into nine types (see Table3).IDRC framework summarized the problems of evaluation into three sets: the policy problems, the objectives and expectations and key outputs and events (see Fig. 1).

Key outputs and events and influence during he process of the project
• What was the problem, gap and opportunity?• Who was responsible for dealing with these problems?• What were their analytical capacities for these problems?• What were the dominant?• What were key events or decisive moment shaping the policy situation, such as the changes of internal government or the emergence of new crisis.• Whether decision-making mechanism was routine or progressive, basic or temporary?
• What was the motivation of the project, and it was for capacity building, for delivering ideas or for pursuing for policy impact?• Who did it try to influence? • Did it depend on policy entrepreneur?Were they government officials?• Did the project foresee the potential difficulties in advance?• Did it need the government to open policy window for it?
• Was the project going well?
• What were key outputs of it?
• What were key events and important external events relating to it？ • Did policy entrepreneur play an anticipated or exceeding role?Was there new policy entrepreneur appeared?• Were there unexpected events or opportunities?• Had the policy impact been reached?Was there an alternative policy impact?• What did the project in the future？   It was found that, unlike natural science, intellectual property was not a main tool to transfer knowledge for social science.Informal exchange and cooperation was a more common than it.
The survey paid special attention to the potential users and beneficiaries.But the results showed the communication between researchers and potential users was disappointing, and some researchers even didn't know who their users were.The main reason may not lie in users (such as the differences of their education and occupation), but the research was not sufficiently disseminated by CSCI.Academic papers were most important indicator of career promotion in CSCI, but knowledge transfer was realized mainly through some commercial activities such as R&D contracts, patents, licenses and derivatives, which were appropriate criteria in natural science but not appropriate in social science.

The guiding theories of evaluation frameworks 3.1 The guiding theories
Many theories and models were put forth when scholars analyzed knowledge transfer and nonacademic impact of research.On the basis of knowledge flow between researchers and users, this paper summarize a logical system for it (see Table5).The frameworks in the second chapter were summarized by research practice on basis of these models and theories.Table6 summarized the guiding theories of these frameworks.

Characteristics analysis
Through the review of evaluation framework, we found that: (1) The research of non-academic impact was not a new direction without any progress, and it had achieved a lot in the aspects of theories and frameworks.
(2) An evaluation framework was often designed for a specific organization or even for a project to satisfy users' demands in line with the objective conditions at that time.Therefore there was no uniform framework to be generally applicable to all problems of non-academic impact.When we assessed non-academic impacts of research we should design a case-by-case framework according to its concrete realities and conditions in order to the objectivity and accuracy of results.
(3)These frameworks abstracted the "influence" into "process".These frameworks evaluated the nonimpact of research mainly based on the project process and time sequence.

Conclusions
The non-academic impact of research was paid more and more attention with social development.These frameworks broke through narrow understanding about "impact".The non-impact of research included not only the policy impact but also social, economic, cultural and environmental impact.Through a comprehensive study, we believed there were following aspects worthed thinking: (1) The non-academic impact of research would be a long-term and top topic for a long time.The topic on non-academic impact of research remained largely ignored until a so-called New Social Compact carrying over to social science.With the emergence of these concepts such as new public administration, new public management, decision consultation, think tank and so on, and with the change of assessment of publicly funded research from academic value evaluation to practical value evaluation, the non-academic impact of research would be a long-term topic.
(2) The studies on non-academic impact often paid more attention to producers rather than users.How can users discover these projects and get new information and absorb new knowledge from them, and what was the result?Those questions need we thoroughly ponder and research.The "policy narrative" theory of Roe was just a start.
(3) The studies on non-academic impact paid more attention to the transfer between knowledge and policy, and paid more attention to policymakers' demand.That showed research was a strongly utilitarian.The users of the publicly funded projects were not only policymakers, but also people of all walks of life who can also spread knowledge and had an influence on policy through various channels.So the future research should also pay attention to the different users' demand and the evaluation criteria should be more pluralistic.
(4) The research of time law on non-academic impact.Garfield believed that paper cited peak time often was two years.Enlightened by Garfield's cited peak time theory and Bernal's half-life of literature, We also asked whether non-academic impact of research was time limited.We thought there can be no doubt about the answer, because research was carried out in a certain environment, and nonacademic impact must be environmentally sensitive.The social value of some basic research may have emerged three or five years later.It was important for researchers to have the patience to wait for an appropriate occasion.However, the non-academic impact of the research for hot spots of society was more time-limited.It was well worth studying the timing of non-academic impact.

Table 1 .
Impact classification of science research Through questionnaire, interview and case analysis, this report drew some valuable conclusions:

Table 3 .
Three types of policy impacts of research in IDRC framework

Table 5 .
The model system of research utilization