For example, some of the key learnings from the evaluation of products and personnel often apply to the evaluation of programs and policies and vice versa. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. The authors propose a new definition for measurement process based on the identification of the type of measurand and other metrological elements at each measurement process identified. The REF will therefore assess three aspects of research: Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment (REF2014 2011a). Different authors have different notions of educational evaluation. 0000008591 00000 n Although based on the RQF, the REF did not adopt all of the suggestions held within, for example, the option of allowing research groups to opt out of impact assessment should the nature or stage of research deem it unsuitable (Donovan 2008). Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning. Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. In the Brunel model, depth refers to the degree to which the research has influenced or caused change, whereas spread refers to the extent to which the change has occurred and influenced end users. This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. (2005), Wooding et al. This work was supported by Jisc [DIINN10]. A Review of International Practice, HM Treasury, Department for Education and Skills, Department of Trade and Industry, Yes, Research can Inform Health Policy; But can we Bridge the Do-Knowing its been Done Gap?, Council for Industry and Higher Education, UK Innovation Research Centre. Table 1 summarizes some of the advantages and disadvantages of the case study approach. Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines. What are the challenges associated with understanding and evaluating research impact? 0000346296 00000 n Definitions of Evaluation ( by different authors) According to Hanna- "The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". The . At least, this is the function which it should perform for society. This might include the citation of a piece of research in policy documents or reference to a piece of research being cited within the media. More details on SROI can be found in A guide to Social Return on Investment produced by The SROI Network (2012). PDF Decentralization: a Sampling of Definitions - Undp If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). 0000002109 00000 n The RQF pioneered the case study approach to assessing research impact; however, with a change in government in 2007, this framework was never implemented in Australia, although it has since been taken up and adapted for the UK REF. To demonstrate to government, stakeholders, and the wider public the value of research. Impact is often the culmination of work within spanning research communities (Duryea et al. Co-author. CERIF (Common European Research Information Format) was developed for this purpose, first released in 1991; a number of projects and systems across Europe such as the ERC Research Information System (Mugabushaka and Papazoglou 2012) are being developed as CERIF-compatible. Evaluation Research - an overview | ScienceDirect Topics Definition of Evaluation "Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have." Mary Thorpe 2. Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. There are areas of basic research where the impacts are so far removed from the research or are impractical to demonstrate; in these cases, it might be prudent to accept the limitations of impact assessment, and provide the potential for exclusion in appropriate circumstances. These metrics may be used in the UK to understand the benefits of research within academia and are often incorporated into the broader perspective of impact seen internationally, for example, within the Excellence in Research for Australia and using Star Metrics in the USA, in which quantitative measures are used to assess impact, for example, publications, citation, and research income. Research findings including outputs (e.g., presentations and publications), Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc), Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics), Research developments (based on stakeholder input and discussions), Outcomes (e.g., commercial and cultural, citations), Impacts (changes, e.g., behavioural and economic). HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact. While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. For full access to this pdf, sign in to an existing account, or purchase an annual subscription. Impact can be temporary or long-lasting. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. 0000012122 00000 n 4. The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. However, the Achilles heel of any such attempt, as critics suggest, is the creation of a system that rewards what it can measure and codify, with the knock-on effect of directing research projects to deliver within the measures and categories that reward. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. 2010; Hanney and Gonzlez-Block 2011) and can be thought of in two parts: a model that allows the research and subsequent dissemination process to be broken into specific components within which the benefits of research can be studied, and second, a multi-dimensional classification scheme into which the various outputs, outcomes, and impacts can be placed (Hanney and Gonzalez Block 2011). It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. From 2014, research within UK universities and institutions will be assessed through the REF; this will replace the Research Assessment Exercise, which has been used to assess UK research since the 1980s. Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. 0000001087 00000 n This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. What are the challenges associated with understanding and evaluating research impact? This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. Author: HPER Created Date: 3/2/2007 10:12:16 AM . 6. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. What is The Concept of Evaluation With its Importance? Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. Many times . Dennis Atsu Dake. The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. 2007). working paper). A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. In demonstrating research impact, we can provide accountability upwards to funders and downwards to users on a project and strategic basis (Kelly and McNicoll 2011). What is the Concept and Importance of Continuous and Comprehensive Evaluation. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. The exploitation of research to provide impact occurs through a complex variety of processes, individuals, and organizations, and therefore, attributing the contribution made by a specific individual, piece of research, funding, strategy, or organization to an impact is not straight forward. This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. 0000003495 00000 n What indicators, evidence, and impacts need to be captured within developing systems? Test, measurement, and evaluation: Understanding and use of the 10312. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter. Definition of Assessment & Evaluation in Education by Different Authors with Its Characteristics, Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have., 2. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Any tool for impact evaluation needs to be flexible, such that it enables access to impact data for a variety of purposes (Scoble et al. 0000002868 00000 n This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. 0000006922 00000 n 2008; CAHS 2009; Spaapen et al. 0000001883 00000 n Reviews and guidance on developing and evidencing impact in particular disciplines include the London School of Economics (LSE) Public Policy Groups impact handbook (LSE n.d.), a review of the social and economic impacts arising from the arts produced by Reeve (Reeves 2002), and a review by Kuruvilla et al. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on.