Evaluation. What associations does this word bring to your mind? 

Sure, all of us now have some different perception on evaluation in our minds and, most probably, have a lot of “What?” and “Why?” questions. And one of the first questions that could come to your mind is “What is evaluation?”.

So, obviously, to begin the Module “Evaluation in youth work”, we first need to understand what evaluation means.

There are many definitions of evaluation depending on the context, discipline and sphere of implementation.  So for better understanding of concept of evaluation, our main point of interest in the frames of this Module, we will look through different opinions and definitions and will finally find a common thread and common understanding of concept of evaluation.

After getting a general understanding of what evaluation means we will dive deeper into the topic and find out what are the main aims and objectives of evaluation in youth work. So we will answer two more basic questions “What is evaluation for?” and “Why to evaluate?”.

Together with the question “Why evaluate?”, “What to evaluate?” is another key to understanding  the different options and approaches to educational evaluation available to us in our youth work. To answer this question, we will find out some evaluation models, which propose a certain structure of “What to evaluate?.

Having reviewed the possible definitions, aims, objectives and models, it will be not difficult to identify the actors of educational evaluation and answer the questions “who?” and “for whom?” (that are deliberately formulated together because everybody involved in the educational process should participate in its evaluation and should be informed about its results)

About when to evaluate. Here or in 10/2

Types of evaluation.

Although there are no “golden theoretical rules” for the development of a perfect educational evaluation, but we believe, that the definitions, models and questions explored in this mostly theoretical part of the Module “Evaluation in youth work” can help a lot in planning, preparing and actually conducting an evaluation process full of quality, free of resistance and mishaps.

So let’s make our first step of a journey that promises to bring us to the point where we will have enough skills to conduct effective evaluation in the frames of youth work and educational projects for youth.

*****

1.1. Definitions

The first thing that needs to be understood when talking about evaluation is what evaluation means. There are many definitions of evaluation depending on the context and discipline.

“Evaluation is a systematic process of collecting, analysing and interpreting information to determine the extent to which pupil’s are achievement instructional objectives.” (Norman E. Gronlund and Robert L. Linn)

“Evaluation is the process of determining merit, worth, or significance; an evaluation is a product of that process”(Scriven, 1991, p. 53)

“Educational evaluation is the systematic investigation, observation and interpretation of information.” (Tenbrink, T. & Cooper, J. M. (2003). Educator’s Guide. P. 64)

“Educational evaluation is a method (procedure) and to prove if the expectations and aims of an educational process reflect reality (results of the process).” (Nydia Elola, Lilia V. Toranzos 2000 Evaluación educativa: una aproximación conceptual. P. 2)

Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions (Patton, 1987)

Although here are many definitions of evaluation depending on the context and discipline, most of them have a common thread. All the definitions in common talk about the process  of determining to what extent the educational objectives are actually being realized.

So, to sum up, for the purpose of this Module, we will define evaluation as a structured process of assessing the success of a project in meeting its goals and to reflect on the learning outcomes.

It is necessary to distinguish educational evaluation from the process of collecting and obtaining certain kinds of information. The collection of information is something punctual, and it is done in the evaluation process at certain moments. But educational evaluation is an ongoing process. It implies judgment (good, bad, acceptable, OK, advantageous, disadvantageous, of high quality, of low quality, etc.) Educational evaluation implies measurement. But, educational evaluation goes beyond the mere measurement: it provides explanations and conclusions. For the purpose of this Module all the attention will be focused on educational evaluation.

The analysis of all the above definitions makes us able to draw following characteristics of evaluation:

1. Evaluation implies a systematic process which omits the casual uncontrolled observation participants.

2. Evaluation is a continuous process. In an ideal situation, the learning process on the one hand and the evaluation procedure on the other hand, go together. It is certainly a wrong belief that the evaluation procedure follows the learning process.

3. Evaluation emphasises the broad personality changes and major objectives of an educational programme. Therefore, it includes not only subject-matter achievements but also attitudes, interests and ideals, ways of thinking, work habits and personal and social adaptability.

4. Evaluation always assumes that educational objectives have previously been identified and defined. This is the reason why youth workers are expected not to lose sight of educational objectives while planning and carrying out the learning process.

5. A comprehensive programme of evaluation involves the use of many procedures (for example, analytico-synthetic, heuristic, experimental, lecture, etc.); a great variety of tests (for example, essay type, objective type, etc.); and other necessary techniques (for example, socio-metric, controlled-observation techniques, etc.).

6. Learning is more important than teaching. Teaching has no value if it does not result in learning.

7. Objectives and accordingly learning experiences should be so relevant that ultimately they should direct the pupils towards the accomplishment of educational goals.

8. To assess the participants and their complete development brought about through education is evaluation.

9. Evaluation is the determination of the congruence between the performance and objectives.

YouTube video: What is evaluation (non-profit sector) https://www.youtube.com/watch?v=ZT0x1Tmp60s

1.2 Aims and objectives

Educational evaluation has a huge meaning in delivering high-quality educational projects. The analysis of all the above definitions and nature of education makes us able to draw following reasons to conduct evaluation in the frame of educational project. 

Reasons to evaluate in the frames of projects:

• To assess whether a project has achieved its intended goals 
• To understand how the project has achieved its intended purpose, or why it may not have done so 
• To identify how efficient the project was in converting resources (funded and in-kind) into activities, objectives and goals 
• To assess how sustainable and meaningful the project was for participants 
• To inform decision makers about how to build on or improve a project 

Evaluation has several aims. The first and most basic aim of educational evaluation is to learn.  Also the evaluation process should lead to improvements and change. And among the other, educational evaluation is an opportunity both to promote the values of participation and to practice it. 

In addition to the general aims of educational evaluation it is possible to identify some objectives or uses of educational evaluation.

Objestives:

• To plan better - Educational evaluation can help to change things and to plan “different things”, but it can also help us to plan things better, in order to prevent negative consequences and to compensate for possible shortcomings
• To take stock of achievements - it is important to recognise, name and give value to the achievements of the educational process so that they do not get lost or not sufficiently used
• To consolidate results - Identified results can be consolidated by making them explicit at the end of the evaluation process. The description, sharing and further use of results are natural follow-up steps of educational evaluation
• To check if we met the interests of the funding institutions - When funding institutions support a certain educational project, they do it according to certain criteria: the nature of the project, its objectives, their priorities. Funding institutions usually ask to receive a descriptive and evaluative report at the end of the project. Even so, educational evaluation plans and criteria should not be limited to the expectations of the organisations that fund the project. But, it is important to consider and include them. Usually this is not a difficult exercise: in most cases the questions they would like to answer through evaluation would be part of our evaluation anyway.
• To reinforce co-operation with partners - If partners are involved in the educational project, they will be involved in its evaluation. A constructive and participative evaluation will naturally reinforce co-operation. But, even if your partners are not directly involved, the results of the evaluation can be of interest to them. You might share new ideas for common projects, other fields of common interest and ways of co-operating, new partners and networks with them (t-kit)

The following objectives are linked to the implementation and practice of educational evaluation and, to the uses of its outcomes at different moments of the process. The following list is certainly not exhaustive. Educational evaluation has a lot of potential and uses.

1.3 Evaluation models

In order to guide trainers in conducting effective evaluations researchers have developed models. There are dozens of learning evaluation models currently in practice. The most commonly used models are: Kirkpatrick, CIPP, Bushnell and Brinkerhoff. Each of these four models has a somewhat different perspective, but there are also similarities.

1. Kirkpatrick's model 

Donald Kirkpatrick developed one of the most commonly used models for evaluating training. Kirkpatrick's model of learning evaluation has been used for more than 50 years. 

It is a four-level model of training evaluation that allows the measurement of different training outcomes including participant reactions, learning, on-the-job behavior, and organizational results.

Evaluation Models: Key Points

Level 1. Reaction – Were the participants pleased with the activity? Did the participants like  it?

Level one measures feelings, energy, enthusiasm, interest, attitude and support. However, what is reflected at this level is words, not deeds (usually in evaluation forms or oral evaluation). As with any exercise in interpersonal feedback there is a possibility of dissonance between what people say and what they eventually do. The daily evaluation groups, mid-term evaluation and final evaluation form are the most common methods used for evaluation on this level. 

Some examples of final evaluation forms and a design for daily evaluation groups are presented in Appendix 2.

Level 2. Learning – What did the participants learn in the activity?

Assessing learning in non-formal education and in youth training is a rather complicated task. On these training courses the knowledge and skills transferred are very much interrelated with the changes in awareness, attitudes and reflection of the participants. For formal training courses, tests and various certified forms are used. In non-formal youth training based on voluntary participation this can have counterproductive effects.

In non-formal contexts the evaluation of participants (whether they have learned a certain theoretical concept, or a certain skill) can be implemented by giving opportunities to the participants to demonstrate their knowledge and skills. It might be in workshops run by them, role-play and simulation, accompanied by reflection and feedback.

Level 3. Behaviour – Did the participants change their behaviour based on what they learned?

Levels 3 and 4 focus the evaluation on the participants’ reality. It is not easy to analyse behavioural change in participants. In youth work especially, where short training courses are an everyday reality and funding organisations are not eager to finance ex-post evaluation, this situation is heightened. During many youth training activities, the focus is mainly cast on self-realisations and self-knowledge, and their relevance in the context of European youth work. This is very hard to evaluate. How can we tell whether somebody has become a trainer after participating at a ‘training of trainers’ course? Is it even a question we can ask? There is a continuous need to track the participants’ development and achievements. The only reasonable thing that can be done is ex-post evaluation, where after longer periods of time the trainer or the organisers of the training contact the participant and co-assess developments in the meantime. Usually this is done by a questionnaire, due to restrictions of finance and time. Without these restrictions, direct observation or in-depth interviews with the participant would generate more useful data.

Level 4. Results – Did the changes in behaviour positively affect their organisation?

This level of evaluation focuses on the participants’ organisation and the benefits of trained participants to them. Did the participants implement the knowledge gained during the training activity? Did they make an impact on the overall work of the organisation or conditions of their target group?

The main disadvantage at this level is the near impossibility of singling out the long-term impact of the specific training activity as opposed to other learning experiences and stimuli. Participants follow their own path and in the process of their development learn new things which consolidate, refine and challenge what has gone before. It is also important to bear in mind that an impact on an organisation cannot be accomplished by a lone individual, but by people collectively involved in the whole organisation. In this context it is problematic to try and indicate the exact impact of individual participation. 

A long–term training gives an opportunity to evaluate between sessions which can include evaluation on levels 3 & 4.

Levels 1 & 2 can be found in the evaluation designs for almost every training activity today. Levels 3 & 4 are clearly more complicated and time, effort and money consuming, but ultimately crucial in assessing whether a training has really made an impact or not. Therefore, with the environment and limitations of youth work training in mind, evaluations on level 1 and 2 should be executed as a normal standard. Partial coverage of evaluation on level 3 is achievable, in the form of a questionnaire distributed some time after the activity for example.

It is important to be aware of the existence of evaluations on levels 3 & 4 and whenever possible to implement them. Available results can indicate huge justification of the training program and inform a needs assessment for future training activities.

2. Brinkerhoff’s model

Brinkerhoff's Model of Evaluation is based on the Instructional Systems Design training cycle and follows a circular pattern. It stresses the importance of continuous evaluation and the need to change a course of action if the proposed approach is not working.

Evaluation Models: Key Points

1. Goal setting – what is the need? Are these the real needs?

2. Program design – what is required to meet these needs? Is this design going to meet the needs?

3. Program implementation – how do we evaluate the program in practice?

4. Immediate outcomes – did the participants learn? What did they learn?

5. Intermediate or usage outcomes – are the participants implementing their learning?

6. Impacts and worth – did it make it a worthwhile difference to the participants’ organisations and their personal development?

3. CIPP model

Stufflebeam's Context, Input, Process, Product (CIPP) model is one evaluation model that is used for evaluating management training. It involves four types of evaluation and has some connections to the Instructional Systems Design model. The CIPP model emphasizes collecting information from a variety of sources to provide data for making better decisions.

The acronym CIPP represents:

• Context - What needs to be done?
• Input - How should it be done?
• Process - Is it being done?
• Product evaluation - Did it succeed? For example: impact, effectiveness, sustainability, and transportability.

Evaluation Models: Key Points

Originally the CIPP was used mainly to evaluate the impact of the introduction of learning technology but it is now used across the education sector.

To explain product evaluation in the context of evaluating learning technology:

• impact assesses whether the deployment of ICT facilities in teaching and learning has a direct effect on the lecturers and students, what the effects are and whether other aspects of the system change as a result of the development or implementation
• effectiveness checks whether the programme achieves the intended and unintended benefits, or whether it is effective for the purpose of improved teaching and learning for which it was designed.
• transportability measures whether the changes in teaching and learning and its improved effects can be directly attributed or associated to the deployment of ICT.
• sustainability considers the lasting effects of the ICT deployment on the students and lecturers and how well they utilise and maintain it for teaching and learning purposes.

YouTube video: The CIPP Model of Evaluation https://www.youtube.com/watch?v=qOcm5nT-o2U

4. Systems approach (Bushnell) model

Bushnell's Systems Approach to Evaluation is based on the idea that the outcome will be only as good as what goes into the process. It emphasizes that evaluation measurement should occur between each of the stages and between the four stages in the process stage to ensure that the program is well designed and meets its objectives.

Evaluation Models: Key Points

According to this Model evaluation consists of four levels:

1. Input – what goes into the training effort? (trainee qualifications, trainer’s ability, resources etc).

2. Process – how are the planning, design, development and implementation of the activity?

3. Output – what are the participants’ reactions, knowledge, skills gained, behaviour reflected on, attitudes changed?

4. Outcomes – what are the effects on the participants’ organisations?

1.4 Target groups

Having reviewed the possible definitions, aims, operational objectives and fields, it is not difficult to identify the actors of educational evaluation. Everybody involved in the youth work process should participate in its evaluation and should be informed about its results, as follows:

• The participants: as learners and the target group of the activity
• The facilitators, leaders or team members: as responsible persons for animating the activity
• The organisers and partners: as promoters of the activity
• The funders: as supporters of the activity
• The decision-makers: as those “responsible” for considering the results of the evaluation in further decision making processes.

These different actors have different responsibilities and tasks in the evaluation. They should participate at different levels and in different phases of the evaluation. Nevertheless, it is important to involve everybody. The evaluation should be democratic, transparent and simultaneously accessible for all actors concerned.

Sometimes evaluation is conducted by an “external evaluator”. But, this does not play against the idea of involving all the actors. Moreover, one of the tasks of the external evaluator is to involve the different actors and facilitate their interaction during the evaluation process. A common assumption is that the external evaluator will be in a position to do that with more easer and with fewer compromises than so-called “insiders”.

1.5 Typologies of evaluation

Many types of evaluation exist. It is important to understand the different types of evaluation that can be conducted over a program’s life-cycle and when they should be used. 

• Formative and summative. 

Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program. 

Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals.

Within the categories of formative and summative, there are different types of evaluation.

Evaluation: What is it and why do it?

• Process/ impact and outcome evaluation

Process evaluation is used to measure the activities of the program, program quality and who it is reaching. Process evaluation will help answer questions about your program such as:

• Has the project reached the target group?
• Are all project activities reaching all parts of the target group?
• Are participants and other key stakeholders satisfied with all aspects of the project?
• Are all activities being implemented as intended? If not why?
• What if any changes have been made to intended activities?
• Are all materials, information and presentations suitable for the target audience?

Impact evaluation is used to measure the immediate effect of the program and is aligned with the programs objectives. Impact evaluation measures how well the programs objectives (and sub-objectives) have been achieved.

Impact evaluation will help answer questions such as:

• How well has the project achieved its objectives (and sub-objectives)?
• How well have the desired short term changes been achieved?

For example, one of the objectives of the My-Peer project is to provide a safe space and learning environment for young people, without fear of judgment, misunderstanding, harassment or abuse. Impact evaluation will assess the attitudes of young people towards the learning environment and how they perceived it. It may also assess changes in participants’ self esteem, confidence and social connectedness.

Impact evaluation measures the program effectiveness immediate after the completion of the program and up to six months after the completion of the program.

Outcome evaluation is concerned with the long term effects of the program and is generally used to measure the program goal. Consequently, outcome evaluation measures how well the program goal has been achieved.

Outcome evaluation will help answer questions such as:

• Has the overall program goal been achieved?
• What, if any factors outside the program have contributed or hindered the desired change?
• What, if any unintended change has occurred as a result of the program?

In peer-based youth programs outcome evaluation may measure changes to: mental and physical well-being, education and employment and help-seeking behaviours.

Outcome evaluation measures changes at least six months after the implementation of the program (longer term). Although outcome evaluation measures the main goal of the program, it can also be used to assess program objectives over time. It should be noted that it is not always possible or appropriate to conduct outcome evaluation in peer-based programs.

Video: Different types of evaluation. https://cyfar.org/different-types-evaluation-0

• Personal/interpersonal/group

Personal evaluation is the kind of evaluation in which each individual or actor involved in the educational process makes their own judgements and draws their own conclusions about the experience they have had.

Interpersonal evaluation happens when more than one individual actor involved in the educational process shares and discusses their judgements and conclusions. Often this kind of evaluation takes place in a small group setting. While individual judgments may change as a result interpersonal evaluation, arriving to a consensus is not the aim. The purpose is simply to share and discuss those individual evaluations.

Group evaluation has an added dimension. It is not simply an interpersonal evaluation in which the number of actors involved is bigger. Since the group as such is part of the context and very often an important source of learning in non-formal education, group evaluation specifically at aspects and dimensions of the learning process that can be observed and judged from a group point of view, including for example the atmosphere, the co-operation among participants, the contribution of the group to the learning and the group process. It should be noted however that this is not its exclusive function.

• Initial/mid-term and final evaluation

Initial evaluation is the evaluation done at the very beginning. Before we start our project, it is a good idea to take some time out to check our aims, objectives and methodologies. Some questions you might want to ask yourself at this stage could be:

• Are the objectives of our project in line with our overall aim?
• Do the methods chosen serve the fulfilment of this aim?
• Does the program cover everything we want to address? Is it realistic?
• Does our team have the necessary expertise and capacity to run this program or do we need to call in some support?

Information about the participants is another important part of initial evaluation, and is commonly called “profile information”. It can be understood as “an initial snapshot of participants”. This information is vital for checking how much effect your program will have on the lives of young people. In this information there are data, which do not change, for example the age of the participants. This type of information is useful for checking, who on our program is most likely to benefit. Profile information is crucial for designing your program in a way that fits the needs of participants.

The mid-term evaluation is our chance to check how things are going. As the name implies, it takes place during your project. It may be an “ongoing evaluation”, such as daily reviews at the end of each day, or a whole set of methods that take place at the mid-point of the program.

Mid-term evaluation may prove to be extremely useful for project teams, as it gives them the possibility of identifying potential problems or shortcomings as the program progresses. Team members may decide to make necessary changes in the program, to tackle these challenges, or meet additional needs that the participants may raise.

Final evaluation happens when the project is completed. Using a variety of methods, the participants and the team evaluate the whole project in light of the outcomes (i.e. fulfilment of objectives, learning achievements, organisational implications and impact in a wider social context) with a certain distance and perspective.