UKCME logo
UK Centre for Materials Education
Working with you to enhance the student experience
People working at tables, water molecule magnified, groups of people working, split over three small windows.
Home | Important Themes | Resources | Events and Workshops | About the Centre | Search

by Ivan Moore

What is Evaluation?

The term evaluation is used in educational literature in different ways. It is often thought of as a systemic process. For example, it is common to evaluate the effectiveness of a system or outcomes of a project - usually at the end. This form of evaluation has limited benefits for the formative development of an improved project or system, unless the outcomes of the evaluation are fed back in time to make adjustments to the system. For the purposes of these guidelines, evaluation is taken to mean the collection, by an individual teacher, of feedback information so as to inform their developing practice in supporting the students in attaining the desired learning outcomes.

Why Should We Evaluate?

As material scientists and engineers, we all recognise the role of feedback in any design or control system. As professional teachers, we should recognise the need to continually evaluate our approach to supporting students in their learning, so as to improve it. Even if we believe that we are providing an excellent learning environment for our students, we know that the nature of students is constantly changing, so what may be appropriate today may not be appropriate in a year's time.

Fig. 1 shows an adaptation of a model of evaluation first provided by Ramsden (1996). We begin our teaching with some form of theory about what constitutes good teaching. Often this is based on our own experience as a learner. Occasionally, it is based, in part, on scholarship or formal study. This theory relates to different contexts of teaching, such as lectures, labs, seminars etc. or contexts such as undergraduate programmes, postgraduate programmes or professional programmes. This, in turn, influences our teaching practice, which is designed to support the students in achieving some explicit outcomes. Where these outcomes are not comprehensively achieved by all students, then feedback allows us to modify our inputs (theory, contexts and practice) in order to achieve closer alignment with the outcomes. Evaluation is the process of identifying the difference between the desired and achieved outcomes and of determining what changes might usefully improve the outcomes.

Developing teaching practice
Figure 1 - Developing Teaching Practice

What Outcomes are We Trying to Evaluate?

There are two main ways of considering outcomes. The first relates to the subject benchmark statements or the Engineering Output Standard. The benchmark statements are widely available and vary from one engineering discipline to another. However, the output standard is rather less well known, but is generic and is outlined below:

Put simply, the EPC sees engineering graduates as having:

Ability to exercise Key Skills in the completion of engineering-related tasks at a level implied by the benchmarks subject.

They then see the engineering process in six stages, as below:

  • Ability to transform Existing Systems into Conceptual Models.
  • Ability to transform Conceptual Models into Determinable Models
  • Ability to use Determinable Models to obtain system Specifications in terms of parametric values
  • Ability to select optimum Specifications and create Physical Models
  • Ability to apply the results from Physical Models to create Real Target Systems
  • Ability to critically review Real Target Systems and personal performance

(These form the 7 Ability to [A2] statements. Each one is expanded in the EPC booklet on the Output Standard.)

These forms of output are often reflected in programme and unit specifications and clearly identify the intended outcomes of a course of study. (Of course, assessing whether students have developed these abilities is far from simple.) All of these kinds of statements attempt to identify the knowledge, understanding and skills that students will have on graduation. An alternative (or additional) way of considering outcomes is in terms of the students' learning skills. That is, their developing capacity to engage in deep learning and to learn independently. This includes taking responsibility for their own learning, developing inform- ation and study skills, and maintaining motivation to learn.

An independent learner might be identified as someone who can identify:

  • their learning goals (what they need to learn)
  • their learning processes (how they will learn it)
  • how they will evaluate and use their learning

In addition, it could be argued that:

  • they have well-founded conceptions of learning
  • they have a range of learning approaches and skills
  • they can organize their learning
  • they have good information processing skills
  • they are well motivated to learn

What should be the principles of evaluation?

  1. evaluation should be an integral part of our teaching practice
  2. evaluation should be an ongoing process, so that we learn from systematic reflection
  3. evaluation should be participatory
  4. evaluation should enable us to make appropriate modifications along the way
  5. evaluation should enable us to make judgements on specific teaching sessions, but also to draw out wider implications

What Are the Standard Techniques for Evaluation?

Most institutions have their own evaluation questionnaires. These have often been designed to satisfy external quality agencies and to audit quality but not necessarily to help to enhance it. Ramsden has developed a Course Experience Questionnaire, based on research into student learning, which focuses on identifying those approaches which foster deep learning in students. The appendix shows a 25-question version of this questionnaire which comprises 5 sub-scales:

  • Good teaching
  • Clear goals and standards
  • Appropriate assessment
  • Appropriate workload
  • Generic skills

This questionnaire attempts to identify the attitudes and intentions of teachers in a way that allows them to improve the way they encourage deep learning approaches in the students.

What do the scales mean?

The Good Teaching Scale.
Questions 3, 7, 15, 17, 18 and 20

The scale is characterised by teaching practices which include the following: providing useful and timely feedback, clear explanations, motivating students, making the course interesting, and understanding students' problems. Lower scores on this scale are associated with the perception by students that such practices occur less frequently.

The Clear Goals and Standards Scale.
Questions 1, 6, 13 and 24

Practices characteristic of this scale relate to the establishing of clear aims and objectives for a course and clear expectations of the standard of work expected from students. It is possible to employ the good teaching practices described under the Good Teaching Scale, without implementing practices characteristic of the Clear Goals and Standards Scale.

The Appropriate Assessment Scale.
Questions 8, 12, 16 and 19

This scale deals with the extent to which assessment measures higher order thinking and understanding rather than simple factual recall. This scale does not probe other important aspects of assessment practices such as the congruence of the assessment with the material actually taught, the level of difficulty and the consistency of the quality of the assessment.

The Appropriate Workload Scale.
Questions 4, 14, 21 and 23

Higher scores on this scale indicate a perception of reasonable workloads. Heavy workloads do not necessarily equate to high standards and expectations so the wording of the items probes the extent to which heavy workloads interfere with student learning. Heavy workloads tend to preclude students from engaging with and understanding the material they are learning. Instead, many students adopt surface approaches to learning as a strategy for dealing with high workloads.

The Generic Skills Scale.
Questions 2, 5, 9, 10, 11 and 22

This scale reflects the extent to which students perceive their studies to have fostered the development of the generic skills recognised by the university as being a valuable outcome of university education, in addition to discipline specific skills and knowledge.
Note: these generic skills may be determined to be key skills, employability or any such skills as may be deemed to be important in your course, department or university.

When Should We Evaluate?

The Ramsden questionnaire meets the last principle of evaluation above. That is 'evaluation should enable us to make judgements on specific teaching sessions, but also to draw out wider implications'. However, a common difficulty with this form of evaluation questionnaire is that it is often only processed at the end of a period of study, and any modifications to teaching practice will not benefit the students who completed the forms. It is often very effective (and easy) to conduct short, regular evaluation activities. Below are three possible sets of questions that could be used at the end of ANY teaching session. They all help to provide immediate feedback, both to the student and to the teacher in a manner that allows immediate action to be taken to improve the learning.

Please answer each question in 1 or 2 sentences:

  1. What was the most useful or meaningful thing you learned during this session?
  2. What question(s) remain uppermost in your mind as we end this session?
  3. What was the 'muddiest' point in this session?
    (In other words, what was least clear to you?)
  1. What would you like me to stop doing?
  2. What would you like me to start doing?
  3. What do you want me to continue to do?
  1. What are 1 or 2 specific things I do that help you learn in this course?
  2. What are 1 or 2 specific things I do that hinder or interfere with your learning?
  3. Please give me 1 or 2 specific, practical suggestions on ways to help you improve your learning in this course.

The first set of questions relates to teaching (or learning) sessions, such as a lecture or practical class, so it might be used at the end of virtually every session. The third set makes reference to learning throughout a course, so it might be used at the mid point of a course of teaching. The second set might be used on, say, a monthly basis. Of course, useful feedback can be gained from processing these questions at any stage in a course of teaching.

How Might We Use These Questions?

Perhaps the most immediate way to use these sets of questions is in a 'one-minute questionnaire'. You could display the questions on a screen and invite students to produce short responses on paper and drop them into a box as they leave the room. However, the students might provide more thoughtful and meaningful evaluations if they are given a few minutes to consider the questions. You could try paired discussions around the questions, for example, with written or poster feedback, or invite them to submit a short piece of reflective writing. These techniques take a little longer, but often provide more considered responses.

Designing Your Own Evaluation

The model presented in the introduction to this guide discussed different contexts, such as the lecture, the lab etc. It is possible to take such context, and devise four or five questions designed to evaluate in what ways that form of learning activity was helping or hindering the students to achieve the desired learning outcomes. For example, in laboratory teaching, we might construct five simple statements, which the students would score from 1 to 4 (1= do not agree, 4= strongly agree). A four-point scale avoids the temptation for students to opt for a neutral response. If you prefer to allow them that option, then make it a five-point scale.

  • The laboratory brought the lecture material to life.
  • The laboratory allowed me to see key concepts working in practice and so understand how theories are applied in real solutions.
  • The laboratory stimulated my interest in the subject.
  • The laboratory helped my ability to work as a member of a team.
  • The laboratory gave me confidence in researching an open-ended engineering problem.

This approach can be applied to any/all contexts, such as lectures, problems classes or group projects. A typical design is shown below.

  1. The laboratory brought the lecture material to life.
  2. The laboratory allowed me to see key concepts working in practice and so understand how theories are applied in real solutions.
  3. The laboratory stimulated my interest in the subject.
  4. The laboratory helped me to work as a member of a team.
  5. The laboratory gave me confidence in researching an open-ended engineering problem.
  6. In the mini group project we worked well as a team, with all members contributing effectively.
  7. The mini group project helped me to use presentation skills effectively.
  8. Investigative skills were used and new skills learned.
  9. I learnt more in this context about the subject matter than I would if I had been in a lecture.
  10. The problem class enhanced my understanding of a 'Concept X'.
  11. The problem class has helped prepare me for assessment in 'Concept X'.
  12. I felt uncomfortable about contributing to the problem class.
  13. The problem class helped me make connections between different concepts and relate them to each other.
  14. I could explain the concepts of this lecture to another student.
  15. I found it difficult to concentrate throughout the lecture.

These evaluation statements were actually designed by participants in a workshop on evaluation. They clearly identify the kinds of learning activities or contexts (in this case laboratories, group projects, problems class and lectures) and the aims of the teaching staff in designing these activities. The next stage is to randomise the questions, and to reword some of them in a negative sense - as in question 12. This avoids the tendency for a student to simply go down the score list and award the same mark to each question, based on an overall view of the course.

Likert scales are useful, but limited, in providing evaluative information. It would be a simple and effective development to invite the students to provide examples to support their evaluations or to give reasons why they made each response, either for each statement, each section or the questionnaire as a whole.

In Conclusion

As material scientists and engineers, we recognise the importance of feedback in maintaining a stable situation or achieving a goal. If the goal is to improve our practice as teachers, then evaluation is an important component of our professional practice as teachers. There are standard questionnaires that can be used and it is relatively easy to design your own questionnaires. However, more reflective, meaningful feedback can be derived from group discussion. Each group can consider all the questions, or you can give each group only one question to consider. Short questionnaires, of three or so questions can be processed frequently and quickly. These will give very useful information to teaching staff which will allow them consistently to adapt and improve their teaching practice. Such improvements can only be to the benefit of the students and of their learning.

Appendix 1

The Course Experience Questionnaire

  1. It was always easy to know the standard of work expected.
  2. The course developed my problem-solving skills.
  3. The teaching staff of this course motivated me to do my best work.
  4. *The workload was too heavy.
  5. The course sharpened my analytic skills.
  6. I usually had a clear idea of where I was going and what was expected of me in this course.
  7. The staff put a lot of time into commenting on my work.
  8. *To do well in this course all you really needed was a good memory.
  9. The course helped me develop my ability to work as a team member.
  10. As a result of my course, I feel confident about tackling unfamiliar problems.
  11. The course improved my skills in written communication.
  12. *The staff seemed more interested in testing what I had memorised than what I had understood.
  13. *It was often hard to discover what was expected of me in this course.
  14. I was generally given enough time to understand the things I had to learn.
  15. The staff made a real effort to understand difficulties I might be having with my work.
  16. The assessment methods employed in this course required an in-depth understanding of the course content.
  17. The teaching staff normally gave me helpful feedback on how I was going.
  18. My lecturers were extremely good at explaining things.
  19. *Too many staff asked me questions just about facts.
  20. The teaching staff worked hard to make their subjects interesting.
  21. *There was a lot of pressure on me to do well in this course.
  22. My course helped me to develop the ability to plan my own work.
  23. *The sheer volume of work to be got through in this course meant it couldn't all be thoroughly comprehended.
  24. The staff made it clear right from the start what they expected from students.
  25. Overall, I was satisfied with the quality of this course.

* Negatively scored

Bibliography

Ramsden P. Learning to Teach in Higher Education, Routledge, London, 1996

The Engineering Professors' Council. The EPC Engineering Graduate Output Standard, EPC occasional paper number 10. 2000