- Database of Resources
- Important Themes
- Guides for Lecturers
- Events and Workshops
- Teaching Development Projects
- Materials Awareness Projects
The term evaluation is used in educational literature in different ways. It is often thought of as a systemic process. For example, it is common to evaluate the effectiveness of a system or outcomes of a project - usually at the end. This form of evaluation has limited benefits for the formative development of an improved project or system, unless the outcomes of the evaluation are fed back in time to make adjustments to the system. For the purposes of these guidelines, evaluation is taken to mean the collection, by an individual teacher, of feedback information so as to inform their developing practice in supporting the students in attaining the desired learning outcomes.
As material scientists and engineers, we all recognise the role of feedback in any design or control system. As professional teachers, we should recognise the need to continually evaluate our approach to supporting students in their learning, so as to improve it. Even if we believe that we are providing an excellent learning environment for our students, we know that the nature of students is constantly changing, so what may be appropriate today may not be appropriate in a year's time.
Fig. 1 shows an adaptation of a model of evaluation first provided by Ramsden (1996). We begin our teaching with some form of theory about what constitutes good teaching. Often this is based on our own experience as a learner. Occasionally, it is based, in part, on scholarship or formal study. This theory relates to different contexts of teaching, such as lectures, labs, seminars etc. or contexts such as undergraduate programmes, postgraduate programmes or professional programmes. This, in turn, influences our teaching practice, which is designed to support the students in achieving some explicit outcomes. Where these outcomes are not comprehensively achieved by all students, then feedback allows us to modify our inputs (theory, contexts and practice) in order to achieve closer alignment with the outcomes. Evaluation is the process of identifying the difference between the desired and achieved outcomes and of determining what changes might usefully improve the outcomes.
Figure 1 - Developing Teaching Practice
There are two main ways of considering outcomes. The first relates to the subject benchmark statements or the Engineering Output Standard. The benchmark statements are widely available and vary from one engineering discipline to another. However, the output standard is rather less well known, but is generic and is outlined below:
Put simply, the EPC sees engineering graduates as having:
|Ability to exercise Key Skills in the completion of engineering-related tasks at a level implied by the benchmarks subject.|
They then see the engineering process in six stages, as below:
(These form the 7 Ability to [A2] statements. Each one is expanded in the EPC booklet on the Output Standard.)
These forms of output are often reflected in programme and unit specifications and clearly identify the intended outcomes of a course of study. (Of course, assessing whether students have developed these abilities is far from simple.) All of these kinds of statements attempt to identify the knowledge, understanding and skills that students will have on graduation. An alternative (or additional) way of considering outcomes is in terms of the students' learning skills. That is, their developing capacity to engage in deep learning and to learn independently. This includes taking responsibility for their own learning, developing inform- ation and study skills, and maintaining motivation to learn.
Most institutions have their own evaluation questionnaires. These have often been designed to satisfy external quality agencies and to audit quality but not necessarily to help to enhance it. Ramsden has developed a Course Experience Questionnaire, based on research into student learning, which focuses on identifying those approaches which foster deep learning in students. The appendix shows a 25-question version of this questionnaire which comprises 5 sub-scales:
This questionnaire attempts to identify the attitudes and intentions of teachers in a way that allows them to improve the way they encourage deep learning approaches in the students.
The Good Teaching Scale.
|The scale is characterised by teaching practices which include the following: providing useful and timely feedback, clear explanations, motivating students, making the course interesting, and understanding students' problems. Lower scores on this scale are associated with the perception by students that such practices occur less frequently.|
The Clear Goals and Standards Scale.
|Practices characteristic of this scale relate to the establishing of clear aims and objectives for a course and clear expectations of the standard of work expected from students. It is possible to employ the good teaching practices described under the Good Teaching Scale, without implementing practices characteristic of the Clear Goals and Standards Scale.|
The Appropriate Assessment Scale.
|This scale deals with the extent to which assessment measures higher order thinking and understanding rather than simple factual recall. This scale does not probe other important aspects of assessment practices such as the congruence of the assessment with the material actually taught, the level of difficulty and the consistency of the quality of the assessment.|
The Appropriate Workload Scale.
|Higher scores on this scale indicate a perception of reasonable workloads. Heavy workloads do not necessarily equate to high standards and expectations so the wording of the items probes the extent to which heavy workloads interfere with student learning. Heavy workloads tend to preclude students from engaging with and understanding the material they are learning. Instead, many students adopt surface approaches to learning as a strategy for dealing with high workloads.|
The Generic Skills Scale.
|This scale reflects the extent to which students perceive their
studies to have fostered the development of the generic skills recognised
by the university as being a valuable outcome of university education,
in addition to discipline specific skills and knowledge.
Note: these generic skills may be determined to be key skills, employability or any such skills as may be deemed to be important in your course, department or university.
The Ramsden questionnaire meets the last principle of evaluation above. That is 'evaluation should enable us to make judgements on specific teaching sessions, but also to draw out wider implications'. However, a common difficulty with this form of evaluation questionnaire is that it is often only processed at the end of a period of study, and any modifications to teaching practice will not benefit the students who completed the forms. It is often very effective (and easy) to conduct short, regular evaluation activities. Below are three possible sets of questions that could be used at the end of ANY teaching session. They all help to provide immediate feedback, both to the student and to the teacher in a manner that allows immediate action to be taken to improve the learning.
Please answer each question in 1 or 2 sentences:
The first set of questions relates to teaching (or learning) sessions, such as a lecture or practical class, so it might be used at the end of virtually every session. The third set makes reference to learning throughout a course, so it might be used at the mid point of a course of teaching. The second set might be used on, say, a monthly basis. Of course, useful feedback can be gained from processing these questions at any stage in a course of teaching.
Perhaps the most immediate way to use these sets of questions is in a 'one-minute questionnaire'. You could display the questions on a screen and invite students to produce short responses on paper and drop them into a box as they leave the room. However, the students might provide more thoughtful and meaningful evaluations if they are given a few minutes to consider the questions. You could try paired discussions around the questions, for example, with written or poster feedback, or invite them to submit a short piece of reflective writing. These techniques take a little longer, but often provide more considered responses.
The model presented in the introduction to this guide discussed different contexts, such as the lecture, the lab etc. It is possible to take such context, and devise four or five questions designed to evaluate in what ways that form of learning activity was helping or hindering the students to achieve the desired learning outcomes. For example, in laboratory teaching, we might construct five simple statements, which the students would score from 1 to 4 (1= do not agree, 4= strongly agree). A four-point scale avoids the temptation for students to opt for a neutral response. If you prefer to allow them that option, then make it a five-point scale.
This approach can be applied to any/all contexts, such as lectures, problems classes or group projects. A typical design is shown below.
These evaluation statements were actually designed by participants in a workshop on evaluation. They clearly identify the kinds of learning activities or contexts (in this case laboratories, group projects, problems class and lectures) and the aims of the teaching staff in designing these activities. The next stage is to randomise the questions, and to reword some of them in a negative sense - as in question 12. This avoids the tendency for a student to simply go down the score list and award the same mark to each question, based on an overall view of the course.
Likert scales are useful, but limited, in providing evaluative information. It would be a simple and effective development to invite the students to provide examples to support their evaluations or to give reasons why they made each response, either for each statement, each section or the questionnaire as a whole.
As material scientists and engineers, we recognise the importance of feedback in maintaining a stable situation or achieving a goal. If the goal is to improve our practice as teachers, then evaluation is an important component of our professional practice as teachers. There are standard questionnaires that can be used and it is relatively easy to design your own questionnaires. However, more reflective, meaningful feedback can be derived from group discussion. Each group can consider all the questions, or you can give each group only one question to consider. Short questionnaires, of three or so questions can be processed frequently and quickly. These will give very useful information to teaching staff which will allow them consistently to adapt and improve their teaching practice. Such improvements can only be to the benefit of the students and of their learning.
* Negatively scored
Ramsden P. Learning to Teach in Higher Education, Routledge, London, 1996
The Engineering Professors' Council. The EPC Engineering Graduate Output Standard, EPC occasional paper number 10. 2000