APPROACHES TO EVALUATION
A. Background
Planning and
teaching a language course needs approaches to evaluation which it could be
planned and developed by language programs and language teaching materials. This
overall and interlinked system of elements (i.e., needs, goals, teachers,
learners, syllabuses, materials, and teaching) is known as the second language
curriculum. However, once a curriculum is in place, a number of important
questions still need to be answered. These include:
1.
Is the curriculum achieving its goals?
2.
What is happening in classrooms
and schools where it is being implemented?
3.
Are those affected by the
curriculum (e.g., teachers, administrators, students, parents, employers)
satisfied with he curriculum?
4.
Have those involved in developing
and teaching a
5.
Language course done a
satisfactory job?
6.
Does the curriculum compare
favorably with others of its kind?
Curriculum evaluation is concerned with answering
these above questions. It focuses in collecting information about different
aspects of a language program and understand how the program works, how
successfully it works, enabling different kinds of decisions to be made about
the program, such as when the program responds to learner’s need, whether
further teachers training is required for teachers working in the program, or
when students are learning sufficiently from it.
B. SCOPE
Thus, the paper will cover on the Evaluation
focuses on many different aspects of a language program (Sanders 1992,; Weir
and Roberts), such as:
1.
Curriculum design
2.
The syllabuls and program
content;
3.
Classroom processes;
4.
Materials of instruction;
5.
The teachers;
6.
Teacher training;
7.
The students;
8.
Monitoring of pupil progress;
9.
Learner motivation;
10.
The Institution;
11.
Learning environment;
12.
Staff development;
13.
Decision making.
The Scope of evaluation has changed from a concern with test results
to the need information collection and make judgments about all aspects of the
curriculum, from planning to implementation (Hewings and Dudley-Evans 1996).
The points that will be discussed in this paper are as follows :
a.
Purposes of Evaluation:
·
Formative evaluation;
·
Illuminative evaluation;
·
Summative evaluation.
b.
Issues in Program Evaluation:
·
The audience for evaluation;
·
Participants in the evaluation
process;
·
Quantitative qualitative
evaluation;
·
The importance of
documentation;
·
Implementation;
C.
Procedures used in conducting
evaluations:
·
Tests
·
Comparison of two approaches to
a course;
·
Interviews;
·
Questionnaires;
·
Teachers written evaluation;
·
Diaries and Journals;
·
Teachers’ records;
·
Student logs;
·
Case study;
·
Student evaluations;
·
Audio or video recording;
·
Observation.
d.
Discussion questions and
activities:
e.
Appendix Examples of Program Evaluations:
·
Example 1: Evaluation of a
primary English course in an EFL Country;
·
Example 2: Evaluation of
courses in a private language institute.
C. OBJECTIVES
We hope this paper can give an
insight on approaches to evaluation with Focus from test results to the need to
collection information and make judgments about all aspects of the curriculum,
from planning to implementation.
CHAPTER II
DISCUSSION
A.
CURRICULUM EVALUATION
The aim of this part of Curriculum Evaluation is concerned with
answering questions such as :
·
Is the curriculum achieving its
goals?
·
What is happening in classrooms
and schools where it is being implemented?
·
Are those affected by the
curriculum (e.g., teachers, administrators, students, parents, and employers)
satisfied with the curriculum?
·
Have those involved in
developing and teaching a language course done a satisfactory job?
·
Does the curriculum compare
favorably with others of its kind?
It focuses on collecting information about different aspects of
language program in order to understand how the program works, and how
successfully it works, enabling different kinds of decisions to be made out the
program.
Evaluation may
focus on many different aspects of a language program, such as:
·
Curriculum Design: to provide
insights about the quality of program planning and organization;
·
The syllabus and program
content: how relevant and engaging it was, how easy or difficult,
how successful tests and assessment
procedures were;
·
Classroom Processes: to provide
insights about the extent to which a program is being
implemented appropriately;
·
materials of instruction: to
provide insights about whether specific materials are aiding
student learning;
·
the teachers: how they
conducted their teaching, what their perceptions were of the program, their
perceptions of it, and how they participated in it;
·
monitoring of pupil progress:
to conduct formative (in-progress) evaluations of student
learning;
·
learner motivation: to provide
insights about the effectiveness of teachers in aiding students
to achieve goals and objectives of the
school;
·
the institution : for example,
what administrative support was provided, what resources were used, what
communication networks were employed;
·
learning environment: to provide insights about the extent to which
students are provided with a responsive environment in terms of the educational
needs;
·
staff development: to provide
insights about the extents to which the school system provides the staff
opportunities to increase their effectiveness;
·
decision making: to provide
insight about how well the school staff principles, teachers, and others – make
decisions that the result in learner benefits.
A.1. According to Weir and Roberts (1994):
ü to examine the effects of a program or a project
ü Program development:to improve two major purposes for language program evaluation,
program accountability and program
development, such as:
1. FORMATIVE EVALUATION
·
To find out what is working
well, and what is not,and what problems need to be addressed;
·
Ongoing development and
improvement of the program;
·
Some typical questions that
relate to formative evaluation (p. 288);
·
To address problems and to improve.
Example 1:
·
Situation: task-oriented communicative
methodology
·
Problem: teachers are resorting to a
teacher-dominated drill and practice
mode of teaching during the implementation.
·
Solution: provide videos to model teaching
strategies
Example 2:
·
Situation: to implement integrated skills
·
Problem: different perceptions of what
the major points in the course (after
few
weeks)
· Solution:
to held meetings to review teachers’ understanding and to clarify the
weighting
as well as to provide peer observation.
Example 3:
·
Situation: to implement Conversation
Skills
·
Problem: a number of students have
persistent and major pronunciation problems that the course is not addressing
(four weeks after the course).
·
Solution: to refocus one section of the
course to include a pronunciation component. Individual diagnostic sessions are
held with students who have the most serious pronunciation problems, and
laboratory work as well as classroom time is allotted to systematic
pronunciation work for the remainder of the course.
2.
ILLUMINATIVE EVALUATION
§ Another type of evaluation can be described as Illuminative
Evaluation. This refers to evaluation that seeks to find out how different
aspects of the program work or are being implemented.
§ Otherwise, Illuminative Evaluation seeks to provide a deeper understanding
of the processes of teaching and learning that occur in the program, without
necessarily seeking to change the course in any way result. Some questions that
might be asked within this framework are:
a.
How the students carry out
group work tasks? Do all students participate equally in them?
b.
What type of error-correction
strategies do teachers use?
c.
What kinds of decisions do
teachers employ while teaching?
d.
How do the teachers use lesson
plan when teaching?
e.
What type of teacher student
interaction patterns typically occur in classes?
f.
What reading strategies do
students use with different kinds of texts?
g.
How do the students understand
the teacher’s intentions during a lesson?
h.
Which students in a class are
most or least active?
Example 1:
·
Situation:
A teacher is teaching a course on reading skills and has developed a
course which focuses on wide variety of reading skills, such as skimming,
reading for details, surveying a text, critical reading, and vocabulary
development. The teacher is interested in finding out what the students
perceived to be the main point of the course. Students complete a short
questionnaire at different times during the course in order to describe their
perceptions of what the course is seeking to achieve.
·
Evaluated Way: conduct a short questionnaire;
=> understand students’ opinion and need.
It means the learners’ perceptions of a course may reflect what they are most
interested in or what they feel they need most help with at a particular point
in time.
Example 2:
·
Situation: A teacher is interested in learning more about teacher-student in her
own classroom. She invites a colleague to visit her class and to carry out a
series of classroom observations. The observer is given the task of nothing how
often the teacher interacts with different students in the class and the kind
of interaction that occurs.
·
Evaluated way: to invited colleague to do the classroom observation
=>
assess what happened during the course. From the data collected by the Observer
that the teacher was able to assess the extent to which the teacher or the
students control classroom interaction and got better understanding of how the
teacher used questions to “scaffold” lesson content.
Example 3:
· Situation: A teacher wants to know how students carryout
group work and whether he is sufficiently preparing students for group-work
task.
· Evaluated Way: record and review the recordings.It
means the teacher arranged to record different groups of students carrying out
a group-work task and reviewed the recordings to find out the extent to which
students participate in group discussions and kind of language they use.
=> know
what kind of roles for each member in a group and make sure students
participate
actively.
Ø Much classroom action research or
teacher inquiry : it can be regarded as a type of Illuminative Evaluation.
Ø Block (1998) : interview learners
regularly, it means to find out how they interpret what is going on in a
course.
Ø Richards and Lockhart (1994): describe a piece of classroom
action research (classroom observation,
learner journals, and interviews) The teachers found the successful
learners had identified a number of helpful learning strategies that they
applied in different ways, there are some questions about their classes as
below:
§ What learning strategies were used
by successful learners in their classes?
§ Do the learners use English
outside of the classroom?
§ Do they feel good about learning
English?
=>it
was useful to confirm and make explicit some things which we knew intuitively.
We have learned a useful strategy to use in order to more effectively
facilitate our students’ learning. The teachers collected the information on
two learners over a term, using classroom observation, learner journals and
interviews. In helping to remember things they had studied, the children gave
these examples:
- It is easy to remember when you
listen
- I do it ever and over again
- I practice with my friend and
family
- I stick sentences on my wall in
my room
- I spend lots of time going over
with my books because I like it and I learn. I would
still study it if my teacher
didn’t see it or mark it.
3.
SUMMATIVE EVALUATION
Ø Summative Evaluation is used to
determine the effectiveness of a program, its efficiency, and to some extent
with its acceptability
Ø Used “after” a program: it take
place after program has been implemented and seeks to answer questions such as
these below;
Ø Some typical questions (p. 292):
a. How effective was the course?
b. What did the students learn?
c. How well was the course received
by students and teachers?
d. Did the materials work well?
e. Were the objectives adequate or do
they need to be revised?
f.
Were the placement and achievement tests adequate?
g. Was the amount of the time spent
on each unit sufficient?
h. How appropriate were the teaching
methods?
i.
What problems were encountered during the course?
Ø There are many different measures
of a course’s effectiveness and each measure can be used for different
purposes. For example:
a.
Mastery of Objectives: “How far have the objectives been achieved?”
It is one way of measuring the effectiveness of a course.
For example: in a course on speaking
skills.
So the Objective might
be: In group discussions students will
listen to and respond to the opinions of others in their group. The extent
to which the students have mastered this objective at the end of the course can
be assessed by the teacher’s observing students during group discussions and
recording on a scale the extent to which they listen and respond to opinions.
BUT mastery of objectives does not provide a full picture of the
effectiveness of a course. It means
Objectives can be achieved despite defects in a course. Students may have
realized that the teaching or materials were poor or insufficient and so spent
a lot of extra time in private study to compensate for it.
v Performance on Tests:
a. Formal
tests, these are probably the commonest means used to measure
achievement. Such tests might be unit tests given at the end of each unit of
teaching materials, class tests or quizzes devised by teachers and administered
at various stage throughout the course, or as formal exit tests designed to
measure the extent to which the objectives have been achieved.
b.
Weir (1995): it is helpful for
teaching and learning. They can help in the making of decisions about needed
changes to a program. Such as which objectives need more attention or revision.
c. Brindley (1989): informal methods;
reports, however, that in programs he studied in Australia, teachers preferred
to rely on informal methods of ongoing assessment rather than formal exit
tests.
BUT Weir (1995) stated that summative evaluation and for the
development of progress-sensitive performance tests for use are necessarily
during course;such as:
v Measures of acceptability:
◎satisfactory achievement of the objectives and good levels of performance
on exit tests ≠ teachers and
learners’ opinion;
◎ should be
considered some factors: time-tabling; class
size, choice of materials,
or teachers’ teaching styles.
v Retention rate or
reenrollment rate : A measure of a course’s effectiveness that may be important from an
institution’s point of view is the extent to which students continue in the
course throughout its duration and the percentage of students who reenroll for
another course at the end. If there is a
significant dropout rate, is this true of other courses in the institution and
the community or is it a factor of a given course only?
v Efficiency of the course: one of the success of a course
is how straightforward the course was to develop and implement. The time spent
on planning and couse development the need for specialized materials and
teacher training, and the amount of time needed for consultations and meetings.
4.
ISSUES IN PROGRAM
EVALUATION
Weir and Roberts (1994,42) propose a broad view of evaluation
that is characterized by:
v Insider and outsider commitment
and involvement to ensure adequate evaluation;
v The “product value” of a program
or project or their components;
v A deeper professional understanding
of the processes of educational change, as well as the results of that change;
v Systematic documentation for
evaluation purposes both during implementation and the beginning and end of a
program or project’s life;
v A willingness to embrace both
quantitative methodology appropriate to the purpose of the evaluation and the
context under review.
So these principles
raise the following issues in the evaluation process.
5.
THE AUDIENCE FOR
EVALUATION
It is important to identify who the different audiences are and what kind
of information they are most interested in (Elley 1989). For example:
To develop a new textbook series for public schools funded by the
ministry of education, officers in the
ministry (who might not be specialists in language teaching), such as “money” which provided for the project
is spent and when all components of the project (student books, teacher guides,
and workbook) are available in schools by a specific date. Teachers: the
sufficient materials for all the classes on the school time table. Outside
Consultant: might be interested in the design of the materials Interaction and
language practice. Vocational training centers: school leavers’ English.
Therefore, evaluation has to satisfy all interested
parties. Questions different audiences might be interested in are:
Students:
What did I learn?
How well did I do
compare to others?
How well will I rate this course?
Teachers:
How well did I teach?
What did my students learn?
Were my students satisfied with the course?
Curriculum developers:
Is the design of the course and materials appropriate?
What aspects of the course need replacing or revising?
Do teachers and students respond favorably to the course?
Administrators:
Was the time frame of the course appropriate?
Were the management and monitoring of the course successful in
identifying and rectifying problems?
Were clients’ expectations met?
Sponsors:
Was the cost of the course justified?
Did the course deliver what was promised?
Three Audiences are identifiable
for all summative evaluation of language courses (Shaw and Dowsett, 1986):
a. Other teachers in the program, for
course design and planning purposes (the main audience);
b. Managers of the institution or
program, for the purpose of determining course offerings and placement;
c. The Curriculum support or
development unit, for the purpose of monitoring the curriculum.
The audiences need to be carefully identified and the results of
the evaluation presented in a way that is appropriate for each audience.
6.
Participants in the Evaluation
Process
Two types of participants are typically
involved in evaluation – Insiders and Outsiders. Insiders refers to teachers,
students, and anyone else closely involved in the development and
implementation of the program. Outsiders are others who are not involved in the
program and who may be asked to give an objective view of aspects of the
program. They may be Consultants, inspectors, or administrators.
Example: Formative evaluation: teachers
Summative evaluation: students
Why Insiders are important? Because they are
the direct participants for a Curriculum.
7.
Quantitative and
Qualitative Evaluation
Quantitative measurement: something can be
expressed numerically.
=>
collect information from a large number of people and analyze statistically
more fair but some limitations.
Qualitative measurement:
something can not be expressed numerically and => collect information from
classroom observation, interviews, journal and so on
more holistic and
naturalistic but hard to analyze
◎ Combine both Quantitative and Qualitative
8. The importance of Documentation
Relevant documentations:
a) Course statistics: information on why
students chose the course,
student numbers, attendance, class size,
drop outs, use of facilities such as
library or self-access center.
b) Relevant course documents:
Compilation of all relevant documentation about
the course. Such as descriptions,
publicity materials, statements of aims,
objectives, and syllabus, course
materials, teaching guides, newsletters,
newspaper articles, report of planning
meetings.
c) Course work: examples of tests, class
assignments, examples of students work.
d)
Written comments : anything that has been written about the course by external
assessors, teachers, learners,
managers.
e)
Institutional documents: anything that is available about the school or
institution, hiring policy, job
description, need analyses that have been
conducted, reports of previous
courses.
f) Course Reviews: a written account of a
course, prepared by the teacher or
teachers who taught the course.
9.
Procedures used in
Conducting Evaluations
1.
Tests
1) institutionally prepared tests
2) international tests
3) textbook tests
4) student records
- Advantage:
1)
direct measure of achievement or performance
- Disadvantages:
1) hard to make sure the tests are a direct of
teaching or are liked to other
factors=> further investigation.
2)
“reliability and validity” problems.
2.
Comparison of two approaches to a course
＊ two different
versions of the course
=> compare the effects of two or more different teaching conditions.
- Advantage:
1)
control all relevant factors and investigate strictly
- Disadvantage:
1)
teachers’ load (maintain the difference
at the same time)
3.
Interview
＊ could get
many different views of the course
＊ structured
interview is more useful
◎ Advantage:
1)
obtain more deeper (in-depth) information
◎ Disadvantages:
1)
time-consuming
2)
could not be generalized
