Essay writing is a must for Malaysian secondary schools’ students. There are few types of essays that the students need to master. Guided writing, summary writing and continuous writing are the three main types of essay that are tested in examination. These three different types of writing test students’ ability to understand and use correct grammar, to apply language skills for interpersonal purposes, to apply language skills for informational purposes, and to apply language skills for aesthetic purposes (Malaysian Examinations Syndicate, 2004).
Get Help With Your Essay
If you need assistance with writing your essay, our professional essay writing service is here to help!
Students have to excel in these three types of writing to be able to apply it in real life situation. Students are assessed on their writing skills in school tests as well as in the national standardized test. Different methods of assessment are used in evaluating students’ essays. At different levels and tests, there will be at least a minor difference in the method of assessing. In Malaysia secondary school, teachers are not given any specific guides to assess students’ written works. They assess according to their own methods and experiences.
Many methods can be used for assessing students’ essays. Every method has its own specific guidance and teachers can use them based on their needs and students performance. For example, holistic scoring method is used to assess students’ essays in a generalized way that is to look at their works as a whole piece without analyzing it in details. Any method is analytic scoring method where teachers look at students’ performance in details. Both methods can be applied by teachers in classroom- based assessment.
In this study, only two methods of essay scoring will be discussed which are mentioned above. Holistic and analytic scoring methods are very different and unique in their own way. Therefore, it is important that teachers know how to use both of them in assessing students’ essays. The subjects that will be chosen for this study are English teachers in secondary schools in Malaysia. They will be investigated on the ways they use to assess their students’ essay. A few students from their class will also be chosen to give feedbacks about their teachers’ assessment.
1.2 Statement of Problem
Students’ essays are marked without a standard scale in Malaysian secondary schools. Teachers score their students based on what they think is good for them. They use mostly holistic scoring method which might not be fair to students (Bloom, Daiker & White). The ministry of education might be responsible for what had happened. Teachers mark students’ essays according to what the ministry suggests.
Holistic scoring method has long been implemented in schools. Is it the best method for teachers and students? Some problems have occurred with this method because it is not specific enough. It might also create biasness in the case where students are close to their teachers. This method views the essay as a whole without further analysis on each component. While analytic scoring method views the essay in a more specific way, it provides details for students to be able to know what is lacking in their essays. Analytic scoring method uses a rubric as a standard scale to mark essays.
Analytic scoring method is regarded as requiring more work and is expensive while holistic scoring method is considered cheaper and more efficient. Holistic scoring method provides less information to students than analytic soring method. Each of the methods has their own advantages but there should be one method that is more suitable for essay scoring.
1.3 Purpose of the Study
This study is aimed at investigating the use of holistic scoring method and analytic rubrics in essay scoring among teachers of Malaysian secondary school. This study is also aimed at discovering the relationship between the two methods. Another goal of this study is to find out how analytic scoring method can be applied in Malaysian secondary schools.
1.4 Significance of the Study
Methods of essay scoring have been discussed by many experts all around the globe. It is important in a sense that it reflects students’ ability in writing essay. The wrong method used to mark their essays will cause their performance to drop. How students write essays, sometimes depend on how the teachers mark their essay. There is a vital connection between the two.
Basically, Malaysian secondary schools’ teachers mark students’ essay according to what they believe is correct and suitable for their students and the method they use can be considered as holistic scoring method. It is very important that they know a method in specific so that they can mark students’ essay in details.
With this study, it is hope that teachers will realized that analytic scoring method is a more suitable method to be used in essay scoring as it will enhance students’ writing skills precisely and increase the quality of their essays.
1.5 Research Questions
The research questions for this study are as follows:
How is holistic scoring method used in marking students’ essays?
How are analytic scoring rubrics used in marking students’ essays?
Why is the holistic scoring method mostly used in school?
Why is the analytic scoring rubric rarely used for marking essays in school?
How far is it possible to integrate analytic scoring rubric in the curriculum of Malaysian secondary school?
CHAPTER 2
LITERATURE REVIEW
Introduction
According to Normah Othman (2006), there is no specific scoring strategy implemented specially for a classroom-based assessment of direct writing in Malaysian secondary schools. As it is now, ESL teachers currently use a scoring method adopted from the Malaysian Examinations Syndicate. Therefore, teachers can choose to use analytic or holistic scoring. But most of the teachers use holistic scoring in Malaysian Secondary context. Below are the researches done at different countries on analytic and holistic scoring method.
According to Alharby (2006), studies addressing issues related to performance assessment seem to be taking two relatively different directions. One if it is to search for the best scoring method that maximizes reliability and validity of assessment. This is usually accomplished by comparing two of the most popular scoring methods used with performance assessment, the holistic scoring method vs. the analytic scoring method. According to him again, the holistic scoring method is usually used with large scale assessment because it is less time consuming than the analytic method; the latter often producing a very large number of responses that need to be evaluated in a short period of time. The analytic scoring method, on the other hand, is usually used with classroom assessments because it provides more detailed feedback on students’ learning progress for teachers, students, and parents than does the holistic scoring method. In this research paper, the two essay scoring methods will be focused on in Malaysian secondary school context.
Moon and Callahan (2001) reported that performance assessments have become more popular within classroom assessments for the last two decades. Methods used in performance assessment are analytic and holistic scoring methods. This two types of scoring methods have been concerned by many researches regarding issues such as its reliability (Parkers, Zimmaro, Zappe, & Suen, 1998), validity (Kane, Crooks & Cohen, 1999), cost (Picus, 1994), and practicality (Baker & Aschbacher, 1992). In this research paper, all the mentioned issues will only be dealt as general factors that affect teachers of English in Malaysian secondary context in choosing essay scoring method.
Waltman, Kahn & Koency (1998) found that the way raters score individual responses affect reliability as well as the validity of an assessment. The two most common ways to score individual responses such as essays written by students are the holistic scoring method and the analytic scoring method. According to Clauser (2000), analytic scoring method is recommended for simple tasks such as tasks to evaluate student’s ability in spelling and grammar usage. On the other hand, Clauser recommends the holistic scoring method when the tasks are more complicated such as evaluating the adequacy of a haiku. According to Charney (1984), the holistic scoring method tends to be favored by large scale assessment with numerous examinees because it is less time consuming. In contrast, Vacc (1989) claimed that the analytic scoring method tends to be favored by classroom assessment because of the detailed feedback it offers to teachers, students, and their parents in terms of how students performed on the assessment.
Holistic Scoring Method
The holistic scoring method was first introduced in 1960s and was known as the general impression scoring method (Hunter, 1996). Similarly, Ojeda (2004) stated that the holistic scoring approach is an approach where graders read to evaluate an essay as a whole-without focusing on or overemphasizing any one particular part or aspect. It started to become popular in the middle of the 70s (Boring, 2002). Huot (1990) pointed out that the holistic scoring method is a technique in which raters are asked to evaluate individual responses as a whole by their first impression without going to specific possible subdomains of the trait being tested. Klein et al. (1998, p.122) stated that “this approach [holistic scoring method] is usually most appropriate when the whole is greater than the sum of the parts, that is when scores need to be sensitive to general features of answer quality, such as organization, style, and persuasiveness.” In recent year, the holistic scoring method has become even more popular, especially with large scale assessments. In Malaysian secondary schools, this method is widely used.
Besides that, many researchers have also tried to define holistic scoring approach. White (1984, 1985) defined holistic scoring as a global approach to texts, reflecting the idea that a written text is a single entity. White (1984, 1985) maintained that holistic scoring is preferable because it more closely simulates the impression a reader the meaning of the text as a whole rather than focus on individual features of the writing, such as grammar. White (1985), Weigle (2002), and Hyland (2002) describe holistic scoring as involving carefully selected and experienced raters who use their skilled impressions to give a paper a single, integrated score that is supposed to represent the writer’s overall writing ability. Powills, Bowers, and Conlan (1979) explained that this holistic approach to scoring essays is an attempt to evaluate wholes rather than parts so that performance on one facet does not determine the judgment of performance on the whole.
Different researchers have identified different reasons for the increase of the holistic scoring method’s popularity. Some think that it is because the holistic scoring method is the most direct assessment technique (Cooper, 1997); while others think that the reason is that the holistic scoring method is the most economical, flexible and applicable assessment (Huot, 1990; Veal & Hudson, 1983). As for White (2004), he pointed out that despite disagreements that arise about the scorings of some papers; this has become the standard way of scoring gatekeeping writing tests in large-scale writing assessments. Findings by Coffman (1968), Freedman (1979), and Breland and Jones (1984) supported that holistic scoring helps focus raters on meaning, showing that holistic raters of essay writing ability seem to attend more to content features than to mechanics and sentence structure.
One potential advantage of holistic scoring methods in the assessment of second language writing ability is that they are more practical. Godshalk, Swineford, and Coffman, (1966), Alloway (1978), and Powills, Bowers, and Conlan (1979) all found holistic scoring methods to be more efficient than analytic scoring methods, particularly for large-scale exams, noting the advantages of such an approach in saving time and money. Furthermore, Smith, Winters-Edys, Quellmalz, and Baker (1980) estimated that holistic scoring takes approximately two-thirds the time of analytic scoring.
Although holistic scoring method is widely accepted as the most appropriate tool for this purpose, many writing practitioners and administrators reportedly “view the holistic technique with reservation” (Enginarlar, 1991, p. 39). Hamp-Lyons (1996, p. 234) objects to the single point score resulting from holistic grading, which she claims “obscures a pattern of consistent overemphasis or underemphasis on basic language control.” Another claim is that a higher share of midlevel papers seem to end up with split scores-scores 2 points apart that must be resolved by a third grader-so holistic scoring may be less effective for evaluating midlevel performances than an analytic method might be (Elbow, 1996b).
According to East (2009), the strength of holistic scoring lies in its practicality, which in large-scale testing contexts is an important consideration. Besides that, Vaughan (1991) argues that “researchers looking at holistic assessment have often assumed that given a scale that describes the characteristics of an essay at each level, trained raters will assess the essays in the same way every time” (p.112).
Despite all the strength of holistic scoring, Prior research has established that there are some problems with holistic scoring method. Charney (1984) speculates that scores in holistic readings are based on superficial features, such as handwriting, essay length, word choice and spelling errors. This is in accord with Stewart and Grobe (1979), who concluded that raters were primarily influenced by essay length and “freedom from simple mechanical errors” (p. 214).
Diederich, French, and Carlton (1961) found that judgments made by essay raters under holistic scoring conditions were unreliable. They recommended that considerable effort should be done to establish and maintain reliable ratings. Diederich (1974) claimed that until holistic scoring procedures were refined, the score an essay received could depend more on who the rater was than on any qualities of the text itself. Charney (1984), Gere (1980), and Odell and Cooper (1980) also questioned the premises on which certain conclusions about the reliability of holistic scoring have been based. Charney observed that the way holistic raters read texts may create an unnatural reading environment in which scores might only reflect agreement on salient but superficial features of writing, such as the quality of the handwriting or the presence of spelling errors.
According to Hamp-Lyons (1995), English as Second Language writers have a special need for scoring procedures that go beyond just the assign of a single score because one score does not allow raters to distinguish between various aspects of writing, such as language control, range of vocabulary, or organizational control. Hamp-Lyons also pointed out that it is problematic for second language (L2) writers since different aspects of writing ability seem to develop at different rates for different writers. Some writers may be strong in expressing content and organization, but limited in grammatical accuracy; others may have excellent language control at the sentence level, but be unable to organize their writing. All the aspects mentioned by Hamp-Lyons should be taken into consideration when teachers of English in Malaysian secondary schools score an essay.
Additionally, holistic scores are not easy to interpret because raters do not necessarily use the same criteria to arrive at the same scores. One rater may assign a score to a writing sample based on content and organization, while another rater assigns the same score on the basis of language control (grammar) and vocabulary. Charney (1984) noted that time constraints affect raters’ depth of processing. When spending only two minutes on a paper, a rater may assign a score that “only reflects agreement on salient but superficial features of writing, such as the quality of the handwriting or the presence of spelling errors” (p. 78).
In another study, Arthur (1979) found that the holistic rankings of second language learner’s (L2) essays written by lower-intermediate adult learners were related to frequency of spelling and grammar errors, and that the best predictor of rating was the length of composition. In addition, Raforth and Rubin (1984) found that college instructors’ perceptions of composition quality were most influenced by mechanics. A single holistic score assigned to a writing sample may thus represent different aspects of writing ability for different raters.
In holistic rating of L2 writing samples, a significantly larger amount of inconsistent textual information must be processed in a limited amount of time, which may affect the ratings. Several studies have suggested that a quick holistic read of an L2 writing sample may not allow the rater the time necessary to accurately assess the writing performance. Huot (1993) noted that when the rater is required to read a text so quickly for a limited set of criteria outlined in a scoring rubric, the need for agreement may hinder a rich, personal response to the text, thus affecting the consistency of the scores. Hamp- Lyons (2003) has cautioned that it is difficult to accurately capture the criteria used by raters during the quick read of a holistic scoring.
Apart from that, Grobe (1981) concluded from an analysis of essays and correlation with holistically obtained scores that teachers see ‘good writing’ as being “closely associated with vocabulary diversity” (p. 85). Sakyi (2001) supports Grobe’s view that raters’ decisions on awarding scores are significantly influenced by vocabulary diversity and the extent of grammatical error, but also by the range and sophistication of syntax and the style or format of the writing. Sakyi also notes from his own research that “for raters who made a conscious effort to follow the scoring guide, the restrictions imposed on them to assign a single score at the end caused them to depend mostly on only one or two particular features to distinguish between different levels of ability” (p. 129).
Analytic Scoring Method
The analytic scoring method was introduced at the same time as the holistic scoring method (Boring, 2002). Hunter (1996) explained that for the analytic scoring method, the domain being tested is subdivided to smaller subdomains. Each of these subdomains is then evaluated by itself. The sum of these subdomains would, by default, reproduce the whole. Proponents of analytic scoring methods (Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey, 1981; Hamp-Lyons, 1990; Huot, 1993; Weigle, 1999) have found several advantages to this method of scoring.
One of the advantages is with multiple scores; analytic scoring schemes provide more information about a test taker’s performance than holistic scoring schemes and permit a profile of the areas of writing ability. Analytic scoring methods can provide specific feedback, which is needed in measuring achievement, predicting future success, and accurately placing students (Shaw, 2002; Bacha 2001), as well as providing useful diagnostic feedback on writing performance (Jacobs, et al., 1981; Hamp-Lyons, 1991; Cohen, 1994). In the assessment of L2 writing ability, analytic scales are very useful if a student’s writing ability is developing at differing rates for situation such as when vocabulary is good, but organizational control is lacking.
Hamp-Lyons (1991), Huot (1996) and Weigle (2002) have suggested that analytic-type scales are generally more reliable than holistic-type scales since each candidate gets more than one score, and multiple scores contribute to reliability. Regarding intra-rater reliability, some research has suggested that analytic scoring is more useful than holistic in training inexperienced raters. It is argued that the raters can more easily interpret and apply the scoring criteria in separate domains than the criteria as described in one holistic rubric (Francis, 1977, and Adams, 1981, both cited in Weir (1990); Hartog, Rhodes, & Burt, 1936; Cast, 1939; Bauer, 1981; and Weigle, 1998). Cumming (1990) suggested that an analytical scale tends to reflect what raters do when rating samples of writing or other language performance that is consider the individual features of writing one by one.
According to Vacc (1989), the analytic technique is preferred over the holistic scoring method when detailed feedback is needed, especially with classroom assessments. Another reason to use analytic scoring method is that it has been argued that training raters to use analytic scoring rubrics is relatively easier than training them to use holistic scoring rubrics (Cohen, 1994; McNamara, 1996). According to Boring (2002), one advantage of the analytic scoring method over the holistic scoring method is that with the analytic technique, it is possible to give various weights to different subdomains based on their theoretical importance. In addition, Kroll (1990) described that it can help meet the need that many ESL learners who fail writing tests have, for feedback on both the deficiencies and the qualities in their writing (Kroll, 1990).
There are, however, a number of disadvantages to using an analytic scoring scheme for the assessment of direct writing samples. The primary concern regarding analytic scoring is practicality. As Weigle (2002) pointed out, it generally takes longer to score a writing sample using an analytic rubric since the rater must make multiple decisions for each writing sample and thus must read the sample multiple times. Some critics doubt whether writing ability can be measured by a composite score based on different aspects of writing. White (1985) argued that writing is more than the sum of its parts, and that the use of an analytic scoring scheme may in effect divert attention from overall essay effect.
Weigle (1999) noted information is lost when raters revise their scores on the different scales once they consider the single composite score. Raters may also tend to rate holistically rather than analytically so that the rater ends up assigning a writing sample the same score in every domain (Cohen, 1994).
An additional problem with analytic scoring and rater bias surfaced in Cumming’s (1990) descriptive analysis of raters’ comments. Cumming (1990) also found that both novice and expert raters tended to analytically evaluate ESL compositions on both language control and writing ability, implicitly attributing separate values to each of these domains.
Hamp-Lyons (1989) noted another potential problem in the reliability of analytic scoring schemes in that an analytic scoring method may favor essays where performance criteria are easily and quickly extracted from the writing sample. While an analytic scoring method offers the potential of additional information regarding the student’s L2 writing ability, there are potential problems of reliability and validity derived from the scoring procedures.
Conclusion
All the researches had shown that the method used by teachers in assessing students’ essay will definitely have an impact on students writing skills and ability. It is very vital to use the correct method so that students will improve their performance. From all the researches also, it can be concluded that most of them focus on the reliability, validity and practicality of the two scoring methods from many perspective. There is very few or none research is done on the teachers’ point of view on the two methods. In this research paper, the teachers’ opinions will be concern as much as possible.
2.2 Definition of terms
Holistic scoring method is a method which teachers use to evaluate a piece of students’ writing for its overall quality. It is a method used in essay scoring whereby a single mark is awarded to an essay from a range of marks categorized in different performance bands in which each band is characterized by descriptors based on the total impression of the essay as a whole. It is widely used in Malaysian schools to mark essays in school-based and public examinations.
Analytic scoring method is a method that teachers use to assess students’ written works more specifically and in detail. It is usually based on a well structured rubric where all the components that need to be assessed are stated out clearly and in detail. This method views an essay in as technical a manner as possible. It is not a common method used in Malaysian secondary schools.
CHAPTER 3
METHODOLOGY
3.1 Introduction
The first purpose of this study is to investigate the use of holistic scoring method and analytic rubrics in essay scoring among teachers of Malaysian secondary schools. This study is also aimed at discovering the relationship between the two methods. Another goal of this study is to find out how analytic scoring method can be applied in Malaysian secondary school.
As for the research questions in this study, the first one is how holistic scoring method is used in marking students’ essays. The second question is how analytic scoring rubrics are used in marking students’ essays. The third question is why holistic scoring method is mostly used in school. The fourth question is why analytic scoring rubric is rarely used in marking school essays. The last question is how possible is it for analytic scoring rubric to be integrated in the curriculum of Malaysian secondary school.
To fulfill the purpose and research questions for this study, only one method is used. Fifty survey questionnaires were distributed.
3.2 Population and sampling
The population chosen for this study was teachers of English from Malaysian secondary schools. The population was selected randomly. There was no distinction between gender and races. Fifty teachers from different schools were given the questionnaire. Six schools were chosen to answer the questionnaires. All the teachers chosen to answer questionnaire were teachers of English. All the schools chosen are selected randomly to ensure that there is no bias.
3.3 Instrumentation
The instrument used was a questionnaire consisting of 5 sections. The survey is constructed in 5 sections to answer all the research questions. A copy of the survey is attached to appendix A.
3.4 Procedure and time frame
This research took about 6 weeks to complete. It started during the 6th week and ended in the 12th week of the semester. During this period of time, the researcher went to six Malaysian secondary schools to give out the questionnaire.
First of all, the researcher identified the schools that are suitable for the study. Schools that have at least 8 teachers of English were selected.
Secondly, the researcher went to the schools and met with the principal of each school. The researcher asked permission from the principal to give out the questionnaires. Almost all the schools asked the researcher to leave the questionnaires there for collection the next week. The researcher left the questionnaires there according to the number of teachers of English in each school.
Thirdly, the researcher went back to each school the next week and collected all the completed questionnaires. Some questionnaires were not collected mainly because teachers were busy with examinations. After collecting all the questionnaires, data were analyzed.
3.5 Analysis plan
All the data collected from the survey will be analyzed according to research questions. The first and second sections of the questionnaire are to answer the first two research questions on analytic and holistic scoring. These two sections consist of characteristics on both analytic and holistic scoring. The answers respondents have chosen will reveal their scoring method accurately. Item A1 to A3 will be on analytic marking method, while A4 to A6 will be on holistic marking method. The following graph will present the result for each item.
Figure 3.1 . The outline of the presentation of result for section A of the questionnaire
Where:
A1
I use analytic marking method (mark according to specific criterion)
A2
I concentrate on one criterion at a time and read the response specifically for that criterion.
A3
I read the response and mark errors in spelling, sentence structure and coherence
A4
I use holistic marking method (mark essay as a whole)
A5
I read the response and give a mark as a whole
A6
After the first reading I assign one band score to the response and confirm this after subsequent readings.
Another graph will be for section B. The following graph is to summarize the percentage for section B.
Figure 3.2. The outline of the presentation of result for section B of the questionnaire
Where:
B1
Is the marking scheme holistic guide?
B2
Is the marking scheme analytic guide?
B3
Do you find the marking scheme easy to interpret?
B4
Do you find the marking scheme easy to apply?
B5
Do you feel the assessment criteria in the marking scheme provide an accurate and fair assessment?
B6
Do you think it is necessary to change the current marking scheme?
The third section answers the third research question on holistic scoring. This section is to explore the reasons of teachers who use holistic scoring. The fourth section answered the fourth research question on analytic scoring. This section is to explore the reasons of teachers who use analytic scoring. 7 reasons presented in the questionnaires were according to literature review. This is to find out why teachers like to use holistic or analytic scoring. Two graphs as follow will be used to summarize the result.
Figure 3.3. The outline of the presentation of result for section C of the questionnaire
Where:
C1
I like to use it.
C2
It is more convenient.
C3
It costs less.
C4
It saves time.
C5
It is reliable.
C6
It provides good feedback for students.
C7
It is fair.
Figure 3.4. The outline of the presentation of result for section D of the questionnaire
Where:
D1
I like to use it.
D2
It is more convenient.
D3
It costs less.
D4
It saves time.
D5
It is reliable.
D6
It provides good feedback for students.
D7
It is fair.
The last section is to examine the possibility of using analytic scoring in Malaysian secondary context. This answered the last research question. This section is on curriculum of Malaysian secondary school and the use of analytic scoring. If teachers are more positive towards using analytic scoring, it suggests that it is possible to use. The graph used to analyze the result is as follows.
Figure 3.5. The outline of the presentation of result for section E of the questionnaire
Where:
E1
It is possible to mark students’ essays based on specific features and criterion
in school
E2
I think that teachers are willing to use analytic scoring rubric in school
E3
Analytic scoring rubric is suita
Cite This Work
To export a reference to this article please select a referencing style below:
