The Advantages of Using an Analytic Scoring Procedure in Speaking Assessment

This article describes the advantages ofusing analytic procedure in speaking assessrnent. An analytic scoring guide, as compared to the impressionistic one, has a double function: as an instrument to measure the leamer's speaking proficiency and as a diagnostic procedure for remedial teaching. Thus, it provides reliable sources of information in the form of scores of the speaking components and can be used as feedback for the teacher and leamer to identify which component needs im-

teacher does not need to make a formai speaking test with a suitable scoring guide.In this case he/she uses hisftrer feel judgment to score the learner's performance.By doing so, the teacher determines the scores on the basis of his/her general impression without necessarily making an analysis of the aspects or features of speaking ability.
This article is directed to show that using an analytic procedure in speaking assessment, in which the scoring piocedure is based on the arraiy- sis of features of speaking ability, is more advantageous than using an irn- pressionistic one (sometimes called a holistic approach) for two major rea- sons.First, an analytic scoring procedure gives the rater a clear description of the components of speaking ability being assessed.Second, it functions as an instrument to measure the learner's performance and as a diagnostic procedure for remedial teaching.It can be used as reliable information on the leamer's level of proficiency from many different perspectives.Thus, this analytic procedure is a suitable approach for classroom purposes as part of the teaching and learning process.
To show the advantages of using an analytic procedure in speaking assessment, the following discussicn covers a definiton of a speaking test, types of speaking tests, and its scoring procedures.Finally, it describes the advantages of using an analytic scoring procedure, accompanied by a sample of a scoring guide and a scoring sheet for classroom assessment.

SPEAKING TEST
A speaking test is a procedure to measure speaking ability that re- quires the learner to speak, or to produce utterances and he/she is assessed on the basis of hisfner iitterances (underhiil, tgBT).Irr an elementary: leve!, a speaking test rnay be focused on assessing one or two aspects ofspeaking skills such as the pronunciation, intonaiion, and stress.In advancerj level, on the other harrci, ihe assessment is focused on the language tunction.In other words, it does not merely measure ihe language components, which have been mastered by the learner, but it covers ail the communicative abiiity c-'f the learner eovering fluency, grammaticai accuracy, vocabulary chcice, and interactive communication.Hai:mer ( 1993) staies that learning Mu*mirnrien, The Advantage.t of lJsing an Analvtic 37 a fbreign language tbr communicative purposes requires the iearner to master the pronunciation, grammar, vocabulary, discourse, and other relevant skills.Therefore, a functional speaking test should be one which measures the learner's speaking abiiity covering aii the ianguage componenis anci hisi her interactive skills.
In general, a speaking test consists of two parts: ( I ) the speaking prompt (SP), or an instrument to elicit the learner's utterances and (2) the scoring sheet (SS) which contains scoring scales with descriptors or scoring cnteria, accompanied by a scoring guide for the rater on which he/she bases his/ her judgment.A speaking prompt varies, and it might be in the form of a picture, a photograph, a reading passage, or a topic introduced by the ex- aminer, Unfortunately, so far as speaking assessment is concerned, not many experts in evaluaticn proviCe a description on how to develop a scoring guide or a scoring sheet in detail.Experts in language testing such as Oller (1979), Hearon (1975), Valletre (1977), and many orhers have discussed types and techniques of language testing without giving information on how to score the test.Ebel and Frisbie (1986) discuss statistical procedure to estimate the reliability of a test including an essay type one, but they do not provide information on how to make a scoring guide in assessing the productive level of learner's language proficiency especially in speaking.Discussions on the productive level of tests have been dominated by &e measurements of written communicative competence, or writing tests with the approaches, and scoring procedures (Lloyd-Jones, 1977; Jacobs et.al,   l98l; White, 1985; Spandel and Stiggins, 1990).it has been observed that many teachers provide only a speaking prompt without any explicit scoring guide in their evaluation.They base theirjudgment on pre-determined implicit criteria without necessarily making more explicit ones that show what aspects of speaking ability are assessed.In TYPBS O[' SPEAKING TESTS a functional spealcing test is basicaiiy a test to measure the learner's comrnunicative competence.such a test on productive level can be done in two ways in the forms of direct and indirect measurement.A direct mea- surement is done by asking the learner to speak, and he/she is assessed on the basis of his/her utterances.Thus, while the learner is speaking, the rater scores hislher performance.This is what is actually called a speaking test, In contrast to the direct test, the indirect test does not necessarily require the learner to speak but it requires him/trer to make a written response on a given communicative event, such as completing a dialogue, or choosing the best response for a given utterance.This kind oftest, called a paper-and pencil test (Brown, 1987) is not a good test of communicative competence, as it does not ask the learner to speak in a 'real communication'.In other words, the test does not measure the actual production of the learner's communicative competence.It is not suitable for assessing classroom achieve- ment; but it is practical for a large-scale test involving hundreds of examin- ees.i speaking tests vary according to the language elements being assessed and the objectives of the test.Based on the language elements, a la-rguage test in general, can be classified into discrete-point test, integrative test, and pragmatic iest (oller, "979).
A ciscrete-pcint test is a t€st, which mea- sures one of the language compcnents, such as pronunciation, intonation, grammar, vocabuiary.A discrete-point speaking test.then, is a speaking test which measures one oi the ianguage elements mastered by the leamei whiie an integrative test is a test which measures all the language compo- nents at a time.The last type, the pragmatic test, is an integrative test which can be consiciered as a functionai speaking test because it measures the learrr€r's abiiity in using ihe iarget language for communicative purposes in a given context.
According tc the puipose of the evaluation, a speaking test can he classified into two, that is ro measure the learner's ability in using the target language (functional speaking test), similar to oller's pragmatic test, and to measure one or more language components, or a cliscrete-point test.A functional speaking test can be done in arr interactive communication or transactive one.Brown and Yule (1983) proposed two classifications for language function: interactive function and transactive function.The first refers to the function of language to maiiltain social interaction while the second refers to a type of cornmunication that is focuseci on conveying the message, not on the interaction.On the basis of the classification, an interactive speaking test might be conducted in the form of interview or roie play, while a transactive speaking test might be in the forms of story telling, giving a speech" reading an announcement, presenting a repo(, and many others.For classroom purposes, a speaking test should be made by deter- mining the objectives and the types of test and the approach of the scoring procedure.Then, in reierence to ihe objeciives, the types, and approach, the scoring guide and scoring sheet are developed.

THE SCORING PROCEDURE OF SPEAKING TESTS
The scoring procedure ofa discrete-point test and a functional speak- ing test is different.In a discrete-point test, the scoring procedure is simpler because it measures only one or two language components and the rater scores it by refening to the pre-determined criteria.A functional speaking test, on the contrary, is more complicated as it requires the rater to pay lrttention to the overall components of speaking skill at one time.Conse- quently, he/she has to concentrate not only on the language components being assessed but also on the criteria on the scoring sheet.Therefore, due to the nature of the complicated scoring procedure, it is not easy for the cxaminer to obtain a high reliability of this test.
The spealcing test discussed iurther in this article refers to the func- tional speaking test that measures the learner's ability to use the target lan- guage for communicative purposes.Theoretically speaking, there are two approaches in the scoring procedures of a speaking tesc the analytic ap- proach and the impressionistic approach (Underhill, 1987).An analytic scoring procedure is a technique of scoring the learner's speaking ability by separating the components of speaking skill into sub skills, and ihe rater sc()res each component, and then sums the sub scores into a final score.In contrast to the analytic approach, in the impressionistic approach the rater judges the learner's speaking ability on the basis of his/her general impres- sion on the learner's performance without neeessarily separating the speak- ing components.Thus, the rater directly comes to a single score without totaling the sub scores such that in the analytic approach.The term impres- sionistic approach is similar to the term holistic approach for writing as- sessment (Lloyd-iones, i97i; White, i985; Spandel and Stiggins, 1990).They state if the procedure of scoring is based on the analysis of features.it is called analytic, and if the scoring procedure is based on the judgment of the general impression as a whole without any separable features, it is called holistic.

THE ADVANTAGES OF USING AN ANALYTIC PROCEDURE
Even though the analytic procedure has some drawbacks in terms of practicality and reliability, it provides two advantages, especially for classroom purposes.The advantages lie on its ability to provide information on the features or components of speaking sitills achieved by the learner.This information can be used as feedback for the learner to improve his/her per- formance, that is, which language component needs improvement, The second advantage is beneficial for the teaeher a-s it funetions as reliable inforrnation for remedial teaching.On the basis of the scores and the accompanying field notes or comments on eaeh eomponent, the teacher can further make a decision on which speakrng sHll needs more practice in the rerne- dial teaching.This cannat be done in an irr'pressicnistic procedure because the leamer cannot receive explicit feedback and the teacher is unable to icientify the leamer's weaknesses as the sccre does not teil a"rything about which aspects ofspeaking are achieved nor about the ievel ofthe iearner's proficiency.Take for exarnple, a very simpie impressionistic scoring proce- dure taken from l-Jnderhill (i987:60).There are oniy three eategones in the scoring scale as ciescribed beiow.0: inappropriate or seriously incorrect l: relevant but not entirely acceptable 2: appropriate and correct Mukninatien, The Advantages of using an Analytic 4l This kind of scoring procedure is not informative in giving feedback either for the learner or for the teacher, as there is no detail description on the features of speaking skills being assessed.Therefore.this kind of scor- ing procedure cannot serve as a diagnostic procedure in the teaching arrd learning proccss.
A good example of analytic scoring procedure of speaking assessment is the one used by The Cambridge University as part of the First Cenificate in English Examirration.It is used to select foreign students who are going to study in England (Falvey, 1989).This test uses three kinds of speaking prompt (a pieture, a reading passage, and a topic to discuss), and the sp€ak- ing test is done in three steps.In the first step a picture is given to the examinee, and he/she is aske<! to deseribe the picture.The second step re- quires the examinee to read him/herself a passage for a few minutes then the examiner will give some questions.In the third step, two examinees are assigned to discuss a topic so that they can interact each other.There are six components to be assessed in this test: fluency, grammatical accuracy, pronunciation of sentences, and pronunciation of words and sounds, interactive communication, and vocabulary resources.
For classroom purposes in the English Department at Universitas Negeri Malang, a type of analytic scoring procedure shows significant evidence in terms of both validity and reliability.This procedure describes more clearly and explicitly about what is going to be assessed in the speaking achieve- ment test in that it reflects the teaching objectives of a speaking class.In other words, it meets the requirements of content validity as it measures the expected ability stated in the instructional objectives.The major concern about speaking test reliability is the rater's consistency since the scores come from his/trer subjective judgment.Widiati (1998) proves a sufficient inter-rater reliability coefficient in an experimental study: 0.72 far the pretest and 0.88 for the posttest.Thus, it is evident that the analytic procedure shows good reliable scores.The speaking test and the scoring procedure were made by a team of speaking teachers by refening to the Cambridge Irirst Certificate with some modifications as shown in the foliowing scor- ing guide.The scoring guide consists of two parts: (l) a description of the as- pccts of speaking ability to be assessed, and (2) the scale criteria based on which the rater makes a judgment.Before examination, all raters should runderstand the scoring guide.Training or coaching session is held where rrccessary especially for inexperienced raters so that they can come to a t'lose agreement in making judgment.When everything is understood, the cxamination can be done anti two raters are assigned using a separaie scor- irrg sheet.In this case, each learner gets two different scores from two diflcrcnt raters.The two sets of scores are correlated to find out the inter-rater rcliability.A high reliability coeffieient shows that the two raters come to a t'krsc agreement in making judgment.The following is an example of the st'oring shcet.

SCORING SHEBT
Aspecls to be Evaluated J. !i-rieFacii ve LO mrnui'ricauo !-i The ccrmment space is used io ma-Ke noies or commerrts on the leamer's performance and it helps the rater to rnake.judgment and to score the leamer's speaking ability.The comments are useful for planning reme- diai teaching as it provides information on the weaknesses of the learrrer's speaking proficiency.In short, the scoring procedure serves two functions: as an instrument to measure the leamer's speaking ability and as a diagnos- tic proceciure for remediai teaching.