Content Validity of the National Literary Braille Competency Test
by Carol B. Allman and Sandra Lewis
From the Editor: Because Braille users and those who wish they had been taught to use it have such strong convictions about the importance of effective teaching of the code to children, members of the National Federation of the Blind have worked to ensure that teachers of the visually impaired know the code well themselves so that they can teach it. Unfortunately, a number of teachers have opposed our efforts. They offer a variety of arguments in support of their position, but we have been made skeptical through the years by transparently poor teaching of Braille and, too often, a rigid determination to teach print if at all possible. In short we have become convinced that insecurity and fear underlie a large part of the teacher resistance to the movement toward demonstrated competency in Braille reading and writing for teachers of blind students. A few months ago word began to circulate about an astonishing article that had appeared in the Fall, 1996, issue of RE:view, the journal of the Association for Education and Rehabilitation of the Blind and Visually Impaired (AER). Here is the article by Carol Allman and Sandra Lewis as it appeared:
Criticism of teacher competence in using and teaching Braille contributed to the start of a Braille literacy movement in the 1980's. Because of the Braille movement and the general agreement that Braille is a literary code of importance for some people with a visual impairment, twenty- five states [now twenty-eight], including Florida, have passed "Braille Bills" (Turco, 1993; personal communication, B. Pierce, April 13, 1994). Such legislation reiterates the importance of Braille for some students with severe visual impairments and, in most cases, requires testing the Braille competence of teachers of students with visual impairments.
As a result of the Braille literacy movement, the Braille section of the National Library Service for the Blind and Physically Impaired (NLS) of the Library of Congress has developed the National Literary Braille Competency Test (NLBCT), a criterion-referenced test that assesses skill in reading and writing (transcription) Braille. No other test of this kind exists, and states that have passed Braille legislation have considered using the test for one aspect of certifying teachers of students with visual impairments. If the test, which has not been used as yet, is to be considered for partial use in teacher certification, its content must be determined to be valid.
In 1989, national organizations for the blind (American Council of the Blind, American Foundation for the Blind, Association for Education and Rehabilitation of the Blind and Visually Impaired, Blinded Veterans Association, Canadian Council of the Blind, Canadian National Institute for the Blind, National Federation of the Blind, and the NLS) formed the Joint Organizational Effort (JOE) in part to promote Braille literacy. The group chose the NLS to devise a competency test because it had expertise in Braille codes and no affiliation with teachers, universities, or other education or rehabilitation organizations (National Library Service, 1993).
An editorial committee of eleven professionals involved in education or rehabilitation for blind and visually impaired persons was formed in 1991 to advise on test development. The committee recommended limiting the test to literary Braille (excluding math or music codes), including slate and stylus writing, and not testing Braille teaching methodology. It recommended that university training programs assure proficiency in teaching methodology testing through certification standards.
In the spring of 1992, the editorial committee reviewed a trial test, which NLS revised on the basis of that evaluation. Subsequently, thirty-two peer reviewers (64 percent return rate) in fifteen states evaluated the test. The editorial committee or NLS selected those reviewers from a list of individuals who had expressed an unsolicited interest in reviewing the test. Their responses were positive; most agreed that slate writing was important, although a few thought it unnecessary. Most recommended more multiple-choice questions. NLS revised the test based on these recommendations (Stark, 1993b).
The NLBCT is described in news releases (National Library Service, 1992, 1993, 1994) as a three-part evaluation of general knowledge of the Braille literary code. The test assesses (a) the ability to write by using a slate and stylus to Braille one medium-length paragraph, and a Braillewriter to transcribe one full print page, and (b) the ability to identify Braille errors in four medium Braille paragraphs. It also requires the candidate to answer twenty-five questions on the use of Braille rules. Candidates can use a dictionary, but not Braille reference materials, to complete the test within four to six hours. NLS will grade the tests and set passing scores. NLS has set prerequisites for taking the test the first time and guidelines for subsequently retaking it.
The Braille literacy concerns of JOE indicate that professionals and consumers in the field generally support the concept of a Braille competency test. However, the NLBCT has not been rigorously validated, and testing and measurement specialists agree that assessments used to obtain teaching certification should have psychometric characteristics that include assurance that the instrument used has job relevance (Gorth and Chernoff, 1986). The measurement literature on validation of teacher certification tests, although limited (Schmitt and Borman, 1993), supports the need for content validation, particularly for tests like the NLBCT that are criterion-referenced tests of skills used in teacher certification (Shimberg, 1981).
Unfortunately, the current development of the NLBCT consists of expert judging and peer review based on personal expertise (Stark, 1993a) and not of job analysis data. The recommendation by peer reviewers to assess only knowledge of the Braille code and not methodology raises particularly the question of the need for teachers to demonstrate ability to transcribe materials using a slate and stylus. There is agreement in the literature that teachers should teach slate and stylus to students. However, if the NLBCT is designed to assess demonstration of the Braille code and not teaching of Braille and related communication skills, the requirement of slate and stylus writing is questionable.
Issues Surrounding Content Validity
Technical adequacy of any test through the use of psychometric techniques is considered standard procedure as outlined in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1985). Primary standards include evidence of validity. Validity of criterion-referenced tests is widely discussed in the literature, but content validity is generally recommended as the primary validation of interest. For teacher certification purposes, test content validation is generally determined through an investigation of practitioners who either report or demonstrate the skills tested while on the job.
As prospective teachers are tested for competency, it is critical that the competence be based on the knowledge, skill, and ability that is demonstrated by practicing teachers. The job relevance of testing for certification purposes is upheld by Standards for Educational and Psychological Testing (American Educational Research Association et al., 1985) and the Equal Employment Opportunity Commission's overview of the Adoption by Four Agencies of Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, 1978).
The assessment of skills through performance rating of demonstrated skills has traditionally been focused on occupational areas that lend themselves to completion of a product through simulated performance of specified skills, such as secretarial (typing exams) and mechanical functioning (car repair, building construction, or assembly-line skill)(Fleishman, 1982; Hambleton and Rogers, 1991). Typically, teacher skills are not defined in ways that allow their simulation for skill performance assessment. However, Braille transcription competence is a skill that lends itself to a performance rating on a criterion-referenced test.
Legal opinions support the critical nature of content validity for teacher certification testing. In a 1971 case, Griggs v Duke Power Company, employees of Duke Power Company challenged the legality of an employer's using general ability tests to hire and advance employees. That landmark case in personnel testing established the concept of job relatedness in finding that the general ability tests were not validated by correlation with job relevant tasks (Bershoff, 1981). In a 1975 case, Albemarle Paper Company v Moody, an employer's method of determining validity of an employee test was found defective. A psychologist hired to validate an employee test compared test scores of current employees with supervisors' judgements of competence. The court stated that the validation did not analyze attributes or particular skills needed in the job and thus was defective methodology. (Bershoff, 1981).
Two cases in 1981 further addressed the validity of employment testing. The United States v City of St. Louis case challenged multiple-choice questions and simulation exercises on an employee test developed by a panel of experts. The court concluded that the items were based on opinion rather than actual observation of correlation between mastery of knowledge and abilities measured by the test. Although the test assessed reading and writing skills that were dissimilar to those needed in the work situation, it threatened the use of simulated situations in the testing situation. In Guardians Association of New York City v Civil Service Commission, perhaps the most sophisticated opinion on employment test validity, the court found the functional approach of job relatedness for content validation to be appropriate. The court upheld the city's use of content validation strategies, supporting further use of job-relevant content validation for tests of teacher certification. (Bershoff, 1981).
Bishop (1993), Harrell and Curry (1987), Heinze (1986), Olmstead (1991), and Torres and Corn (1990) provided descriptions of duties of a teacher of visually impaired and included the transcription of materials into Braille and the interlining of Braille materials with print. Teachers may demonstrate these transcription skills by using a Braillewriter, a slate and stylus, a Braille computer program, or a Braille transcriber aide. Any assessment of these Braille transcription skills must necessarily reflect on-the-job activities as carried out by teachers. Unfortunately there are no quantitative studies or job analyses to suggest how teachers proceed with Braille transcription duties.
The Present Study
In the present study, we address the content validity of Braille transcription on the NLBCT. The need for the test is not at issue. If the test is an instrument for demonstrating "basic competency in literary Braille" (National Library Service, 1994, p.1), then proper validation procedures should show that the test is related to the job performed by the teacher of students with visual impairments.
Method
The Questionnaire
We designed a Braille Skills Analysis Questionnaire (BSAQ), choosing the questions by reviewing the literature on Braille- related communication skills and by evaluating the purpose, content, and process of Braille transcription skills on the NLBCT practice test. We designed the questionnaire so that the teacher respondents could indicate how and how often they used Braille transcription skills, particularly their use of slate and stylus, Braillewriter, Braille computer programs, and transcriber aides in transcribing materials and transcribing with the use of Braille reference materials.
We asked fifteen visual impairment professionals to review a draft of the BSAQ. Using responses from nine of those reviewers (60 percent), we revised the draft. We then asked ten potential participants to complete the revised test to determine its test-retest reliability. We had established a priori that a test-retest reliability of .85 would be acceptable for ascertaining that the questionnaire would produce reliable information. Eight participants (80 percent) returned the completed questionnaire. Two weeks later we sent a second questionnaire to those participants who returned the first one. Six of the eight participants (75 percent) returned the questionnaires for test-retest reliability computation. We obtained an average test- retest reliability of .87 from those six responses. The field test participants were not part of the initial review, and both groups of participants were deleted from the participant pool.
Participants
The participants were 233 teachers of students with visual impairments in Florida, whose names we obtained from the Florida Instructional Materials Center for the Visually Handicapped, which maintains an annually updated list of all teachers of students with visual impairments in Florida. These individuals would be a source of current on-the-job information about skill in Braille transcription. The sample of teacher participants consisted of thirty-two men (14 percent) and 201 women (86 percent).
Procedure
The Florida Department of Education mailed the test and a return-address, stamped envelope to the 233 teachers. The tests were coded to the addresses of the participants. The coding was accessed only by a research assistant who maintained records on the return of the tests and sent follow-up letters four weeks after the original mailing to those who had not completed the questionnaires.
Results
Eighty-one percent (189 of 233) returned the test. Of those 189, 181 questionnaires (96 percent) were completed in a usable manner. The eight unusable questionnaires were not completed because the recipients were no longer teaching. Twenty-six men (14 percent) and 155 women (86 percent) completed the questionnaire. Seven participants reported that they were tactual Braille readers. Seventy-five percent of the respondents were teachers of visually impaired children; 7 percent were orientation and mobility specialists; 14 percent had dual assignments; and 4 percent gave no identification. Of these teachers, 42 percent served prekindergarten through secondary school children; the remaining 58 percent, in about equal proportion, taught secondary school children only, elementary and secondary school children, elementary school children only, or some other combination of ages. Sixty-seven percent of the teachers reported that they had taught for more than ten years; 59 percent stated that they had taught students with visual impairments for more than ten years. The data in Tables One and Two indicate the locations of teaching assignments and the number of visually impaired children each teacher taught.
Teachers who did not use Braille in their teaching assignments were asked not to answer the remaining questions. Eighty-nine teachers (49 percent) reported transcribing Braille; 96 percent of those used a Braillewriter, 64 percent used computer software, and only 12 percent used a slate and stylus.
We had decided before mailing the tests that to be reported as content-valid a skill had to be used by 85 percent of transcribing teachers. Based on the report of eighty-nine teachers in Florida who transcribe materials for students as part of their current job, the NLBCT skill of producing print into Braille by a Braillewriter can be considered content-valid. The NLBCT skill of producing print into Braille using a slate and stylus is not content-valid based on the Florida responses. Although the skill of using a slate and stylus is described in the literature as desirable for teachers who teach students with visual impairments, the reproduction of materials into Braille by that method is not a skill that many teachers in Florida use. Most teachers who transcribe materials do so with a Braillewriter or computer software.
Of the 104 teachers responding to the question about using Braille transcriber aides to transcribe materials, sixty-five (63 percent) reported not using an aide. However, sixteen (15 percent) reported using an aide for 3-5 hours a week; 16 (15 percent) reported using an aide less than 1 to 2 hours; and 7 (7 percent) used an aide for transcription from 6 hours to more than 10 hours weekly.
One hundred teachers answered the question about using reference materials for transcription, and eighty-five teachers answered the question on using reference materials for interlining Braille with print. The data in Table three show the frequency of reported use of reference materials. Table four contains data on the amount of time teachers spend weekly in interlining and transcribing Braille.
Interpretations of the findings in this study need to consider the following limitations: 1. Participants in this study were volunteers and may not be representative of the population of teachers. 2. Participants were limited to the state of Florida. 3. Participants in this study may have previously participated in some aspect of the NLBCT development. 4. Data from this study are self-reported information and may reflect the participants' biases.
TABLE One. Teaching Assignments of Respondents Assignment Number Percent
Residential School 25 14% Resource Room 24 13% Itinerant Teaching 99 55% Special Class 21 12% Other (supervisor, media spec.) 12 6%
Total 181 100%
TABLE Two. Number of Students With Visual Impairment That Respondents Serve
Program Mode Range M Mode Itinerant teaching 2-32 14.1 10 Resource room 1-65 11.5 9 Residential School 3-100 27.4 27
TABLE Three. Frequency of Teacher Use of Braille Reference Materials Use Always Sometimes Never Transcribing Braille Materials 21 70 9 Interlining Print Materials 14 47 24
TABLE Four. Time Teachers Spend per Week (in Hours) in Transcription and Interlining Skill <1 1-2 3-5 6-10 >10 Slate and stylus 10 1 0 0 0 Braillewriter 18 30 21 11 5 Computer Software 7 21 13 6 3 Interlining Braille 9 27 15 7 2
Conclusions and Discussion
We designed the collection of data in this study to determine if teachers transcribe Braille using a Braillewriter and a slate and stylus without the use of reference materials as assessed on the NLBCT. The data from this survey support the assertion that transcribing Braille with a Braillewriter is a valid skill to assess as a certification requirement for prospective teachers of students with visual impairments. Using a slate and stylus and transcribing Braille without using reference materials are not valid components for certification requirements.
Wittenstein (1993a, 1993b) found that over half of the subjects he surveyed felt that it was not desirable for teachers to be certified transcribers of Braille. Currently, Braille transcribers are certified through a test similar to the NLBCT that requires transcription on a Braillewriter with use of reference materials and with particular attention given to format, structure, and lack of errors on the transcribed document. If teachers are to be assessed on their ability to transcribe Braille for student use, those skills should be assessed through ways typically used by teachers and with attention to lack of errors on the transcribed document. The data from this study indicate that transcribing using a Braillewriter and reference materials is a skill used by over 90 percent of the teachers who use Braille in their classrooms. Unlike the NLBCT, the applicants for the Braille transcriber test may use reference materials and may complete the test in a setting of their choice.
Over half of the teachers using Braille spent one to five hours a week interlining print with Braille. Interlining appears to be a critical skill for a number of teachers and is necessary in the management of students with visual impairments in regular classrooms. If regular classroom teachers are readily to accept students with visual impairments in their classrooms, they need assurance that the materials are accessible. This suggestion is supported by Bishop (1986), who identified factors in the successful mainstreaming of students with visual impairments. This finding supports the suggestion that teacher preparation programs include the skill of interlining in Braille coursework.
The data from this study indicate that 64 percent of teachers using Braille in their classrooms transcribe with computer software. This skill may reflect a future trend and probably reflects teachers' desires to complete needed transcription in a timely, simplified fashion. It does not suggest that teachers are illiterate in the Braille code.
The finding of some use of aides for transcription may indicate a trend in the use of trained transcribers, which is supported by Currey and Hatlen (1989), who reported that teacher "aides are often assigned the job of Braille transcription and that teachers of the visually impaired are assigned the job of training those aides in the fundamentals of Braille transcription" (p.61). This information suggests that a Braille skill that may need to be included in teacher competency in the future is the ability to train teacher aides in the transcription of Braille.
Based on the results of this study, we do not recommend using the NLBCT in its present format in certifying teachers of students with visual impairments. Competence in transcription of Braille on a Braillewriter using reference materials is a job-relevant, content-valid skill expected of any teacher of students with visual impairments. In addition, teachers should be competent in interlining materials.
Teachers must have the opportunity to continuously renew and upgrade teaching skills. In this study, we report that 51 percent of the respondents indicated that they do not currently use Braille, and often teachers go for several years with no Braille-reading students; these teachers require Braille and Braille device updates (Olmstead, 1991). New and improved technology, methodologies, and materials become available and require learning or renewing (Maron, 1983). Commonwealth of Virginia (1991) and Wittenstein (1993a, 1993b) report that although teachers generally feel confident in their Braille skills, they desire some level of inservice training on various Braille-related communication devices.
Based on the findings of this study concerning the content validity of the NLBCT, the Braille Competency Committee of the Florida Department of Education recommended that this test not be used for teacher certification. The Braille Competency Committee established Braille competence standards for teachers and recommended that prospective teachers' competence in Braille be assured through the content of university courses, including passing an examination that allows use of reference materials while (a) transcribing a lengthy passage from print to Braille on the Braillewriter and (b) interlining Braille to print. This committee, recognizing that caseloads of teachers may only sporadically include students who are Braille readers, also recommended that regular inservice training in Braille be initiated for teachers who believe that their skills are rusty. More than 100 individuals participated in four regional two-day Braille refresher workshops in the spring of 1996. Future plans are to provide advanced Braille updating, which would include teacher competence in the use of software for transcription and the training of teacher aides to assist in the transcription of Braille.
NLS has announced that it will proceed with a nationwide effort to validate the content of the NLBCT. Our research, conducted in only one state, can be used as a pilot study for the larger investigation. It will be interesting to see if the transcribing practices of Florida teachers are similar to those of teachers in other states. Should the nationwide validation confirm that teachers primarily use Braillewriters and reference materials when transcribing materials from print to Braille, it seems reasonable that the NLBCT can be made more content-valid by making changes to the testing procedures that reflect these job-related practices.
In the meantime, states that have adopted the current version of the NLBCT for teacher certification may want to reevaluate their decision. If the test lacks content validity, as determined in this study, continued use of the NLBCT as a determinant of employability may not be upheld in the courts.