Documenting Praxis II Data for NCATE Standard One and NCATE/SPA Reports

Some institutions have indicated that they have had trouble obtaining summary reports of Praxis II test data from ETS. Here we provide an overview of NCATE’s requirements regarding use of licensure examinations as one measure of candidate knowledge, and an explanation of how to access ETS reports for your use.
NCATE 80 percent pass rate requirement for units and programs  
NCATE requires its accredited and candidate institutions to maintain an 80 percent (minimum) program completer pass rate on state licensure exams in the content or subject area1. This requirement has different ramifications and expectations at the unit level, and at the program level.

At the unit level, the 80 percent criterion is based on the aggregate pass rate across programs offered by the unit that are subject to an applicable test. An accredited unit must maintain that pass rate in order to meet Standard 1; a pre-candidate unit must demonstrate that it meets the state pass rate in order to meet the preconditions required for a first accreditation visit.

NCATE also requires an 80 percent content test pass rate for individual programs that seek national recognition through the NCATE national program review process. (The NCATE program review is based on national standards that have been developed by professional specialty associations, or SPAs, and adopted by NCATE. More than half of the states with which NCATE has partnerships require programs to undergo NCATE program review. In other states that do not require national program review, some institutions opt to seek national recognition for all or some of their programs.) To be eligible for national recognition, a program must demonstrate an 80 percent pass rate for the most recent annual reporting period2.
Licensure Test Assessment (Assessment #1) in the NCATE Program Report  
As noted above, programs must meet the 80 percent requirement in order to qualify for national recognition. In addition to meeting that requirement, the state content exam and its resultant data must be submitted as Assessment #1 in NCATE’s assessment-based program report3. As with the other 6 to 8 assessments required in the program report, the institution must document the alignment of the licensure test specifications with SPA standards, and must also (1) report test data as evidence of meeting individual standards, and (2) demonstrate its analysis and application of data results to program improvement. These latter two expectations require the program to report candidate distribution of scores on the (typically four to six) categories within a typical licensure test, often referred to by SPAs as the subscores. For example, the Praxis II Elementary Education Content test (0014) consists of questions in the categories of mathematics, language arts, science and social studies. Thus, a candidate’s overall score on the test would not provide evidence that he or she meets the discrete ACEI elementary education standards in each of those four curriculum areas. Nor could a program use overall data on candidate performance on the test as a means to analyze its own program’s strengths and weaknesses in content preparation. Category scores, on the other hand, might reveal that candidates score significantly lower in science than other areas (for example), signaling the program to strengthen coursework, add prerequisites, or take other measures to augment elementary science preparation.

ETS does not provide content category “scores” for Praxis tests. They provide only the number of test questions answered correctly for each category on the test. ETS urges caution in the use of this information. Category-level information is based on a relatively small subset of questions on the test. Because of this, category performance information is less reliable that the official scaled scores which are based on a full set of questions. Also, questions within any given category may vary in difficulty from one test form to another. Therefore, the category performance information on one form of the test may not be comparable to performance on another edition of the test. For these reasons, category scores should not be considered a precise reflection of a candidate’s level of knowledge in that category. However, they can be helpful as one of several measures to demonstrate candidate mastery of a SPA’s standards.
Obtaining Institutional Summary Reports from ETS  

Many of you have asked how to obtain and report licensure test category data in states that use the ETS Praxis II series for licensure test purposes. (Institutions in states that use NES or another test provider may want to contact their state Department of Education liaison for help in obtaining similar information for their own tests.)

Each fall ETS sends an Annual Institutional Summary Report4 to all institutions of higher education (IHEs) that have been selected as the attending institution by one or more Praxis test-takers during the previous testing period which runs between September through the end of August. Prior to publication of the Summary Report, usually in September, ETS sends each IHE a list of all individuals who designated it as their attending institution. This allows the institution to remove from the roster any students who attended the college or university, but are not enrolled in or completers of specific education programs offered by the IHE—thus circumscribing the report to a data set can be used for Title II reporting purposes.

The Summary Report, as well as the preliminary list of test-takers, is a paper document that is sent to the specific individual(s) who have been designated by the institution to receive the report. Most commonly, the designated individual is the person responsible for testing at the institution or the dean of the program. [likely titles, e.g. the dean of education]. If program report compilers needing access to this information are unable to locate the report for their campus, ETS suggests that they contact Karen Palmeri at 609-683-2296.
Data Provided by Institutional Summary Reports  
The Annual Institutional Summary Reports provide a great deal of information for each test, including performance distributions based on candidate-reported demographic characteristics; it also provides aggregated category score information5. In the past, and in order to protect the confidentiality of test-takers, data distribution was only provided when there were ten or more test-takers in each group. Since NCATE program reports require category data for all candidates or completers reported by a program, regardless of program size, ETS’ (certainly reasonable) policy in this regard has caused problems for small programs attempting to meet NCATE program report requirements. As a result, many programs have resorted to obtaining copies of test results from the candidates themselves. Beginning with the 2005-2006 Annual Summary Report in fall 2006, however, ETS will report data distributions for test groups numbering five or greater. This change is largely in response to the request from IHEs for that information.

Category score information is reported on the last two pages of each test report in the l Summary Report. For each category the report provides the range of available points in that category, and the average number of correct responses for test-takers at the institution, for all state test-takers, and for all test-takers in the same time period.
When Candidate Category Scores (“Subscores”) Are Not Provided  
For many if not most programs, the move by ETS to report category data for five or more test-takers is a welcome change. However, especially in smaller institutions, it is not uncommon to have only one or two completers per year particularly in some secondary preparation programs (e.g. physics, French). In the past, ETS has been willing to provide category data upon request to smaller programs, and indicates that small programs should still contact them for help in obtaining the data. Because data in these cases need to be retrieved and compiled manually, ETS charges the institution what it costs them to do the work Beginning with the 2006-07 testing year, ETS will provide those institutions authorized to receive individual candidate scores with the same category information provided the individual candidate on his score report. This will allow small programs to accumulate the data required for fewer than 5 candidates.

In some instances, however, a program may simply be unable to report licensure category data for all candidates and completers across the three-year period that NCATE expects program data to encompass. (In addition, NCATE does not know what accommodations in category data reporting may be available in states that do not use ETS.) No program should assume that it must resort to extraordinary and perhaps unreasonable means in order to obtain category data from its enrollees (e.g., asking a program completer to purchase an additional copy of the report if she has lost the original). When category data cannot be reported for Assessment #1, or cannot be reported for the full roster of applicable test-takers, it is sufficient to simply indicate as much in the program report. Since the recognition decision is based on a holistic approach to the multiple assessments submitted, no SPA can deny recognition to a program on the basis of incomplete data for one assessment. However, lack of category data for the licensure assessment will deprive the program of a source of evidence that is considered eminently reliable by most SPAs, especially when close alignment of the test to national standards can be demonstrated.

1. This requirement does not apply to licensure exams that measure basic skills, such as the Praxis I, and pedagogy, such as the Praxis PLT or discipline-based pedagogy tests.

2. Programs with fewer than 10 candidates in the most current year must base the pass rate on the aggregate of completers over three years. Programs with fewer than 10 candidates over a three-year period are exempt from the 80% requirement. Programs are also exempt when there is no applicable test, or when the state has not established a pass rate for a test.

3. Programs for which there is no applicable state content test must substitute another content-based test for Assessment #1.

4. The Annual Institutional Summary Report should not be confused with Individual Candidate Score Reports that are also sent to IHEs. Individual score reports are sent to an institution (by request of the test-taker) for documentation purposes only, and do not contain category breakdowns of candidate scores. ETS does not currently provide category scores for individual students due to its concerns that some categories have few scored items and the difficult level of items within a category can vary across different editions of a test. Thus ETS believes that category scores would not be a valid determinant of either mastery, or need for remediation, for an individual student in any one specific content category.

5. See a sample of an Annual Institution Summary Report on the ETS website. (provide, click on Praxis, click on Institutions).