In general, NCATE expects reviewers to use their professional judgment in making decisions. However, it is important that decisions are made consistently by reviewers within SPAs and across SPAs. This is difficult to do when we are unable to all meet together and share our thoughts, comments, and opinions until we reach consensus. This list of Decision-Making FAQs has been developed to help reviewers, as much as possible, make consistent decisions when their response is not clear-cut.
How Much Data Must be Presented?
Beginning in Spring 2010, NCATE will be expecting 2 applications of data for all programs submitting initial program reports and at least 1 application of data for programs submitting revised and response to conditions program reports for each of the 6-8 assessments.
Reviewers might give institutions some leeway is in those instances in which a program had to make some radical shifts in their assessments because of the limitation to 6 to 8 assessments or because recent changes have been made to assessments. While programs can have many assessments that they use to determine how candidates are meeting the standards, a program must submit those 6-8 key assessments that are required by the program for all candidates to take. Please see response to the next question.
What if the Program has Good Assessments that Appropriately Address the SPA Standards But Only Have Data on Some of the Assessments?
Reviewers should expect to see 2 applications of data for all assessments on initial program reports and at least 1 application of data for those submitting revised or respoonse to conditions program reports. However, reviewers also need to have the information they will need to make an accurate decision about an overall program. Reviewers should pay close attention to the context in which the report was written. For example, if the assessments have less than the required amount of data, what information has been provided by the program to explain the lack of data? The program should have written an explanation as to why their assessments have less than the required amount. For example, the program might explain that the data from the assessments indicated that candidates were not mastering the standards, so the program decided to create a new assessment that better aligned with the standards. The program should be given the benefit of the doubt if the context is well explained, if there are data collection mechanisms in place and if the program has submitted at least an application of data on all the assessments.
How Many Standards Can Be “Not Met” and the Program Still Receive National Recognition?
The criterion for required percentage of “met” standards is set by your SPA. There are wide differences among the SPAs in their total number of standards so there cannot be a single answer that is appropriate for all SPAs. This information, for each SPA, is included in the program report template (all are available here).
How Many of the Assessments Must be Performance-Based?
All of the assessments should be those in which the candidates demonstrate their mastery of the standard and should be appropriate for the standards being addressed. For example, a paper and pencil test can be a good measure of a candidate’s knowledge base but it is certainly limited in its capability of demonstrating a candidate’s skill level. Most of the assessments should include activities--tasks that are conducted in a classroom, that provide direct measures of classroom performance, or are similar to daily activities a candidate would face in initial employment. Beginning in Fall 2008, all SPAs began accepting course grades as a key assessment. Course grades can only be used for Assessment #1, if there is no state licensure exam, Assessment #2 or for one of the optional assessments.
See also: Guidelines for Using and Documenting Course Grades as an Assessment of Candidate Content Knowledge.
What is a Linked Program Report and How is it Reviewed?
A linked report is a set of program reports that have identical assessments and scoring guides. These programs can ask to have their shells “linked” so that they really only need to submit one program report. When reviewers are reviewing a linked program report, they too, only need to write one recognition report. Once they have completed their recognition report, they can submit it. To move the first report into the second template, they click on the icon (notepad and pen) and it will ask them if they want to move this report into the template. The reviewers click ok and then click on the notepad and pen icon again. This will take them into the filled out recognition report template. The reviewers will need to change the cover page and then can make any changes to the report that they need to. Once changes have been made, reviewers can submit these reports. For a mini video explaining how to review linked reports, please go here.
Where Can I Find Additional Resources for More on Program Review?
The NCATE website has many resources for program reviewers. You can find information on how to become a program reviewer, reviewer trainings, archived web seminars and mini videos, information on the program review process and other important material about program review. Please click here to find out more about NCATE program review.