Standard 2

2.1.a Unit Assessment System

Content

The unit assessment system is designed to be a cyclical process, with data collected, organized and analyzed from candidate performance, program quality, and unit effectiveness providing feedback for revision and improvement. Data on candidates are collected at four transition points, and candidates are not allowed to proceed through the program if the criteria at one point are not met. Data include performance tasks at each stage, GPA in education courses, evaluation of dispositions, and evaluation of classroom performance at multiple points.

Content of program evaluations include number of applications and admissions; candidate performance data (aggregated by program); at-risk student data; end-of-program evaluations; alumni and employer reviews, and comparison of action plans to actions completed each year.

The unit is evaluated on number and quality of applicants and admitted candidates; evaluation of clinical experiences by candidates, university supervisors and school mentors; candidate perception of diversity in programs; diversity demographics on faculty, candidates, and school; and faculty performance evaluations, scholarship, service and professional development.

Constructs

The principles that underlie the assessment system are:

(1) Validity: Are we focusing on the appropriate characteristics of candidate, program, and unit that define quality in teacher education?

(2) Rigor: Do the measures both reflect and lead to excellence?

(3) Fairness: Are all measures fair to the individuals to whom they apply? Are they scored in a way that is fair and reliable?

(4) Transparency: Are all (aggregated) data available and disseminated in a way that leads to collaborative and informed decision making?

Process

The unit assessment system was developed through extensive collaboration among stakeholders. The coordinating body is the Accreditation Steering Committee, chaired by the assessment coordinator and composed of the unit’s administrators, program coordinators, assessment coordinator data manager, and the chair of each of the six NCATE standards committees. All issues related to accreditation, such as instruments, procedures of data collection and analysis, policies related to admission and retention, etc., are discussed in the committee and recommendations go to the departments for approval. Any major issues are also sent to the Education Partners Committee; comprised of members of the Steering Committee, plus a range of stakeholders from the community, which meets twice per year. Descriptions of all proposed initiatives and/or changes are sent to the members in advance, in Arabic and English. Although committee members who or not also part of the Accreditation Steering Committee do not have an official vote in decision-making, their input is highly valued and considered.

The Office of the Registrar monitors admissions and sends a list to the unit of applicants who meet entry requirements. The coordinators oversea the program-specific entry requirements, such as the personal interview (all programs), writing sample (all initial programs), and content tests (diplomas). The Office of Student Affairs then works with the coordinators, assessment coordinator, and data manager to ensure that all required data are collected at each transition point in the programs (see Office of Student Affairs Annual Report).

The Curriculum Committee reviews program content and course syllabi periodically. Programs file annual program reports with the assessment office of the university reporting candidate performance unit learning outcomes, derived directly from the conceptual framework, on a three-year cycle with each outcome measured in at least two courses. The report includes an action plan based on the data analysis and developed during program-level meetings and an analysis of the previous year’s action plan. In addition, each program is required to submit a reflective self-study to the university every five years. The feedback loop is closed when the various data analyses are reviewed progressively at the program level, department level, in the Steering Committee, and in the Education Partners Committee. If there is disagreement at any level, the participants engage in additional discussion. The Accreditation Steering Committee, chaired by the dean, has final say.

Evaluation

Every effort is made to ensure that the assessment process is thorough, rigorous, fair, and free from bias. Each year instructors meet with the assessment coordinator to review all rubrics used in the assessment system and revise as needed. All instructors in the same course use the same assessment and score from the same rubric. Rubric criteria are based on the program learning outcomes, the Qatar National Standards for Teachers and School Leaders (QNSTST) and the standards of the specialized professional organization for that program. All unit learning outcomes and standards are evaluated by multiple assignments.

All clinical practice supervisors meet once per semester to engage in inter-rater reliability training on the Clinical Experience Evaluation Survey.

CEES Surveys

Mentor teachers also receive calibration training on all assessment instruments. E-Portfolios are also scored by multiple reviewers.

All evaluation points are measured on multiple measures to provide triangulation. Candidates are given multiple opportunities to voice any concerns about the fairness of scoring on their assignments (from which data are gathered), including access to a grade complaint process through the Office of the Assistant Dean of Student Affairs, End of Course Evaluations and the Exit Survey, which is completed by each graduating candidate at the end of the program.

Each program submits a report annually to the university and receives feedback and also submits an extensive self-review every five years. Each department submits an annual report to the university, as does the unit as a whole.

Key Assessments

Key assessments for transition points for initial programs include a content exam (diplomas) or a comprehensive exam (all other programs), a lesson plan, one or more micro-teaches, e-folio analysis, a unit plan, and an extensive performance-based evaluation survey. Dispositions are measured on a survey twice during the programs. Key assessments in the two advanced programs include comprehensive exams, GPAs, portfolios, and capstone project reports. Other assessments are tracked to assess professional and national standards and unit learning outcomes; all initial programs include an assessment project to show effect on K-12 student learning, an instructional technology project, an Individual Education Plan (IEP), and other selected performance-based assignments according to program.

Assessment Projects

2.1.b Data Collection, Analysis, and Evaluation

Candidate progress

Evaluation of candidate performance is ongoing, with checkpoints at each of four transition points in every program (called checkpoints). Each program office is responsible for collecting the checkpoint data for that program and forwarding it to the data manager, who sends the completed set of data to the assessment coordinator. Program coordinators work directly with instructors and the Assistant Dean for Student Affairs if intervention is necessary with any candidates. For example, candidates may be given remediation and an opportunity to resubmit a performance assessment. Candidates who do not meet GPA requirements are given opportunity to retake a course or courses. The assessment coordinator aggregates the checkpoint data by program, and the results of the analyses are distributed to the programs for analysis and the development of action plans for improvement.

Timeline for candidate assessment

Initial Programs

Each program has three checkpoints between admission and graduation. For the B.Ed. programs, the first checkpoint occurs after the candidates have completed a minimum of two specified education courses. The second checkpoint is just prior to student teaching, and the third is at the end of student teaching, just prior to graduation. At each checkpoint, candidates must meet GPA requirements, specified levels on a performance task, and other requirements as indicated. For example, at checkpoint 1, all candidates in the English concentration must demonstrate English proficiency on TOEFL or IELTS, all candidates must pass a comprehensive exam at checkpoint two, and a broad range of teaching knowledge, skills, and dispositions are measured at checkpoint three.

Since the diploma programs are only three semesters, there is a checkpoint at the end of each semester. As in the B.Ed. programs, there are GPA and performance tasks at each checkpoint. Dispositions are measured at checkpoints two and three, and, as in the B.Ed. programs, a broad range of teaching knowledge, skills, and dispositions are measured at checkpoint three.

For all programs, the fourth checkpoint is one year after graduation, and is a voluntary survey for candidates and their employers. The unit has not been very successful in obtaining this data, but this process is under review to see how we can improve it.

Key Outcomes – Initial Certification Programs

The key outcomes for all six of the unit’s initial programs by checkpoint are as follows:

Checkpoint 1: Ensure candidates have the foundational abilities to proceed in teacher education and to become a teacher.
Checkpoint 2: Ensure that candidates have the knowledge, skills, and dispositions to become teachers.
Checkpoint 3: Ensure that candidates have the knowledge, skills, dispositions, and experience to enter the teaching field, and that they have achieved at satisfactory levels the essential learning outcomes that represent our conceptual framework.

Advanced Programs

Both advanced programs are 18 months in duration, so there is a checkpoint at the end of each semester. As with the initial programs, each checkpoint has GPA requirements, plus other criteria. At each checkpoint, the candidates’ portfolios are assessed to track process. Checkpoint two also has a dispositions evaluation and a comprehensive exam, and checkpoint three includes an extensive evaluation of dispositions, knowledge, and skills on an evaluation by the candidates’ supervisors and on a final internship report.

A complete listing of all checkpoint criteria is included in the Unit Assessment Plan in Exhibit 2.3.a.

Key Outcomes – Advanced Programs

Checkpoint 1: Ensure that candidates have the foundational knowledge to become leaders in education.
Checkpoint 2: Ensure that candidates have the knowledge, skills, and dispositions to become leaders in education.
Checkpoint 3: Ensure that candidates have the knowledge, skills, dispositions, and experience to enter the field of education at leadership levels, and that they have achieved at satisfactory levels the essential learning outcomes that represent the unit’s conceptual framework.

Summary and analyses of all checkpoint data can be found on the unit’s Online Exhibit Library. Full data are available onsite.

Program Quality

Department and Program Level Meetings

All candidate data are reviewed by semester in program-level meetings (above), then in department-level meetings, and finally, by the Accreditation Steering Committee. In addition, in the spring of each year, each program submits a report to the university (Reports from 2010-2015). These reports provide data on the extent to which candidates in the program are achieving the eight unit learning outcomes. By university policy, this assessment process is on a three-year cycle, so that two-to-three program learning outcomes are evaluated each year in at least two different courses. The report requires the data to be reported by course and by outcome on a four-point scale, with the percentage of candidates in each category reported and compared to target levels established by the program. A data based action plan is included, as well as a report on progress toward achievement of action items from previous reports. Every five years an in-depth self-study (below) is completed. In addition, programs reflect internally every spring on the university reports data on unit learning outcomes and on outcomes for professional and national standards and conclude whether changes need to be made in curriculum, instruction, or the assessment process to improve program quality and candidate achievement.

Self Studies

Each program further addresses the standards of the specialized professional association (SPA) appropriate for the program, with the exception of the program concentration areas of Arabic, Islamic Studies, and Social Studies, which, because of the context, have selected and/or written standards that are context-specific. The Masters of Education, Special Education also falls in this category because of the unique status of special educators at this time in Qatar (no certification or licensure available; no national policy related to the status of special education in the schools). For the purpose of program assessment, candidate achievement on these standards is reviewed in the spring of each year.

The key outcomes of this process include:

An annual report of the percentages of candidates demonstrating proficiency on unit learning outcomes by program as compared with established program targets.
Reflection on the on the data by program faculty and staff annually that results in an action plan
Reflection on the on the data by program faculty and staff annually on the previous action plan to gauge progress
Reflection on the on the data by program faculty and staff annually on any other activities or events that had impact on the quality or effectiveness of the program

In the spring of each year, each program is provided a summary of all data collected throughout the year. In program level meetings, all data are considered and decisions are made based on that data to ensure program quality. Summaries of results from each program are then shared at the department level and with external stakeholders in the annual Education Partners’ meeting.

Key outcomes of this process include:

Collaborative, data-based decisions on program strengths, weaknesses, and overall quality
Collaborative decisions related to changes in curriculum, instruction, and/or assessment to improve program quality

Unit Operations

Each member of the faculty is on a committee that is responsible for ensuring the unit is operating at target or approaching target levels in that standard. Specific individuals are responsible for collecting the data and analyzing it (see Attachment 2.3.a, Table 10). The data are delivered to the appropriate person or group at the end of each semester or the end of the academic year, as appropriate for the data source. In the spring of each year, each department within the unit and the unit as a whole complete a self-study report that includes a review of progress and an action plan for the coming year. This process is repeated in the spring Education Partners’ Committee meeting so that the input of stakeholders may be considered. Table 9 in Exhibit 2.3.a provides a list of the data, its source, who collects it, when it is collected, who analyzes and summarizes it, in what format it is delivered, and who receives it. The key outcome of the unit assessment is overall unit quality.

2.1.c Use of Data for Program Improvement

Use of Data for Candidate, Program, and Unit Improvement

At each transition point, which for most programs is every semester, there are performance-based assessments that the candidates must complete successfully to proceed in the program. For these key assignments, candidates may receive additional help and be allowed to complete the assignment again. The goal is to ensure that all candidates are demonstrating the knowledge, skills, and dispositions for teaching appropriate for that level of development. In addition to assessment through key assignments, candidates are also evaluated early in their programs (by the end of their first semester in teacher education) on a dispositions instrument so that the candidate and the program coordinator knows if there are any issues that need special attention as the candidate moves through the program. By the mid-point in a candidate’s clinical experience, they are also evaluated by the mentor teacher and university supervisor (as well as do a self-evaluation). The evaluation is very thorough, requiring assessment of professional practices, unit learning outcomes, program dispositions, the Qatar National Professional Standards, and the standards of the specialized professional organization of that program. If a candidate is not performing satisfactory on any of the above, the university supervisor, the mentor teacher, and the candidate prepare a remediation contract that provides a clear plan for improvement. The goal of all of these actions is to improve individual candidate performance so that all graduates are prepared to be highly qualified teachers.

During the spring reflection sessions, the aggregated data are reviewed to identify areas that may need improvement, including such factors as admission, assessments, instrumentation, curriculum, collaboration with stakeholder, and candidate support. The faculty of each program helps develop an action plan for the coming academic year, based on the data analysis. The action plans from previous years are also reviewed to see whether adequate improvement has been made. Each program completes an annual report to the university that includes the data, the data analysis, and the action plans. In addition, each program completes a report for the dean’s office that includes not only a summary of what has been accomplished and what needs to be done, but also such factors as faculty performance and facilities and other expense needs. The goal of these efforts is to ensure that the programs respond to the data and continue.

After each program has reviewed the data for the program and developed the action plan, the program reports are presented at the department level. The purpose of this is so that faculty from other programs may offer suggestions for improvement or find ways to collaborate for program and unit improvement.

To provide consistently and integrity within the unit, all program data and reports are also reviewed by the Accreditation Steering Committee, which is composed of the unit administrators, program coordinators, the assessment coordinator, the data manager, and the chairs of each of the six standards committees (based on the NCATE standards). The steering committee reviews any issues, concerns, or suggested changes to ensure that any proposed actions contribute to the quality of the unit as a whole and also to offer support as needed. A summary report of all programs is provided to the Education Partners Committee, composed of the members of the steering committee plus faculty from other colleges, candidates and alumni, teachers and principals from K-12 schools, ministry officials, and parents. The advice and input from these committee members is highly considered by the steering committee in making decisions. As appropriate, issues, recommendations or action items may be referred to the curriculum committee or one of the six standards committees for consideration. The goal of this process is to ensure that the data based decision-making by all parts of the unit work together to support each other.