Project Bright IDEA 2: Interest Development Early Abilities

Project Bright IDEA 2: Interest Development Early Abilities 
Final Evaluation Report - Dr. Ron Tzur (09-29-10)

A. Introduction

Project Bright IDEA concluded 5 years of funded activities and an additional year of no-cost extension. This report provides an evaluation synopsis of all 6 years on the basis of the three domains articulated in the proposal: Project goals, activities, and outcomes. Overall, the empirically grounded conclusion of the project evaluator is a solid ‘two-thumbs up’. The intensive, comprehensive, thought provoking, and consistent 21st Century professional development programs were regarded highly by hundreds of participating teachers and principals, and nurtured substantial transformation in their pedagogical perspectives and practices. In turn, this transformation yielded significant impact on the #1 target of any such project—increasing the number of students nominated for, and placed, in AIG programs. This was true not only for students in participating teachers’ classrooms, but also for the entire student populations in project schools/counties (a ‘ripple’ effect). One key evidence to the project’s success were efforts to extend the work to other K-2 classes in those schools, to other grades beyond K-2, and to other schools/districts. It is the evaluator’s contention, and hope, that such efforts will be sustained and extended via appropriate funding (state and/or Federal) within North Carolina and beyond.  Efforts are underway in six of the districts to continue to expand and scale-up across districts and to grades Pre-K-12 using their funds.  Grants have been submitted by Duke University through AAGC and The Research Network to continue to evaluate and scale-up the model across North Carolina.  The Exceptional Children Division of The North Carolina Department of Public Instruction is implementing pilot programs for Coordinated Early Intervening Services (CEIS). 

The mixed-methods of evaluation activities and data sources included:

  1. Extensive search of the literature and discussions with colleagues, and/or participation in their presentations at professional meetings, to situate the project (goals, activities) within the larger context of gifted and talented endeavors.
  2. Videotaped observations, with extensive field notes, of 2-3 days in each of the Summer Institutes for BI teachers, the entire Summer Workshop for principals and AIG coordinators, and two of the BI Annual Teacher Fairs.
  3. Videotaped interviews/field notes with every Institute/Workshop trainer and with a sample of participants in each professional development event (teachers, coordinators, principals).
  4. Videotaped observations/interviews/field notes of the project Leadership Team meetings and of project personnel site visits (hence, also observations of teachers’ work in their schools).
  5. Participant Exit Survey—a written instrument consisting of both Likert-scale and open-ended items—administered at the end of each Summer Institute to all participants.
  1. A mathematics problem-based questionnaire (PBQ) administered to all third grade students who were nominated (and/or placed) for gifted programs in their respective schools (both BI and non-BI).

B. Project Goals

The goals of a project drive its activities and determine criteria for success. In the case of Bright IDEA-2, examining these goals was particularly important because it aspired to become a model program for creating and sustaining new pathways for diversifying the country’s intellectual leadership. To this end, the project set out three goals:

  1. To increase the number of third graders from underrepresented populations who enroll in gifted and talented programs.
  2. To improve teachers’ dispositions toward the nurturing of giftedness in these student populations.
  3. To promote the quality of these students’ meta-cognitive and cognitive capacities.

Project goals were evaluated according to the five questions introduced in the project’s proposal.

A. Were project goals comprehensive and focused enough?

The three goals were found comprehensive, as they link between improvement in student learning and transformation of teachers’ held and practiced dispositions. The focus on teacher dispositions extended beyond the goals of several, partly compatible Javits projects (e.g., Take Five: Unfolding Gifted Education), by stressing the critical role that teachers’ orientations toward giftedness and expectations of underserved students play in nurturing student learning (c.f., Baldwin et al., 2000). The project evaluator suggested to add a fourth goal: To create and ensure implementation of a coherent process (not just a single event) for identifying and placing gifted students in Bright IDEA schools.

The project goals were also found highly focused. First, they insightfully centered on measuring the real impact—number of students identified as gifted—at the commencement of gifted programs in NC (third grade). Second, they made it lucidly explicit that teacher dispositions are indispensable constituents of their practice and its impact on student outcomes. Last but not least, the emphasis on both cognitive and meta-cognitive facets of student development was consistent with cutting-edge, research-based accounts of mental processes that characterize gifted people (Bransford et al., 1999; Marzano, 2001; Sternberg, 2000). The project evaluator suggested augmenting Goal #3 by adding to it Renzulli’s (1978) two requisites for actualization of giftedness, namely, creativity and task commitment.

B. Were project goals well grounded?

A review of national and international literature on giftedness/talent and gifted student education (references above are a good sample), as well as on giftedness in underserved populations (c.f., Borland et al., 2000), revealed that the goals of project Bright IDEA-2 were well grounded. This review was consistent with numerous comments made by expert instructors (e.g., Costa and Kalick, Parks, Moirao, Olive) and consultants (Marzano) with whom the evaluator had conversed.

C. To what extent were project goals unique and scalable?

A central goal stated by several Javits-funded projects was to create, deliver, and promote teacher professional development. Project Bright IDEA, in contrast, uniquely emphasized that transforming teachers’ pedagogies is a means to the central purpose of bringing about changes in the quantity and quality of gifted students from underserved populations. As previously pointed out, Goal #2 makes reference to a specific change in teachers—dispositions—because such a change directly impacts student outcomes. In addition, Bright IDEA-2 was unique in its focus on third graders and on the development of student meta-cognition. It also fit with the array of programs that nurture gifted students by nurturing each and every student in a classroom, that is, the school-wide enrichment approach (Renzulli et al., 2000).

The project goals were found scalable in terms of changes in students beyond grade 3, or changes in teachers beyond dispositions (e.g., teachers becoming change agents through mentoring). The suggestions to add Identification Criteria/Methods as a fourth goal and creativity/task commitment as two aspects of Goal #3 are also examples of goal scalability.

D & E. How do project personnel and participants understand/interpret the goals and to what extent they adhere to these goals?

Interviews with and observations of project personnel can be summarized succinctly as follows. The project team deeply understands each of the goals as well as interrelations among them and faithfully adhered to these goals. The extent to which participants remember, understand, and adhere to the goals varied according to their role (i.e., mentors from Cohort-1 surpass beginners from Cohort-2) and the leadership exhibited by their school/county administration.

C. Project Activities

Project activities could be organized into three main types: Summer Institutes/Workshops, site visits in schools/counties, and Teacher Annual Fairs. The single, most important aspect of P.D. activity evaluation was the team’s serious and comprehensive attempts to continue improving each and every area pointed to as requiring attention in previous years’ evaluation reports. In particular, efforts were made, successfully, to coordinate among the different trainers’ approaches and activities. This focused effort brought about significant increases in teacher appreciation for the summer institute and the 2/3-day workshops. Table 1 below summarizes data of teacher satisfaction levels regarding the Summer Institute and the entire BI professional development program (increases from year to year on the 5-point Likert scale were significant at p < .05 level).

Table 1: Participant satisfaction of BI professional Development Activities

Cohort

Summer Institute

Entire BI Program

1

3.7

3.5

2

4.1

3.7

3

4.2

3.9

It should be noted that in the first year, statistically significant differences were found among counties. The leadership team’s use of evaluator’s feedback led to diminishing those differences, mainly due to improved engagement by county leaders. Similarly, major differences found in the first year among teachers’ satisfaction with their trainers were diminished in Years 2 and 3, with important increases in satisfaction with the components of intro to the overarching (21st Century) model and mathematics.

For each Cohort, the project organized the Annual BI Teacher Fair. These were extremely powerful events, highly attended by teachers and school/county administrators, as well as NC-DPI officials. Teachers’ presentations focused on changes in students’ work along parameters set by the project (e.g., meta-cognition, motivation, self-control, humor, etc.). What has been presented matched well with the evaluator’s observations of site visits, and reflected substantial growth from ‘hard to believe this can work in MY classrooms, with MY students’ to ‘this new approach and the training received are the best thing that happened to me, and my students, in my entire career’. The level of student curiosity, engagement, performance, and products indicated a huge shift in both what teachers seemed to expect of every student and what students expected of themselves. This shift was not easy to accomplish—as indicated by teachers’ initial pedagogical attempts (rather superficial and behavior-oriented). However, as the evaluator’s observations of repeated site visits revealed, promoted teachers’ re-focus on the essence of changes needed, and engendered a substantial transformation. This transformation is further discussed in the next section.

D. Project Outcomes (Goal Accomplishment)

Three measures were set forth to determine accomplishment of project goals: (1) Head Count - number of third graders who were nominated for and/or placed in a school’s gifted and talented program, (2) these students’ performance on a Math PBQ, and (3) changes from pre- to post-intervention in teachers’ responses to the Teacher Disposition Questionnaire items.

D-1: Teacher Disposition Questionnaire

In all three cohorts, the Bright IDEA professional development program made an impact on teachers’ dispositions, toward consistency with the project’s agenda. The most important change found in Cohort-3 was that, unlike the two previous years, NO negative changes in teacher dispositions were found (e.g., views of parents’ contribution to educating their children as gifted)! For Cohort-3, dispositions on 27 (out of 50 items) improved during the first year after initial training (17 items for Cohort-1 and 22 items for Cohort-2). Among those Cohort-3 items, 17 increases reached statistical significance (only 12 for Cohort-1 and 7 for Cohort-2). Due to the small number of participants no county-by-county analysis was possible. The items with statistical significance (< .05) included:

Q. 1a – Teacher seeking opportunities for professional development (4.73 -> 4.96!)

Q. 3 – Teacher decreased thinking of the school’s wealth as a reason for student outcomes (3.52 -> 3.91)

Q. 16 – Teacher tendency to be flexible and experiment with the unknown (3.97->4.47)

Q. 19 – Decrease in teacher’s sole focus on students figuring out correct answers (3.72 -> 3.93)

Q. 22 – Teacher consideration of student racial background as an important resource for their practice (3.05 -> 3.53)

Q. 23 – Teacher effort to involve parents in what s/he does with students in class (3.93 -> 4.09, note clear improvements from previous two cohorts!)

Q. 24 – Teacher actively seeking for professional development (4.17 -> 4.36)

Q. 27c – Love for teaching science (4.24 -> 4.59)

Q. 28 – Teacher awareness that professionalism requires more than a 4-year college (4.24 -> 4.59)

Q. 32a – Teacher increased sense of intimacy with Language Arts (4.16 -> 4.40, compared with 3.88-> 4.38 for Cohort-2!)

Q. 32b - Teacher increased sense of intimacy with Mathematics (4.02 -> 4.53, compared to no change in Cohort 1 & 2!!!)

Q. 33 – Teacher responsibility for actively nurturing G&T already at the K-2 level (4.28 -> 4.71, compared to 4.33 -> 4.60 in Cohort-2)

Q. 34 – Teacher awareness of link between goal accomplishment and student interests (4.31 -> 4.53)

Q. 35b – Teacher establishment of high expectations of ALL students (4.41 -> 4.67, no such change in Cohort-2)

Q. 37a – Regarding a given sample of math problems as suitable for the earliest (K-1) grade levels (4.52 -> 4.90!!!)

Q. 42 – Teacher view of giftedness as a function of nature, not nurture (3.86 -> 4.14, no such change in Cohort-2)

Q. 43 – Teacher increased understanding of the role of metacognition in student learning (3.84 -> 4.22, compared to 4.07 -> 4.35 in Cohort-2)

Thus, the goals of the project were accomplished in terms of teachers adopting key pedagogical principles, including two areas of concern in the first two years: dispositions toward parents’ role and the teacher’s need to proactively partner with the parents, and toward math. These improvements from Cohort-1 through Cohort-2 to Cohort-3 reflect the leadership team’s proactive agenda following specific evaluation feedback and recommendations.

D-2: Head Count

Throughout the BI project tenure, data analyses of student identification for and/or placement in G&T programs (see Table 2 below) demonstrated four main trends (both were statistically significant at p < .05 level or better):

(a) Compared to the control classrooms in the same schools, assessed by the same criteria (set and tested for by the counties), many more BI graduates were proportionally identified/placed;

(b) The BI program caused an increase in identification/placement of students from the control classes, which before the project’s commencement was virtually 0%;

(c) Variance among counties was high and seemed to reflect the aforementioned differences in county leadership (data on those differences were available in previous Annual Reports); and

(d) No disproportional representations of ethnicity and/or gender were found in either group (BI, control).

Additionally, in the last year of the project its team managed to obtain differentiated data for identification and placement. The figures indicate that, in contrast to the control classes, BI identification and placement matched closely. Assuming a similar trend in previous years stresses the substantial impact that changes in teacher dispositions and practices brought forth in students’ learning and excellence. (Note: The extremely high figure for BI Cohort-2 was related to local (two counties) data that most likely reflect a non-recurring situation.}

Table 2: Comparison of Head-Count Figures for BI and non-BI students: Identified based on District’s AIG Five Year Plan which includes: Cognitive Abilities Test; IOWA Test of Basic Skills and Other Tests and Criteria.  See Note on Cohort-3 Data in Section C.

 

Identified

 

Cohort

BI

Non-BI

 

 

1

24%

10%

 

 

2

46%

10%

 

 

3

15%

10%

 

 

D-3: Math PBQ

The first three years of the project demonstrated that a change in the teaching (and learning) of mathematics required a much more concerted effort than what has been provided by the project. Based on the evaluator’s experience and expertise as a mathematics educator, such effort would better follow 2-3 years of implementing the transformed, generic pedagogical approaches and practices. This is particularly the case due to mathematics being a difficult topic for most of the teachers, both in terms of their content knowledge and traditional practices. Consequently, in the structure and evolution of the BI project, students’ outcomes on the math PBQ seemed like an “Achilles Heel.” That is, in the first 3 years of the projects students’ overall performance on the PBQ was disturbingly low, with no differences between the BI and non-BI groups. Findings from the last year of the project suggest that the math-focused efforts with AIG coordinators, principals, and teachers began making some impact, even if modest.

Table 3 below provides data for that last year. Initially, 5 nominal categories for student responses to each item were used: 0 – No answer or “I don’t know”; 1 – Wrong answer (attempt); 2 – Correct answer with no reasoning (except, maybe, for algorithm); 3 – Correct answer with minimal reasoning; 4 – Correct answer with good reasoning. To better compare BI and non-BI students the above categories were ‘collapsed’ into the following three: 0 – No answer, I don’t know, or wrong; 2 – Correct with no reason; 4 – Correct with reason. By these, a brief look at the first category (‘0’) provides an immediate impression of the percentage of students who failed on each item. Fortunately for a comparison between BI schools and the regular population, one county mistakenly administered the questionnaire to all its third graders. Figures for that county appear in the third row (“Others”) and show a rather stark difference with student in BI schools. Of course, these results should be taken with much care, as the students tested in BI schools (either BI or non-BI participants) were those identified for gifted programs. However, in previous years, results of those very students (BI and non-BI in project schools) were substantially lower and resembled the “Others” results this year. The first two rows of each item in the table show that, overall, there has not been a substantial difference between BI and non-BI students. However, on four items (questions 5b, 6a, 21, and 22a, gray background), BI students outperformed their non-BI counterparts mainly due to better reasoning. On one item of those (question 9b) non-BI students outperformed their BI counterparts mainly due to BIs’ wrong/no answers.

E. Affirmative Kudos

After six years, project Bright IDEA-2 demonstrated two essential attributes for which, in times when too many children are still left behind, investment of Federal (and state) funds seem worthy of national recognition and attention: (a) high capacity to initiate and sustain, in a remarkable number of teachers and principals, a desired transformation in their notoriously resistant-to-change modes of teaching and (b) high capacity of the team to foster project improvements via continual, intensive reflection on unexpected problems and application of ongoing, formative evaluation feedback. Combined, these capacities produced a remarkable increase in the number of students who become eligible for Gifted & Talented programs. These findings suggest that Bright IDEA can, and should, serve as a model transformative program for K-2 education and beyond (to gifted as well as general populations).

F.  2010-2011 – Sustainability – Lessons Learned and Scale-Up of Project Bright IDEA – Submitted by Mary Watson, Principal Investigator and Margaret Gayle, Project Director

Many of the research districts have scaled-up many of the components to grades K-5 and they are continuing to train teachers and principals as district funding becomes available.  These districts began scaling-up as soon as they were out of the three-year commitment to the research, based on project data, their local assessments and feedback from teachers and principals.  Models have been adapted for middle and high school and are being implemented in two of the districts.  In addition to identifying and placing more Title 1 students in gifted programs, these districts have shown academic gains for all of their students and many of their schools meeting AYP for the first time, especially where all of their staffs have been trained. 

Feedback through surveys and on-site visits from teachers, in districts that have not expanded, indicate that they will continue to teach using the Bright IDEA Pedagogy and Strategies.  The districts that have not expanded had major leadership changes with principals, curriculum specialists and superintendents, making it difficult to continue.  Changes in key leadership positions became the biggest barrier to continuing with the project.  The other major barrier is the use of instructional funds for many programs that do not work, making it difficult to find funds to purchase materials for students.  In the successful districts, Title 1 personnel, with the Curriculum Coordinators pooled funds to train all of the teachers and to purchase the student materials.  This became a key factor in the districts being able to sustain the scaling up.  At the high school that implemented Bright IDEA, the principal is using his local funds to expand across content areas.   The biggest lesson learned is that to sustain an innovative professional development program of this complexity, superintendents, local boards and policy makers need a long range plan and to be willing to stay on course with re-training teachers to have the tools and the skills necessary to teach a curriculum that is full of rigor and high level strategies to a diverse group of students.    Through the research it has become evident that the PD Model and the impact that it has on student development and achievement is a model for all teachers and principals because the focus is on what works: 1) raise the level of knowledge about rigor and best practices for teachers and principals and to help them to understand more complex research-based strategies for engaging students in complex tasks that will enrich and improve their academic opportunities.

Gifted Intelligent Behaviors: In addition to the Evaluator’s evaluation instruments, the leadership team designed rubrics, validated by experts, for teachers to assess students on the Habits of Mind, Talents, Attributes and Behaviors on a five-level scale for improving the “job or soft skills” desired by employers. See attachment on Results. These GIB’s were integrated into the concept-based curriculum units designed by teachers. 

A Model for Exceptional Children funded by Exceptional Children Division at NCDPI: 2010-2011

The designers of Project Bright IDEA through the Exceptional Children Division of The North Carolina Department of Public Instruction have designed a model for Coordinating Early Intervening Services (CEIS) and Professional Development for their teachers to: 1) prepare children to exit EC classes and perform at successful academic levels; 2) support Reading and Math Foundations; and move students to a deeper level of understanding; and 3) support and enhance Positive Behavior Models. Two districts are serving as a pilot for implementing this model during this school year.

A Model for Alignment with International Baccalaureate (IB) Programs:  Guilford County aligned Bright IDEA components with IB in a middle school with outstanding results, under the leadership of the Assistant Principal who was trained in the Javits Cohort-1 Group.  Teachers indicated that the Bright IDEA training helped them with deeper understanding of IB and how to better apply IB in their classrooms.

A Professional Development Model for All Teachers, with a focus on Low Performing Schools: This model works for all teachers and all students because the focus in on re-training all teachers in understanding how to raise the level of rigor in all curriculum through an integration of state standards and best practices for differentiating instructional delivery and using concept-based interdisciplinary curriculum units with their students.  All students are taught five analysis thinking skills that are needed for understanding basic skills, universal concepts and processes necessary for academic achievement throughout their schooling.  All districts that are using the Building Thinking Skills (BTS) Program consistently have reported success in their schools by making AYP and/or evidence from state and national tests. The Gifted Intelligent Behaviors (GIB’s) provide a model for positive behavior support and for students to be successful in school and to be prepared for the future of work and life.  One of the Wake County Schools received a national award for Closing The Achievement Gap after training the staff in the two components of Project Bright IDEA and implementing BTS and GIB’s for the students.  The principal credited the success of their students as a result of Bright IDEA training.  Lenoir County opened a new Pre-K-5 Bright Tomorrow School in August 2009 and the students have made impressive progress after a year, (See Attachment, Title 1, Northeast Elementary School) School districts continuing include: Brunswick, Elizabeth City, Guilford, Lenoir, Thomasville and Wake.