Experimentation with a Purpose: The Evaluators’ Consortium

As we approach the end of any year, I customarily take time to reflect on the Foundation’s efforts over the previous twelve months. In 2015, with dedicated grantee partners, the Foundation continued to pursue its vision of “increasing numbers of Jews engaging in Jewish life and learning.” There were landmark new grants; grants that concluded with goals exceeded; and evaluations that both offered key lessons and demonstrated outcomes achieved. 2015 also marked our tenth year of grantmaking, which the Foundation celebrated by honoring our founder and highlighting the important work of grantees and evaluators over the decade.

In this, my final blog of the year, I want to share some exciting developments around the Foundation’s Evaluators’ Consortium, comprised of the small number of highly skilled evaluators and researchers with whom the Foundation works. The Consortium’s efforts most likely are not well known to you. Yet the Consortium’s work is deeply important to the Foundation’s efforts and potentially could to lead to novel results in the measurement and assessment of programs in the field of Jewish education.

As we noted, the initial goal in forming the Evaluators’ Consortium evolved into something bigger: moving toward a common set of measures (survey items, interview schedules, frameworks for documenting distinctive features of programs) to be developed and used as outcomes and indicators of Jewish learning and growth for teens and young adults. The Consortium’s convening at the Foundation last month charted new territory for the Foundation in this important direction. For a day and a half, we explored how research and evaluation methods in other fields can be applied to the measurement of Jewish learning and growth. Participants and presenters included leading scholars and researchers from both the Jewish and secular education worlds.

We were fortunate to hear from Professor Christian Smith of Notre Dame University. Dr. Smith began the National Study on Youth and Religion in 1999, examining religious formation, identity, and engagement among predominantly (but not exclusively) Christian youth. His analysis of the habits of religious teens and their families—both Jewish and non-Jewish—along with his own evolution as a researcher examining this area were insightful and certainly will inform the Consortium’s future efforts.

Ms. Cinnamon Daniel, Director of Research & Evaluation for Girl Scouts of Northern California, shared the Scouts’ efforts to develop measurable outcomes across the broad array of scouting programs. Ms. Daniel cautioned about collecting too much data, while noting that data utilization over the long run holds genuine promise for improving the Girl Scout experience.

Professor Anne Colby of Stanford discussed her work in moral development of adolescents. She reviewed several research methodologies the Consortium could consider adapting. Professor Tomas Jimenez, also of Stanford, shared his highly regarded research on Mexican American identity, including that community’s own challenges with assimilation.

The presentation laden with the greatest implications for the Foundation was made by Professor Michael Feuer, Dean of the George Washington University (GWU) Graduate School of Education & Human Development, and Dr. Naomi Chudowsky of TrueScore Consulting. With funding received from the Jim Joseph Foundation, GWU currently is developing a common set of long term outcomes and shared metrics to improve the Foundation’s ability to look at programs and outcomes across grantees and over time. Meeting participants were especially excited to learn about Feuer and Chudowsky’s team’s plan to develop an online menu—in consultation with evaluation experts and practitioners—from which grantees can choose to measure their program outcomes. This would inherently mean that organizations would use common language and measures, a critical step for the field.

Frederick M. Hess, resident scholar and Director of Education Policy Studies at the American Enterprise Institute, observes:

The right mix of experts can help identify tensions, incentives and the contours of possible solutions…Expertise has a terrifically useful place [in problem solving], as long as we understand what the experts actually know, which is how to do specific, concrete tasks right.
– Hess, Frederick. “You say ‘expert,’ I say…not so much.” I used to think…And I now I think… Ed. Richard F. Elmore. pg. 79.

With that principle in mind, a diversity of great minds is critical to the Consortium’s evolution and its ability to pursue its ambitious mission. If you would like an in-depth look at this evolution, please see Cindy Reich’s abridged version of her dissertation, which is a case study on the Consortium. It is an informative document for the Foundation and, I believe, the broader field.

The Evaluator Consortium’s efforts are not front and center in the Foundation’s philanthropy. But the Consortium’s contributions to the Jim Joseph Foundation’s efforts to continuously improve its strategic grantmaking are critical.  What began as a collection of researchers and evaluators with an experimental idea of goals evolved in 2015 into a focused, collaborative effort that I believe puts us on the brink of producing highly valuable tools for Jewish education. Evaluators and grantees working closely with Jim Joseph Foundation professionals bring focus and sophistication to the Foundation’s grantmaking.

Wishing you a Happy New Year.