Reflecting on Evaluations at the Jim Joseph Foundation
August 16th, 2017
A core part of the Jim Joseph Foundation’s relational approach to grantmaking is supporting grantees to evaluate their programs—either through engaging an external evaluator or by collecting and analyzing data internally. The Foundation has always believed this is good grantmaking; it builds the capacity of organizations to ask questions, to collect data, and to reflect on findings in a way that then enables changes to be made to increase success.
In this period of transition at the Foundation, the grantmaking team has asked some pertinent questions regarding our evaluation program: “What are we learning from the evaluation work we have supported over the past eleven years?” and “Are there common lessons and emerging themes that we should recognize and reflect upon?” To begin exploring these and other questions, this spring the entire Foundation team gathered for a full day to share and discuss learnings and common themes discovered from a comprehensive review of nearly all of the key evaluations and reports commissioned by the Foundation since its inception.
To make this day as substantive and productive as possible, the Foundation grantmaking team completed “homework” in the weeks preceding the day. We divided up responsibility to collectively review a sample of 42 evaluation reports, capacity building and business plans, and field building research reports—all completed in the first eleven years of the Foundation. Each team member summarized the outcomes, successes and challenges that they identified in the documents.
The “Day of Evaluation Reflection” was highly worthwhile and offered space for the team to explore these evaluations in their totality and to discuss how this information might guide future work. This summary shares a few highlights from the resulting discussions.
The Foundation’s effect on Jewish life and learning.
How has the Foundation affected Judaism and peoples’ approach to Jewish life and learning? This overarching question speaks directly to the Foundation’s mission. A very common theme across many program grants funded and evaluated is fostering positive Jewish relationships and community. With few exceptions, evaluations show that participants in Foundation-supported programs report feeling more connected to their Jewish identity and to Israel when those are the intended outcomes of the program. Since the Foundation DNA includes a broad interpretation of and approach to Jewish learning, these programs include settings from camps, to schools, to service experiences, to Jewish Outdoor Food and Environmental Education, and more. They all are proving effective and align with our mission statement and values.
Lessons learned that have potential to inform Foundation grantmaking.
Several key themes emerged from the day’s discussion that provide opportunities for reflection, focus and improvement, including:
- Various young adult communities can come together successfully through different interests and avenues that resonate and are relevant to the lives of young adults. Social justice and service are increasingly becoming reasons for young Jews to engage in Jewish life. Follow-on programming after an immersive experience is critical to extending programmatic impact, to creating community, and to attaining outcomes.
- Successful programs vary in cost and scale; immersive programs can be expensive and reach a relatively small number of people, but have deep and lasting effect on participants. Particular programs, like doctoral programs in Jewish studies or education, are a long-play, with high-cost per student or participant.
- Mentorship and time for reflection are key elements seen in the success of many programs, particularly those in educator training. Students value a reputable university program and also desire flexible and diverse programs.
- Capacity building for grantees with regards to evaluation, development and growth planning can be important investments. As a relational grantmaker, the Foundation can be in a position to help an organization pivot and/or engage in long-term strategic planning. These plans must be right-sized with realistic revenue targets and investments.
- Relationships among organizations and people matter. There is value in collaboration and strength in building networks; these are also integral components of creating culture change.
- Some grants are set up to leave a system in place to create impact long after the grant concludes. This is an ideal scenario. Local and national funding partners with aligned interests can leverage resources and both widen and deepen impact.
Challenges grantees often encounter.
The day also brought to the fore some of the common challenges grantee partners experience.
- The majority of challenges that programs experience are related to marketing, recruitment and retention. Retaining current participants can be just as valuable as bringing in new participants to a program/initiative. Another common challenge relates to hiring and retaining the right personnel – at all levels.
- Development for sustainability and growth is often challenging — and some very effective programs just are not “sexy” for donors.
- Whole school and/or whole organizational culture change is an effective way to create impact, but often is a lengthy process that takes significant staff capacity and buy-in.
Reflections on evaluation.
In discussions about being intentional in evaluation support moving forward, the team discussed elevating the following concepts:
- Asking good questions and being data informed in our decision-making. Related, evaluations help tell a story for newer Foundation staff members about what is working and what is not.
- Creating opportunities for funding to follow what is working—evaluations can also help inform both if and how to expand a pilot program.
- We can “celebrate failure” in appropriate ways and for productive learning purposes. Furthermore, upon reflection years after a grant and its evaluation are closed, some “failures” actually proved to have success later. Sometimes an evaluation simply captured a moment in time that may not have been the most successful in the life of the program.
- Field building research reports raise the profile of certain programs and certain issues – and dissemination is a very important part of this process so that these reports do not live on a shelf.
- Assessing return-on-investment from our grant-making, which can tell a complete story, is a daunting challenge. Numbers of program participants do not tell the entire story about long-term effects or how someone’s experience influences their heart and feelings. The team reaffirmed a commitment to understand more deeply how Jewish life and learning is experienced and fostered.
Our team viewed the Day of Evaluation Reflection as a productive, enjoyable time for learning. And staff expressed positive sentiments towards the day itself—the structure, the presentations, the team-building environment—and the process outlined for preparation in advance of the day. The conversations were open and honest, signaling that the Foundation’s grantmaking team is comfortable critically examining its past, current and future work with transparency, trust, and patience. The day raised interesting and important questions that we will continue to diligently explore—and, as is our tradition, we will continue to raise new questions and encourage dialogue as a means to improve our work and understanding of the most effective ways to practice and evaluate our philanthropy.