Evaluating My Stay At a Juvenile Detention Center:
Noticing How Training and Change Move With
Each Other
Written by Michael Bischoff
April 14. 2001
I am afraid of teenagers. When I was a teenager, I was afraid of myself, and a good chunk of this prejudice and discomfort with people in high school remains with me. Why, then, did I chose to spend three full days locked inside of a Juvenile Detention Center (JDC), co-facilitating a conflict resolution workshop with youth? And why am I planning to do it again? There is something that pulls me to what I am afraid of in ways that I will never quite understand. There is also a part of me that has fallen in love with facilitating experiential workshops, so that part of me drew me to this experience as well. I have also had some of my deepest and most moving experiences while doing workshops inside of adult prisons--and I was eager to get a taste of the youth version of prison.
Now I am looking back on that workshop at a JDC in Winchester, VA, to evaluate what happened and also to learn more about broader principles of evaluating trainings about conflict transformation. You can already see that an objective, detached evaluation of the workshop is not what I am after. I went into this experience full of biases, insecurities, ego, and fears. I also went into the workshop with my own learning and growth at the forefront of my mind. I hope to intertwine this subjective slant with ideas about broader strategies for evaluation. I will start with my personal reflections and a description of the evaluation processes we did use--and then look to other evaluation models to help think of ways to improve our evaluation process.
Evaluation can happen on so many different levels, and I am interested in the messy mixture of them all. I want to know what changed for me personally, in my attitudes and skills in the course of doing this workshop. I want to know how I did as a facilitator; if I communicated clearly, for instance. I want to know if the youth that were at the workshop enjoyed the workshop, and if they would do it again. I want to know if the youth will actually put into use any of the skills or ideas we covered in the workshop. If they do put the ideas into use, what will happen the next time someone challenges them to fight? Will the workshop influence them to challenge an injustice in their community? I want to know if those kids will commit another crime and end up back in the JDC next year. I want to know if the workshop we did had any influence on the atmosphere of the Detention Center on a day to day basis, especially in regard to the relationships between the staff and residents. I want to know how we can do workshops at this center that will contribute toward a transformed version of juvenile justice, that focuses more on restoration and less on retribution. I want to do this workshop in a way that is a part of the undoing of structural, systemic violence in our country, such as racism. In addition to checking the impact of all my hopes, I want to know about the unintended, unseen consequences of this training. I want it all. While I went in carrying all of these ideals somewhere in my body, most all I saw on the horizon during the workshop was my insecurities and thoughts about what would best engage these kids from minute to minute.
This particular workshop was especially full of aspirations and hopes to evaluate these wishes, because it was the first workshop of its kind at this Detention Center. The other person who facilitated the workshop with me, Geoff Huggins, had interviewed the superintendent of the center as part of a Listening Project about Restorative Justice. Out of their discussion, they agreed on a plan to try out this workshop. Geoff has several years of experience with an elementary school program called Peaceable Classroom, and some recent experience facilitating Help Increase the Peace Project (HIPP) workshops in high schools and middle schools. The superintendent of the JDC thought it would be worth trying a similar program at his center. I have been facilitating Alternatives to Violence Project (AVP) workshops in prisons for a few years, and had connected with Geoff through Quaker connections. Geoff and I agreed to draw ideas from our related experiences, but to modify this workshop based on the particulars of this context. We were making things up so much, that we didn't even decide on a name for the workshop until after it was over. Somewhat based on suggestions from the youth, we came up with the name and acronym, Help Everyone Learn Peace (HELP).
Marie Dugan has said that it is "impossible to judge whether one's immediate actions are helpful without having a long term vision in place against which to understand and evaluation them." (Dugan, 96) I find reminders such as this to embed short term activities in the context of long term visions both inspiring and annoying. Geoff and I certainly had some long term wishes for this program. We were doing this three day workshop as a pilot at this facility, to help find out if an ongoing program would be a good idea--and, if the program continued, we wanted to discover how to further adapt the program to this setting. But a long term vision of what kind of change we were hoping to contribute to was, in my mind, quite fuzzy as we walked into the building on the first day. I was there because I wanted to get more facilitation experience, get to know Geoff, and maybe reduce my fears of teenagers. We were there to do experiments, but I didn't have a clear hypothesis in mind. Geoff and I had done hours of planning of the details and the agenda, but I hadn't given much critical attention to the changes we were hoping for. I saw the workshops as a chance to get to know the facility, the staff, the residents, and Geoff. If and when we do another workshop there, I would want to enter with a crisper long term vision.
What we ended up with on the first day of the training was 17 youth who were residents, 3 JDC staff as participants, and another 2 staff watching over our shoulders in the corner of the room. The residents were a mixture of both boys and girls--some of them were awaiting trial, and some of them had been sentenced to up to six months in this facility. Their ages ranged between about 13 and 17. We had strongly emphasized that we wanted all of the participants to be there on a voluntary basis, but it became clear that the staff put some pressure on some of the kids to volunteer. The workshop lasted for three days, going from 8:30 till 2:30 each day.
Geoff and I planned for evaluation in several ways. After each 2 or 3 hour chunk of the workshop, we asked the participants to brainstorm what they liked, didn't like, and their suggestions for improvement. At the end of the final day of the workshop, we had each participant fill out a written evaluation about the whole workshop, responding to a handful of questions. In the written evaluation, we asked what they thought was the best part of the training, what was the most important skill they got, something they'd change, whether they'd recommend a friend take it, and if they'd be interested in taking an advanced training. After the second day of the workshop, we talked with two of the staff participants about their impressions and suggestions. After each day of the workshop, Geoff and I informally evaluated how things were going, and we made several changes to the agenda throughout the weekend, based on the participants evaluations and our own observations. After the final day of the workshop, Geoff and I gave each other feedback focused on our facilitation. About one week after the workshop, Geoff met with the Superintendent and a Security staff person to talk through how things went and to discuss future options. Geoff also wrote up several pages of his personal reflections, and gave copies to the Superintendent, me, and others. About one month after the training, Geoff and I went through the agenda we used and evaluated how each exercise went and described how we would do it differently the next time.
Evaluating our Evaluation
Looking back at that last paragraph, I realize we did quite a bit of reflecting and evaluating for a three day workshop! Part of the advantage in this situation was working with Geoff, who strikes me as naturally quite a reflective person. Given the context, I am happy with the various kinds of evaluation we did--and I am also interested in thinking of ways to push this evaluation further and broader. My pattern in doing similar workshops is to present a fairly packaged workshop to a group of inmates, hope it does some kind of good, and make small changes in the way we do the workshop based on immediate feedback. I would like to step back and think through how I can do these workshops with a little more strategy and accountability.
A few weeks prior to the workshop, I suggested to Geoff that we meet with some of residents before the workshop to get their input about what would be relevant to them in the workshop. After the workshop, I suggested we meet with the same residents to get their feedback. We didn't end up doing this, and I would be interested in giving this a try in the future. Geoff ended up making 20 minute presentations to each of the living units at the JDC, mostly to explain the program to see if they were interested in attending. Asking input from the youth about how to conduct a workshop they know nothing about can be a challenge--but I think it could useful as well. We would need to think through how to present the purpose of the workshop, and think of focused questions that would draw out relevant information (kind of conflicts they deal with, etc.). In this past workshop, the level of investment from the youth participants varied quite a bit. If a handful of youth entered the workshop feeling like they had a say in how we did things, that might make a difference in how engaged the group was.
A similar evaluation strategy would be to do follow-up interviews with some of the participants a few weeks or months after the workshop happened. In this interview, we could ask if the workshop had any impact on how they interact with people, or in any other ways. An additional option would be doing individual interviews prior to the workshop, asking about their understanding and practice of nonviolence-related skills and attitudes. With the pre-workshop interview, we would have more to compare the post-workshop interviews with. This interview process could be done by either by the facilitators or by additional outside volunteers. A similar interview process would take extra time, and I don't know that it is the best fit in this situation--but it is an evaluation method that would be appropriate in some workshop settings. I helped coordinate a post-workshop interview process with participants in high school and prison Alternatives to Violence Project (AVP) in Minnesota, and this evaluation provided useful, but incomplete feedback about the long-term influence of the program on individual behavior.
Evaluation as Relationship Building
A primary area for improvement in evaluation for this workshop seems to be related to our relationships with the JDC staff. In previous prison workshops I have done, the AVP trainers have tried to keep as independent of the corrections staff and system as possible. For instance, there are no prison staff in the room while we hold the workshops, and we don't accept any money from the prison or any part of corrections. In this situation, Geoff and I intentionally worked closer with the JDC staff. Based on Geoff's interview with the Superintendent, we came to understand that the focus of the JDC is less on punishment than adult prisons, and that the staff see themselves more as teachers and care-givers more than guards. Because of this, we welcomed some staff members to participate in the workshop, and tried to keep our planning and feedback closely connected to the staff.
Geoff did an excellent job of connecting with the Superintendent, Bob, and getting his input about many steps in the planning. But during the three days we were doing the workshop, Bob was not at the facility. While the staff who were present on those days were quite helpful in many ways, none of them seemed as enthusiastic or understanding of the program as Bob was. Geoff had met with two of the staff we volunteered to be participants to ask them to leave their authoritarian, disciplinarian roles outside of the workshop, and to be there strictly as participants. It was very hard for these staff members to let go of their normal roles. Although they contributed many great things as participants and seemed to make some new personal connections with the kids, the staff were fairly heavy-handed throughout the workshop. During breaks, one staff frequently told some kids how disappointed she was with their behavior or lack of participation in the workshop. In the midst of the workshop, the staff sent several looks and pointed fingers to remind kids to "get back in line."
As I said before, it also became apparent that some of the JDC staff had "strongly encouraged" residents to attend to the workshop, which was not our intent. In addition, one or two additional JDC staff often sat in the corner of the workshop room, just observing what was happening. After the workshop, we found out that having these observing staff in the room was not part of the way the Superintendent imagined things happening--but since he was not there during the workshop, other staff decided what to do.
All of these interactions with the staff point me toward a need for more dialogue between the JDC staff and Geoff and me. Partly, the staff didn't know how much they could trust Geoff and me--and partly they were there to be helpful and make sure the program worked. And, in many ways, the staff were right to be concerned about leaving the residents alone with Geoff and me. Neither of us were that confident in "discipline" or "control" of the kids, especially using the approaches that were the norm at the facility. We had our own methods of approaching cooperation, respect, and guidelines--but these methods were pretty untested with these residents, and were not entirely understood by the staff.
After the second day of the workshop, when the JDC staff were telling us their impressions of the workshops, I started to realized how much they saw that I was oblivious to. Both of these staff were stressed because of the tension throughout the day between the boys and girls in the trainings. The staff said that some girls were on the edge of a physical fight when they returned to their living unit after the workshop, primarily because of boy-related tensions. While it is easy for me to think of all the things I want staff to know about and cooperate with our program and philosophy--I have immense amounts to learn from them about their perceptions and the wisdom they have gained in being with these youth.
In retrospect, I am increasingly seeing evaluation of this program in terms of relationship building. Joint reflections with the staff have served as essential parts of building our relationship with the staff--and more could still be used. Asking the residents for their input and evaluation of the program before, during, and after the workshop is key part of building relationships with them. As we listen, respond, and follow through on their input, trust is built. Geoff did a wonderful job of building a relationship with the Superintendent before the workshop, and he also touched base with many other staff and residents throughout. Beyond the connections we did make, additional time with staff and residents prior to the workshop seems like it would have been helpful. Given the restrictions of time and energy, I don't know how much more is realistic, but I recognize it as a gap. If the program continues at this facility, we will have the benefit of building upon these past learnings.
One possible follow up to the workshop I have suggested is a half day training exchange and dialogue with the JDC staff. In this scenario, Geoff and I and others potential trainers from the community would meet with several of the JDC staff for a half day workshop. Some of the time could be for the JDC staff to explain their processes for working with the kids, their guidelines, and tips for outsiders coming in. Geoff and I could present some of the underlying values, framework, and methodology of where we are coming from. After these two presentations, we could brainstorm and discuss ways to move forward with the program--and also think through what we can learn from each other's approaches. I believe this would be useful for the program--but also useful for myself in clarifying and pulling out some of my beliefs about how change can happen in this situation.
Theories of Change
In her article, "Nothing as Practical as Good Theory," Carol Hirshon Weiss describes the relevance of our beliefs about how change happens for social service evaluation. As I said earlier in this paper, I facilitated this workshop while not being very clear about what kind of change I was hoping would happen. I was also not sure what would lead to this ambiguous change.
I take this article as a challenge to put more precision into my theories of change and as a push to build in ways to test these theories. Beneath my desire to be a part of conflict transformation workshops in correctional facilities, there are several half-articulated theories of change swimming around. Here are a handful of those fishes:
I believe that practicing communication and problem solving skills will increase participants' capacity to constructively deal with conflicts in their own lives, which will lead to a lesser chance of committing other crimes and increase the chance of healthier relationships.
I believe that affirmation of one's self and others will lead to an increase in self-esteem. I believe that low self-esteem increases the chances of a person using violence. Thus, an increased level of self-esteem will reduce a need and desire for participants' to use violence, and increase respect for others.
I believe that violence often happens because those using violence don't perceive that they have any other ways to get what they want. Thus, if participants increase their understanding of nonviolent methods of change, they will increase their own sense of power and have less need for violence, and be more likely to get what they need.
I believe that everyone has experience with both violence and nonviolence, and that reflection on what works and doesn't work in those experiences will lead to an increased recognition of the power of nonviolent means of change, which will lead to an increased motivation to learn and practice these nonviolent methods.
I believe that the experience of trust and community is very rare for most people who are in prison (and most people out of prison in the U.S.), and that an experience of a trusting community will give people an understanding of the benefits of cooperation and openness. This increased understanding will lead to participants seeking trusting, open community in other places in their lives.
I believe that, in addition to skills and techniques, conflict is positively transformed by a spirit of love and creativity that is beyond control or predictability. I believe that certain attitudes and actions can cultivate an openness to this spirit--and that a direct experience of this tranformative spirit is the most powerful means of spreading peace. Thus, creating a workshop where people can experience this spirit is a primary goal.
I believe that if residents and staff at a correctional facility have a shared experience of community and learn conflict resolution skills, the amount of mutual respect will increase in the facility.
I believe that society-wide change comes from a complex web of influences, but that individual skill in communication, problem solving, cooperation, and affirmation is an essential ingredient in positive social change.
I could keep listing my theories, but I will stop there. I don't entirely believe each of the theories--but I know that they are motivating theories that shape how I approach this kind of workshop. Writing them out, I am aware of additional, relevant, theories of change that I intellectually believe in, but which don't seem very integrated into my practice in the workshops. A couple examples of these neglected theories include:
Personal transformation for people who have committed a crime involves learning about and acknowledging the impacts of the crime on the victims. Personal healing for the offender requires an acknowledgment of this harm and also requires actions by the offender to repair these harms.
More than anything else, violence is a by-product of social and economic injustice. To meaningfully reduce violence, justice must be increased.
I think that both my active and inactive theories of change have implications for how we evaluation in workshops such as this one at the JDC. For myself, I would like to narrow down my list of underlying theories, and have a few that are consciously in my mind throughout a workshop. I would want to translate these theories into goals that I make explicit to the participants and co-trainers. With these goals more clearly stated, it would be easier to reflect on ways we are and aren't meeting them. I would also expect my own understanding of these theories and goals to evolve with discussion and evaluation.
I think some of my theories of change point to methods of evaluation that we didn't do in this JDC workshop. If I believe that workshop will help reduce recidivism, that points to quantitative measurement of recidivism of participants. I don't know how possible it would be to track this information, but it is at least an option to consider. Likewise, if I believe that reducing violence is a primary objective, there might be some ways to track levels of violence, at least within the facility. And, if I hope that the workshop leads to changes in the atmosphere at the JDC facility, this could be a follow-up question not only for participants, but for JDC staff who did not participate in the workshop.
Much of the evaluation we have done informs these theories to a limited degree. In the written evaluation which participants filled out at the end of the workshop, we got some sense of skills that they felt were useful. We don't yet know if those skills were put into much use outside of the workshop. Follow up interviews would further inform this.
I believe that my underlying theories also point toward a need for partnerships with others to carry out these theories. The JDC staff, the participants families, neighborhoods, and schools are larger factors than this workshop in influencing any of the changes I am wishing for. For instance, the JDC staff have much more influence over the day to day atmosphere of the facility than our workshop will. How could we do the workshop in a way that was more aware of those necessarily partnerships? As a start, I hope to have more discussions with JDC staff and my co-trainers about these theories of change.
Evaluating Systemic Change
Another level to training evaluation that I am very interested in is noticing the relationship between the training and ongoing macro-level social change--changes in the structures and patterns of institutions and the society as a large. To me, this category relates to two of my evaluation wishes that I asked myself at the beginning of this paper:
I want to know how we can do workshops at this center that will contribute toward a transformed version of juvenile justice, that focuses more on restoration and less on retribution. I want to do this workshop in a way that is a part of the undoing of structural, systemic violence in our country, such as racism.
These layers of evaluation are the most difficult for me to wrap myself around--but they are questions that I don't want to lose. Although this training focused on the micro-level skills of interpersonal conflict transformation, I believe that it is irresponsible to ignore the structural realities that influence conflict and that set the context for the workshop. As Paulo Freire emphasizes, no education can be politically neutral. The way we teach can have an influence both on maintaining oppressive systems and/or on dismantling oppression.
There are reasons why a highly disproportionate percentage amount of people of color are incarcerated in this country that go well beyond the personal attitudes and skills of individuals. I am also convinced that there are ways that inmates in the criminal justice system are treated which strongly perpetuate crime and injustice. As Geoff and I go into a correctional facility which embodies systems of punishment and racism, how do we avoid colluding with those unjust forces? The JDC probably won't let us in the door if we make our workshop about resistance to authority. At the same time, many JDC staff probably also share a concern for helping build a just, restorative criminal justice system and society.
In the workshop, we did pay a minimal amount of attention to root causes of violence and our visions for a just society. We also led the workshop in a highly elicitive way, drawing on the experience and input of all the participants. We also have the long term aspiration of building in some type of community service into the program, which helps participants find ways to address root causes of violence in their communities. But I would like the interconnections between social change and these workshops to be clearer.
I can imagine a few ways these aspirations might influence our evaluation--but these aspirations mostly are questions that I hope to continue to ask and live. Marie Dugan's concept of "sub-system" change is helpful to me in this dilemma. In her "Nested Paradigm of Conflict Foci," Dugan distinguishes between the immediate issue, the involved relationship, the sub-system, and the wider system. While undoing the entire system of racism within the U.S. is one of my wishes, I can narrow that down a few steps by thinking about the sub-system of racism within the JDC or within Winchester. If the workshop has a goal of reducing racial discrimination in one of those sub-systems, it feels more do-able and measurable to me. Evaluation of these goals might require further investigation of the ways racism currently plays out in those systems. From that research, certain indicators could be selected that would reflect on how that system of racism is changing. For instance, one indicator of racism in the JDC might be the frequency people say racial slurs in the facility. This indicator might be hard to quantify--but we might ask about people's perceptions of this in follow up interviews. If we took this goal and evaluation of it seriously, that would change the way we do workshops.
Another related concept that helps me think of evaluating influences on social changes is John Paul Lederach's "Nested Paradigm: The Time Dimension in Peacebuilding" which Lederach describes in his book Building Peace. This framework builds upon Dugan's distinctions between layers of response, and adds the dimension of long-term thinking to peacebuilding interventions. In Lederach's model, the immediate action (2-6 months), short-range planning (1-2 years), decade thinking (5-10 years), and generational vision (20+years). This framework helps me remember that any deep social change on the levels that I am wishing for will take many years to happen. A three day workshop is a very small drop in that ocean. If we start with a vision for what we want 20 years from now, how will that affect what we do in one weekend training? This question assumes that we, as trainers and as peacebuilding organizations, have a guiding "generational vision." I long to embed a weekend workshop within a generational vision, but recognize that I do very little of that right now.
If we were doing evaluation based on changes that will take 20 years (or even 1 or 2 years) to happen, that would change a lot of what we do. This long-term evaluation might lead to develop and work within the context longer-term relationships. There are prisons where Alternatives to Violence Project workshops have been happening every month for the past 20 years, and an evaluation of the impact the program has had over that 20 years would be fascinating to me. To my knowledge, no evaluations like this have happened within this program. Even though, Geoff and I are just starting the program at this JDC, I would like to see us evaluate our initial efforts with that 20 year lense in mind. If we imagine doing an evaluation of the program 20 years from now, what outcomes would we like to see? What indicators can we notice this year that will help us know that we are moving toward that 20 year vision? I think these questions point us toward the advantages of embedding this training within a larger, interconnected movement and framework for social change. One such movement within the criminal justice system is Restorative Justice. I have written another paper thinking through what this kind of workshop would look like if it were more closely embedded in a wider vision for social and personal change that is based in Restorative Justice principles (http://home.planetcomm.net/bischoff/rj.html).
I realize that this paper asks more questions than it answers. Strategic, long term evaluation of training is new to me, and mostly foreign to others connected with the Alternatives to Violence Project. I hope that these reflections on the evaluation process will lead me and others to clarifying the intent of our trainings, allowing evaluation to transform the way we lead trainings, and also help connect us more intimately connect our work with social change.
Resources
Bischoff, Michael. "How Restorative is the Alternatives to Violence Project?" http://home.planetcomm.net/bischoff/rj.html . 2001.
Dugan, Marie, "A Nested Theory of Conflict," Women in Leadership 1, no. 1. Summer 1996.
Freire, Paulo. Pedagogy of the Oppressed. Continuum, New York. 1970, 1993.
Lederach, John Paul. Building Peace: Sustainable Reconciliation in Divided Societies. United States Institute of Peace: Washington D.C, 1997.
Lederach, John Paul. Preparing for Peace: Conflict Transformation Across Cultures. Syracuse: Syracuse University Press, 1995.
Weiss, Carol. "Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families." In New Approaches to Evaluating Community Initiatives, edited by James P. Connell, Anne C. Kubish, Lisbeth B. Schott, and Carol Weiss. Washington D.C.: Aspen Institute, 1995.