My work & school life… what I think about all day
For the past 15 months my blog has been on from the heart and about my unfolding inner and spiritual life. By day I am a PhD student in Evaluation Studies. And my part time job is as an evaluator for the Minnesota Evaluation Studies Institute (MESI). What is an evaluator? Great question! An evaluator can determine the strength and weakness of a program or organization, can establish objectives for it, pinpoint what is missing or needed, develop criteria for how to achieve the goal, and then track the progress being made. I tend to work with organizations that work with youth and have some sort of social justice work inherent in their mission. I like to involve the people from the organization being evaluated in the evaluation process so that when the evaluation is over they have more capacity for internal evaluation than before the evaluation started.
There is often too much of a disconnect between these two worlds. So just for fun, I thought I’d share with you the kinds of things I think about at school and at work.
I am interested in how one evaluates change in complex systems by using the developmental evaluation framework presented by Michael Quinn Patton. In order to do so, one must understand complexity and/or systems theory. I don’t know enough about the two theories to distinguish between them in a way that will help my craft my research question. I do, however, know that two books will be central to my thesis:
- Developmental evaluation: Applying complexity concepts to enhance innovation and use (2010) by Michael Quinn Patton
- Systems concepts in action: A practitioner’s toolkit (2011) by Bob Williams.
I wanted to know how other people were using these texts in their work so I performed an interdisciplinary literature review.
Step One: I did a search for each of the books and then I search for all of the journal articles and books that cited each of these two books. I found 57 sources that cited that Patton book and 9 sources that cited the Williams book.
Step Two: I read the abstracts for each of these sources and ultimately selected 17 sources that would help me understand the relationship between systems and/or complexity theory and a developmental evaluation framework.
Step Three: I imported each of these sources and their corresponding abstracts into RefWorks and saved them in a folder called ‘Developmental Evaluation’. I was then able to use this folder to export the citations and abstracts that follow in this paper.
Â
Interesting Findings
- The original article about Developmental Evaluation was published by Patton in 1994. In the following decade, from 1994-2004, people only cited the article 49 times. Whereas his most recent book about Developmental Evaluation, published in 2010 has already been cited 57 times. This supports my hypothesis that there is a growing recognition for the role that Developmental Evaluation in complex settings.
- Most of the articles are in the fields of social innovation and public health. Not surprisingly, these are two fields that work in very complex arenas and that receive a great deal of funding.
- All of the articles that described a government intervention were published in countries other than the United Students. The United States governments still see Formative and Summative Evaluation as the only two valid frameworks.
- There were very few transferrable models for how one would do Developmental Evaluation using complexity or systems theory.
References
Campbell, D. T. (2011). Assessing the impact of planned social change. Journal of MultiDisciplinary Evaluation, 7(15), 41.
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199.
Faber, A., & Alkemade, F. (2011). Success or failure of sustainability transition policies. A framework for the evaluation and assessment of policies in complex systems. Paper presented at the Paper Presented at the DIME Final Conference, , 6 8.
Fagen, M. C., Redman, S. D., Stacks, J., Barrett, V., Thullen, B., Altenor, S., & Neiger, B. L. (2011). Developmental evaluation. Health Promotion Practice, 12(5), 645-650.
Foster-Fishman, P. G., & Watson, E. R.The ABLe change framework: A conceptual and methodological tool for promoting systems change. American Journal of Community Psychology, , 1-14.
Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models Jossey-Bass.
Guidel, A. P. (2010). Methods brief: Evaluating system change: A planning guide
Hawe, P., Bond, L., & Butler, H. (2009). Knowledge theories can inform evaluation practice: What can a complexity lens add? New Directions for Evaluation, 2009(124), 89-100.
Huddart, S. (2010). Patterns, principles, and practices in social innovation. The Philanthropist, 23(3)
Hummelbrunner, R. (2011). Systems thinking and evaluation. Evaluation, 17(4), 395-403.
Jones, H. (2011). Taking responsibility for complexity. Paper presented at the Australasian Evaluation Society International.
Jones, H. (2011). Taking responsibility for complexity: How implementation can achieve results in the face of complex problems. No. Working Paper 330).Overseas Development Institute.
McGeary, J.How complexity influences evaluation.
Moore, M. L., & Westley, F. R. (2011). Public sector policy and strategies for facilitating social innovation. Innovation,
Norman, C. D., Charnawâ€Burger, J., Yip, A. L., Saad, S., & Lombardo, C. (2010). Designing health innovation networks using complexity science and systems thinking: The CoNEKTR model. Journal of Evaluation in Clinical Practice, 16(5), 1016-1023.
Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311-319.
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. The Guilford Press.
Pell, E. (2006). Relationships matter: How agencies can support family and social network development. Annie E. Casey Foundation.
Tremblay, M. C., & Richard, L. (2011). Complexity: A potential paradigm for a health promotion discipline. Health Promotion International,
Norman, C. D., Charnawâ€Burger, J., Yip, A. L., Saad, S., & Lombardo, C. (2010). Designing health innovation networks using complexity science and systems thinking: The CoNEKTR model. Journal of Evaluation in Clinical Practice, 16(5), 1016-1023.
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use The Guilford Press.
Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311-319.
Pell, E. (2006). Relationships matter: How agencies can support family and social network development. Annie E. Casey Foundation.
Tremblay, M. C., & Richard, L. (2011). Complexity: A potential paradigm for a health promotion discipline. Health Promotion International.
Williams, B., & Hummelbrunner, R. (2010). Systems concepts in action: A practitioner’s toolkit. Stanford Business Books.