Monitoring and evaluating communities of practice

We did a research for the IKMemergent program on monitoring and evaluating knowledge management strategies. The full publication will be up soon, but I already discussed the results in a teleconference in CPsquare and had a nice discussion how this relates to assessing results of communities of practice.

There are quite some challenges to overcome in monitoring and evaluating the effect of knowledge management strategies and interventies.

- One of the challenges is dealing with the time lag between the intervention and the impact. Sometimes benefits surface quite rapidly. We once had a new member who picked up an idea for elearning in the community, went back to her organisation and started implemented it organisation-wide. In other cases, ideas need greenhousing and it takes time for things to materialize or practices to change. On the other hand, the sponsors or donors may be eager to see results.

- Another challenge is quantifying the unquantifiable. You can measure whatever you can measure and use whatever data you have available, but you need stories to understand reality and how the community may have influenced the figures. You collect stories to understand the relationship between the figures: “this is where I took the idea and this is how I implemented it”. This will also help to prevent story-telling turning into success story telling.

- Yet anther challenge is to match inherent evaluation goals with extractive goals. An inherent evaluation is for people involved in the community or other effort, an extractive evaluation is for outsiders. At times a donor or manager starts asking for an evaluation at the time when doubt about the effectiveness has arisen. This is not the best time for an honest and open process. The question to be asked then is : does an assessment of results in itself imply a lack of trust in the leaders or the strategy?

In the end we realized that these dilemmas can not be solved by choosing a good method. It is the decision-making about whether or not to monitor and evaluate and ho wit will be done that matters. Therefore we wrote down 12 decisions that need to be taken in the form of answers. Hopefully that will guide people who are starting up a new initiative to improve knowledge management leading to innovation. You will find the 12 questions in the presentation. You will find the slides below.

Monitoring and evaluating knowledge management strategies

View more presentations from Joitske Hulsebosch.