Distillation of „experience based good practices“

September 06, 2011 | Adrian Gnägi | SDC Experiences |


Rating: none


by Kuno Schläfli, Romana Tedeschi, Katharina Walker, Michael Reimann, Matthias Boss, and Adrian Gnägi

SDC used to be structured as a matrix organization. Operational geographic units managed funds and local context, technical units managed thematic knowledge, and together they were thought to implement effective projects. This setup became perceived to have improvement potential. In 2008, technical units were replaced by learning and exchange networks. One of the justifications for this move was that guidance and policies elaborated by the technical units were sometimes perceived to be too abstract, too general, not enough evidence based. One of the expectations towards the newly created networks therefore was that their guidance should look, feel, and act differently – “experience based good practice” was the orientation received. This post documents one of the first attempts by one of the new networks to distill “experience based good practice”.

SDC’s “decentralization and local governance network” (dlgn) met for its first-ever face-to-face encounter in November 2009 in Delhi.The emerging dlgn community there agreed, among other things, that there was merit in reflecting on the ways we provide financial support to municipalities. SDC’s Knowledge and Learning Processes Division (KLP) and dlgn then agreed to partner on a learning project with a triple objective:

  • Distillation of experience based good practice on donor support for sustainable local government finances
  • Development and pilot testing of a methodology to distill experience based good practice
  • Development and pilot testing of a multi-media electronic presentation for experience-based good practice.

Some years earlier, KLP had developed methodological guidance for experience capitalization and experience documentation. A first sketch for adapting this methodology to e-collaboration was also available. On this basis, a tender document for consultant support to the learning project was drawn up. Consultants were invited to present offers, with 3 options:

  1. Elaboration of a document summing up the international state-of-the-art in donor support for sustainable local government finances, including an analytical framework to document SDC’s own experience
  2. Possibly plus a methodology for and support to document SDC’s own experience
  3. Possibly plus a concept and implementation capacities to present the project results innovatively on an electronic multi-media platform.

The tender was received with interest by the consultant industry. In the end, 7 fully documented offers were available for selection, several of them covering all 3 options. A consortium by KEK-CDC Consultants and LerNetz was awarded the contract and a timetable for the project was jointly developed.

The project was planned with four phases (see process matrix):

  1. Elaboration of a state-of-the-art report on international experience with donor support to sustainable local government finances
  2. Documentation of SDC’s experience through case studies
  3. Comparison of state-of-the-art and SDC experience, elaboration of recommendations for improvements in SDC’s practice
  4. Validation of results and recommendations during the second dlgn f2f event

Later on, and for reasons explained below, a fifth phase was added:

    5. e-discussion campaign on results and recommendations (scheduled between phase 3 and 4)

One of the major difficulties encountered early on in project implementation, but then present until the end of the project, was participation by experience carriers in the field. Several factors were involved:

  • Dlgn members had shown vivid interest to participate in a total number of 11 learning projects during the f2f event in Delhi, but they of course later on had to seek approval by their Country Directors to dedicate time for the projects. With the networks in their early stage of development and not delivering services to operations yet, investment of resources by operations into network activities was not always obvious.
  • Some dlgn members had volunteered to participate in several learning projects. There was a potential for competition over scare human resources between the different learning projects that had to be handled carefully by the Country Directors and by the dlgn Focal Point. Several country teams that initially were interested to participate, and that could have provided valuable experience, withdrew from the project for this reason.
  • Probably all dlgn members who participated in the project had time constraints due to other urgent tasks (business trips, report deadlines, workshops etc.) during at least part of project implementation.

On the positive side of participation by field staff, the project met with high interest by some staff of partner organizations. This is especially true for Intercooperation staff, who in the end may have contributed nearly half of field staff time put into the project.

The elaboration of the state-of-the-art report by KEK went smoothly. Simultaneously, KEK elaborated a concept and a template for case studies to document SDC’s experience (taking into account the massive contribution by Intercooperation, it would be more precise to speak of SDC funded experience). The main trade-off decision with the case study concept was whether each case study should document the experience on all 5 major content areas proposed by KEK in the state-of-the-art report, or whether they should focus on one area only. In order to limit the workload of field staff, it was decided to have one content area per case study only.

The elaboration of case studies was done in a 3-step process:

  1. Country teams drafted their case studies (in English, French, or Spanish)
  2. The drafts were reviewed by peers (first in tandems by other country teams, then by all participants) and afterwards revised by the country teams
  3. The revised drafts were reviewed by experts (KEK, SDC, others), and the final texts translated into English.

As the internal review later showed, the case study process was perceived overwhelmingly as a great learning experience by participants, but also as a rather painful process. Some of the difficulties encountered were:

  • By the moderator: The different speed of work by the country teams and the meeting of deadlines. The moderator thus had a double function: push the country teams in order to ensure the ambitious time schedule was respected and coach and motivate the country teams with a more pronounced need of support.
  • By the drafting teams: The drafting teams were asked to compare their experience with the findings in state-of-the-art report, and to reflect on this comparison. This was not an easy task, the first draft of case studies hardly contained such reflection. The requirement to focus their experience on one content area posed another difficulty. Furthermore, the quote of a participant of the face-to-face meeting in Sarajevo “field staffs are not necessarily good editors” points out how challenging it is to analyze one’s broad experience and write it up in form of a case-study.
  • By the tandem country teams peer reviewers (step 2): As a result of the different paces of work, some of the country teams were asked to provide feedback on another case study, even though they had not yet finished their own. The quick teams had to wait for the peer review whereas the other teams were facing a double pressure of drafting their own case study and giving feedback to another one. Due to language issues the case studies written in French and Spanish were reviewed by a small number of other country teams only.
  • By the expert reviewers (step 3): Focus and density of the eight case studies differed considerably. Sometimes it was necessary to ask back whether there really was no experience on certain issues, or whether the case study just did not mention it. Not all case studies focused closely on the agreed upon content area. In one case study, for example, there was more information on capacity development than on donor grants, which was the agreed upon topic.

The comparison of the state-of-the-art with SDC experience led to a crisis moment in the project: the comparison drawn up by KEK was nicely worded, but it basically said that SDC’s experience was limited (experience in 3 out of 5 identified content areas only) and rather old fashioned. Regarding the first point, we realized that a triple selection process had excluded much of SDC’s experience from the project: some country teams with relevant experience had not shown interest in the project in the first place, some others withdrew due to the time constraints mentioned above, and much of the experience was excluded when it was decided that case studies should cover one content area only. The second point – old fashioned approach – left project owners irritated: SDC seemed to work mostly outside the national systems of intergovernmental fund transfers and capacity development processes.

It was at this moment that the e-discussion campaign was added as fifth phase to the project. Through the 3-week discussion of the main results and recommendations in the dlgn electronic exchange forum we hoped to capture the hitherto uncovered experience, and we wanted to understand why SDC’s standard practice was supposed to be old fashioned. The experience with the e-discussion campaign has been documented in an earlier post on this blog. The e-discussion did allow capturing, although in a more rudimentary way than with the case studies, some additional SDC experience. It showed that SDC in fact has (limited) experience in the other two content domains, support for intergovernmental fund transfer schemes and support for borrowing by local governments. Field staff also explained the valid reasons they had to work mostly through parallel projects, and not directly supporting national initiatives.

 The last point turned out to be the main discussion topic during the 3 consecutive workshops held on the project during the second dlgn f2f meeting in Sarajevo in March 2011. We learned that in many partner countries no national initiatives exist that SDC could align with, or that fiduciary risks would be too high for SDC participation. There turned out to be a second reason for SDC’s frequent stand alone approach, though: pride and identity through quality requirements and value orientation. SDC seems to be working outside the national systems frequently since our values (social inclusion, participation, democratic procedures etc.) and our quality requirements hamper acceptance of initiatives that do not correspond to what is perceived as Swiss benchmark.

 The discussions in the e-forum and in Sarajevo led to substantial reformulations in recommendations of the project: SDC should strive to work more within national systems supporting local governments, but there may be valid reasons for parallel or pilot projects, in case they can influence and help to improve national initiatives.

 The KEK/Lernetz consortium had proposed the “LernBuch”, an innovative product developed by Lernetz, as multi-media platform to present project results. The development of the “Donor Support to Local Government Finances LernBuch” started towards the end of the case-study elaboration phase. A test version was presented to dlgn members during the Sarajevo f2f. The final version went life in late July 2011. Main lessons learnt during the process were:

  • Project owners and consultants: Even though Lernetz presented examples of LernBuch developed for other clients, it was difficult to imagine what the final version would look like. This led to products being developed that later on had to be revised substantially. The first draft of the main LernBuch text, for example, was drawn up by KEK, but had to be shortened/focused radically by one of the project owners later on. Also, the final list of media to be included (which videos, which photos, which graphs etc.) was developed late in the process only, which made finding those media tedious and erratic. In the future, it would be useful to develop a kind of “maquette” of the final product before individual products are created. The resource requirements for final editing of texts, including translation, were underestimated in the budget and time planning.
  • LerNetz: From Lernetz perspective, the heterogeneity of the participants and groups involved in the project, as well as the heterogeneity of the media, texts and other material provided by them made it difficult and more time consuming than expected to compile a consistent and convincing end product. Furthermore, the production process that Lernetz normally uses to produce a Lernbuch was reversed: Most often, a Lernbuch project starts out with a more or less undisputed text that has “textbook status”. In this case, the final text of the Lernbuch was discussed and re-written until a late phase of the project. This, in turn, resulted in some uncertainty on the side of Lernetz while developing the final list of media, for which the project owners and consultants waited eagerly (see above).

 The main learning during the entire project is:

  • Case study methodology has potential for improvement: Completeness of experience coverage turned out to be a critical issue with case studies as main instrument to capture SDC experience. Having field staff drafting the case studies themselves might not be the best option: language issues, quality of translations, and final editing turned out to be major difficulties. It might be worth testing whether telephone interviews and then drafting by consultants, followed by completion and review by field staff, would not produce better case studies. A potential trade-off with reduced learning and ownership would have to be monitored carefully, though.
  • Danger of devaluating experience and practice: The comparison between the international state-of-the-art and own experience turned out to be a tricky issue: own experience looked, at least initially, rather not-so-favorably. In retrospect, this probably should have been expected: practice and experience will always look somewhat pale when compared to a state-of-the-art report (a mixture of top-notch science, PR and ideology; the normal devaluation of craftsmanship Sennett talks about). Without the discussions in the e-forum and in Sarajevo, project results would have been disappointing. We will need to think about ways to act against this inherent imbalance. But: as long as the organization is aware of this inherent imbalance, not-so-favorable findings can act as eye openers on organizational dogmas, traditions, and taboos.
  • Organizational acceptance: It is crucial that institutional representativity of experience is assured and recognized. This may be in tension with the fact that case studies depend on the voluntary engagement of network members to devote time for participation. Therefore, a balance must be stroke between time invested for each case study (the more, the better the learning effect for the authors) and satisfactory – because representative – selection of documented cases (which may be easier if the time invested “by case” is more limited). Eventually, a mixture of methodological approaches to collect and write down experiences might be envisaged.
  • Clash between “experience based” and “politically wanted” approaches: The reasons behind practices that do not correspond to what is distilled as “state-of-the-art practice” may be manifold: political imperatives, obligations for quick disbursement or results, power imbalances within operational teams, etc. It is important that these reasons are investigated into and made transparent. What is adopted in the end as “organizational good practice” is the result of a negotiation process between bottom-up experience based evidence and top-down political imperatives. We will need to reflect further how those organizational negotiation processes can be structured so that experience receives adequate recognition.
  • LernBuch as superior communication instrument: We now have a fancy new communication tool – but will it deliver? The LernBuch will need marketing to have a chance to fulfill expectations. But in the end, only experience will show whether it is actually used, whether it helps us to better transport messages. Time is not ripe yet to judge. We will need to develop metrics: how do we know the LernBuch delivers better than the traditional “SDC blue booklets”? How many visitors/day, how many page views/visit mean success?
  • Methodology allows for effective, but resource intensive learning process: Roughly 40 SDC and partner staff participated actively in the project. Participants confirmed that through this involvement they substantially deepened their understanding on how donors can effectively support sustainable local government finances, and it is assumed that ownership for recommendations has developed, too. Strain on scarce field staff time was substantial, though. A network like dlgn (130 members) probably cannot afford more than one or two such intensive learning projects per year, where participating field staff have to invest 5 days+ of their scarce working time. The total cost of the entire project is roughly 400’000 Swiss Francs (100’000 SFr. consultants, 300’000 SDC and partner staff time).
No Comments

Leave a Reply