Chapter 5

Adoption of Methods and Tools: An International Survey

Written with Matteo Kalchschmidt

Both academic literature and managerial experience highlight that the proper use of specific methodologies and tools is critical to manage projects successfully. The general suggestion given to project managers is that the effort in implementing proper methodologies is justified by the benefits achieved by these tools, and significant evidence have been provided in different managerial fields. Certification bodies, managerial and academic literature, and general practice support that companies willing to succeed in managing projects need to be complaint with certain rules and practices. The empirical evidence of these considerations is vast and diffused.

However, there is limited empirical evidence in support of a similar impact when the role of project management is narrowed to the case of ID projects. In particular, there is a significant lack of structured evidence concerning the adoption and impact of project management methods and tools in this field. As previously highlighted, ID projects have specific peculiarities that led to the development of dedicated methodologies (i.e., Logical Framework). ID projects also face specific challenges concerning, for example, how objectives are defined and managed, how stakeholders are involved and how priorities are set. These peculiarities and the specific characteristics of NGOs make questioning the role of project management methodologies non-trivial in this case.

In order to provide an empirical analysis of the developments in the field concerning these aspects, an international survey was designed and administered. The research involved project managers of NGOs operating in the international cooperation field and belonging to different continents and areas of application. The survey aimed to analyze how project managers working for NGOs manage their projects, focusing on the adoption of methodologies and tools.

In particular the key questions this empirical analysis aimed to answer are as follows:

What is the extent of application of project management methodologies and tools in ID projects?

As of today, it is not clear the extent to which project managers involved in ID projects apply structured and formal methodologies for managing their projects. Attention has been paid both to common and widespread tools (e.g., Gantt charts) and to specific methodologies and guidelines (e.g., PM4DEV, PM4NGO). Specific interest has been paid to the adoption and role of the logical framework.

What is the performance achieved by project managers in ID projects?

We aim at evaluating the extent to which ID projects achieve both their strategic goals (i.e., the reasons why a project is initiated) and specific goals (i.e., what the project has to achieve in terms of scope, time, and cost).

Which methodologies and tools are typically associated with successful ID projects?

Projects can perform as planned or face problems, delays, and extra costs. Projects can reach their goals (e.g., improvement in quality of life), or they can have no effect. A key concern is what makes projects successful. We aim at identifying which practices are commonly applied by those project managers who are capable of achieving their goals and thus being successful.

5.1 Methodology

In order to answer to the previous questions, a questionnaire was designed and administered to different NGOs belonging to different countries and contexts. The questionnaire was designed in order to evaluate what NGOs are doing and which results they are achieving. Specifically, the questionnaire was structured in three sections (see Appendix 4):

• Section A - General information. This part was aimed at describing the NGO in terms of size, number of projects managed, and organization. Attention was also devoted to the kind of projects managed and the context in which project are developed (e.g., variability, uncertainty, long-term impact).
• Section B - Methodologies, tools, and results. This part was aimed to assess the extent of knowledge and application of the main methodologies and tools commonly applied in project management. Results were also assed with specific attention both to project performance and strategic (i.e., long term) performance.
• Section C - Logical framework. The last section is devoted to the logical framework. A specific focus was given to this methodology, since it is commonly adopted in ID projects. Specifically, attention was given both to the extent of use of this tool and its role during the development of the project. We also collected information concerning the role of the project manager, NGO staff, the donors and the local community in its application.

In order to define a proper sample of NGOs for the purpose of this study, contacts were made at different levels according to a pyramidal approach. Project managers operating in NGOs were contacted directly by using information provided by public databases and associations of NGOs. Contacts also with NGOs’ networks were very important to increase the size of the sample and also to avoid sample biases. NGOs were contacted by email and phone when possible in order to increase the response rate and to obtain a more significant sample. Contacts were searched and managed globally to avoid limiting our considerations to a specific area or country.

In the end, 496 NGOs provided useful information for the purpose of this research.

The survey was administered globally, thus the NGOs sample is made of companies belonging to all continents (see Figure 5.1). The vast majority of the sample is made of European NGOs (40%), but almost half of the sample is made of NGOs that operate in Africa and Asia. A significant number of respondents (11%) belong to North and South America.

The sample is well distributed both in terms of total income and in terms of people employed (see Figure 5.2 and Figure 5.3). The total income can differ significantly among the NGOs (see Figure 5.2): 30% of the sample have an income higher than US$1,000,000 per year, while, on the other side, almost 40% of the sample has an income lower than US$100,000.

The majority (57%) of the sample is made up of small organizations (i.e., less than 20 workers, see Figure 5.3). Only 5% of the organizations employ more than 500 workers. Thus, the sample is characterized mainly by small organizations, but still these differences highlight that the structure of NGOs can vary significantly.

The reasons for these differences can be several. The country in which they operate can explain part of it: European NGOs (on average, 21 employees) appear to be smaller compared to the overall sample (on average, 166 employees), while Asian and American ones are, in general, larger (on average, 380 and 283 employees, respectively). Also to the specific nature of the projects they manage can influence: NGOs focused mainly on projects for the delivery and integration of products tend to show very different sizes, from very small (i.e., 5–10 employees) to very large (i.e., more than 500 employees), while NGOs that manage predominantly projects aimed at raising awareness are small (the median of our sample is 14 employees).

Moving to the projects managed by interviewed project managers in the last two years, Figure 5.4 shows that the vast majority (72%) of the project managers managed very few projects (less than 10). However the project duration varies significantly (see Figure 5.5). In fact, 25% of the interviewed project managers declared that their projects can last more than two years, while almost 50% of the project managers typically face projects that last less than one year. The nature of the projects explains these differences. Projects aimed at delivering products tend to last less (on average 15 months), compared to construction projects and delivery of services projects (both on average 23 months). Furthermore, a significant majority of projects (63%) have a relatively small size, less than US$200,000 (see Figure 5.6). Here the different project categories show a different picture. In fact, delivery of products, construction, and development of services projects show a relevant variability in size, compared to projects for rising awareness that tend to be smaller on average (50% of these projects are worth less than US$100,000).

In terms of project aims, projects have been classified in five main categories:

• Projects for delivery and integration of products and tools: Here we consider projects aiming at delivering and integrating products and tools, typically to support local communities.
• Projects to develop services: Here we classify projects aiming at developing and providing or improving services such as health, education, and financial aid.
• Projects to construct buildings: Here we consider projects aiming at building infrastructures such as bridges, roads, or wells.
• Awareness-raising projects: Here we aggregate projects aimed at promoting particular issues (e.g., child labor, HIV, global warming) and alerting about their consequences.
• Other projects.

According to this classification, almost 60% of the NGOs are focused mainly on projects to develop services, while 23% of the organizations have operated in the awareness raising field (see Figure 5.7). The sample is characterized by a smaller presence of NGOs aimed at delivery and integration of tools and construction of infrastructures (in total 7% of the sample).

5.2 Project Management Methods and Tools

Different methods and tools can be applied in order to manage projects. Managerial practices and academic research have developed a wide array of solutions that can be applied in order to properly plan the project and accurately control its development. Moreover, over the years several institutions have defined specific guidelines on how projects should be managed.

Here specific attention has been paid to the following tools (see appendix 1):

• Logical Framework
• Gantt chart
• Critical Path Method
• Earned Value Management System
• Work Breakdown Structure
• Responsibility Assignment Matrix
• Stakeholder Analysis Matrix
• Organizational chart or Organizational Breakdown Structure
• Milestone Schedule
• Progress report
• Cost Accounting
• Risk Analysis
• Contingency Reserve
• Communication Plan
• Issue Log
• Scope Management

In terms of guidelines, we have considered the ones developed by the following organizations (see appendix 2):

• Project Cycle Management (PCM)
• Project Management Institute (PMI)
• Project Management For Development Organizations (PM4DEV)
• Project Management for Non-Governmental Organizations (PM4NGOs)
• International Project Management Association (IPMA)
• PRINCE2

The collected information allows a deep look into the practices adopted by NGOs concerning the use of tools and methods typically designed for managing projects.

Different guidelines exist in the field of project management, but respondents are only marginally aware of them. The analysis shows that the Project Cycle Management (PCM) is widely adopted: almost 85% of the NGOs use this guideline, and almost 30% uses it heavily (i.e. for more than 75% of their projects). This result makes the PCM the mostly adopted methodology in this field (see Figure 5.8). Methodologies and frameworks such as the PMBOK® Guide, PM4DEV, PM4NGOs, IPMA and PRINCE2 are adopted only by around 30% of the considered NGOs. We can thus state that essentially PCM is the only reference currently considered by all the NGOs. This result can be partially explained by considering that some of these guidelines don’t have a specific focus to this field (i.e. the PMBOK® Guide, IPMA, PRINCE2) and by the fact that PM4DEV and PM4NGO have been introduced only recently.

Further analyses show that, while the adoption of the PCM is uniform among the NGOs, the adoption of structured methodologies (e.g., PM4DEV, PM4NGOs) is significantly different among the organizations. In fact, larger NGOs, in terms of number of workers, employ more extensively structured methodologies. This means that certain methodologies are more likely to be applied when the organization achieves a minimum critical mass. Interestingly, project size (i.e., total project cost) is not discriminatory in the adoption of the methodologies. This evidence demonstrates that large NGOs employ structured methodologies regardless of whether they have to manage a large or a small project. We can argue that large NGOs thus tend to define specific internal guidelines and ask project managers to follow them.

Finally, we found that the adoption of the considered methodologies is quite similar among NGOs belonging to different geographical areas: PM4NGO is more frequent in North American NGOs, while PM4DEV is relatively more widespread in European NGOs. Surprisingly we also found that about 85% of the North American organizations adopt other methodologies that have not been considered in our survey. These results highlight that different references exists in the different countries, but that in general there is no consensus versus a specific set of guidelines or methodologies. This makes the problem of how to manage ID projects indeed a relevant topic.

Figure 5.9 provides more details concerning the specific tools applied. We can see that there are some methods that are commonly applied by the vast majority of the NGOs (i.e., logical framework, progress report, and cost accounting). This is due to the nature of the projects that are typically managed by NGOs and that usually make mandatory the use of such tools. The logical framework is also the core of the PCM, which is the methodology mostly adopted by NGOs. Only a very small share of organizations is not aware of these tools or does not adopt them.

A second group of methods is characterized by being commonly adopted even if we cannot consider them common practice. These are risk analysis and management, communication plans, Gantt charts, milestone planning, Organizational Breakdown Structure (OBS), contingency allocation, and stakeholder matrix. All these tools are adopted by a percentage of NGOs between 65% and 80%, so they are part of the common methodologies that the vast majority of the sample adopts.

Other methods that are less commonly adopted (i.e., scope management, Responsibility Assignment Matrix - RAM -, WBS, Critical Path Method - CPM -, issue log), The Earned Value Management System (EVMS) methodology appears to be adopted only by a limited part of the sample (37%). This result is partially justified, considering the complexity of these methodologies compared to the peculiarities of the projects considered.

Further analyses show that while the adoption of common tools such as progress report and Gantt charts is uniformly distributed among NGOs, the adoption of structured tools (e.g., CPM, WBS) is significantly different among the organizations. Although the logical framework is a common tool, its use among large NGOs (i.e., 500+ workers) is slightly higher than its use among smaller organizations. Moreover, larger NGOs employ more extensively structured tools such as the WBS and CPM, while the use of structured tools among small and medium organizations is generally lower. Conversely, if we consider the project size, measured in terms of its overall cost, the considered tools are uniformly employed among the different projects. Similarly to the methodologies, this indicates that typically a large NGO adopts a wide range of different tools; these tools are then applied to all projects regardless of their size. We argue that this indicates that large NGOs tend to define specific guidelines and try to organize in a structured way how to manage the projects. On the contrary, smaller organizations seem to rely more on the experience of the project manager and tend to adopt methodologies coherently with the specific requirements of each project.

Unlike what has been found for project management methodologies, the adoption of the considered tools varies significantly among the different geographical areas. Since the logical framework has been developed in the United States, it is not surprisingly that more than 95% of the North American NGOs adopt it. In the other areas, this percentage is slightly lower (85–90%). The adoption of other common tools such as progress report, cost accounting, and Gantt charts among the different geographic areas is relatively uniform. NGOs based in Oceania adopt extensively a wider range of tools, compared to the other geographical areas.

By classifying the analyzed projects according to their aims, it is interesting to observe that project managers rely on specific methodologies and tools depending on the context and, therefore, according to the intervention’s needs. Figure 5.10 shows the distribution of the considered methodologies according to the kind of projects taken into consideration.

The PCM is widely adopted in projects aiming at raising awareness, delivering and integrating products and tools, and constructing buildings and infrastructures, demonstrating that it is a methodology that fits into different non-governmental fields. Projects aiming at raising awareness and constructing building and infrastructures are more likely to be managed employing more extensively a broader range of methodologies. In general, the adoption of methodologies is rather uniformly distributed among the different aims; nevertheless, development of services projects appear to rely more on other methodologies than those that have been taken into account in this study. Overall, the management of projects aiming at delivering and integrating products and tools relies less on standardized methodologies.

A heterogeneous picture arises when techniques are taken into consideration (see Figures 5.11 to Figure 5.16). In general, the adoption of the logical framework (Figure 5.11) results high and uniform among the different projects, even if its employment is significantly higher in projects aiming at raising awareness. Time and cost planning tools are quite adopted among the different project categories, while scope management techniques are significantly less adopted. It also clearly emerges that controlling tools are not much used besides progress reports; in particular the considered project managers do essentially not adopt EVMS.

If we focus attention to the tools for time and cost planning (Figure 5.12), we can observe several differences. Cost accounting methods are highly adopted essentially by all project managers regardless of the specific application. Some projects, apparently, seem to be managed without the use of any Gantt chart, in particular projects focused on delivery and integration of products. This is likely to be due to the nature of these projects that imply often a stronger focus on delivering products rather than managing project activities. The critical path method is essentially not adopted, indicating that project managers prefer simple and intuitive tools, also due to the nature of the projects they are managing.

Scope management tools (Figure 5.13) are not much adopted. In particular several projects are managed without a structured and formalized work breakdown structure, so with a rather simple structure of the project plan.

Human resource management tools are variously adopted (Figure 5.14). Plans to manage communications are often used and also organizational charts are sometimes defined. Few applications of the responsibility assignment matrix can found.

Risk management shows a different pattern (Figure 5.15). In general, project managers declare a diffused adoption of risk analysis tools; less attention is, however, paid to the evaluation and allocation of contingencies.

In the end, control tools show a clear pattern (Figure 5.16): progress reports are adopted very frequently, while issue logs and particularly EVMS are essentially not adopted.

Finally, quite interestingly we found 20 organizations that declare to use no structured methodology and other 16 that declare to use them only in a limited way (the percentage of adoption in projects is less than 25%).

The described differences seem to be at least partially due to the nature of the projects managed by the respondent.

Projects focused on the delivery and integration of products show in general a lower attention to the adoption of the considered tools. This can be partially understood by considering the nature of some of the projects that fall within this category where scheduling, project controlling and project risk management are not a key concern. When we consider the other categories of projects, the use of the tools varies quite significantly. Some tools characterize projects focused on the construction of buildings. For example the use of the WBS is significantly higher here, due to the complex nature of these projects. Also, scheduling and accounting techniques are significantly used, due to the need of integrating different resources on different tasks during the lifecycle of the project. In general, projects aimed at raising awareness are characterized by leveraging on human resource tools, and specifically the responsibility assignment matrix, which is somehow neglected in the other kinds of projects. Also, risk management methods appear to be more adopted in these contexts.

5.3 Projects’ Results, Methodologies, and Tools Adoption

Overall, the NGOs that participated to the research declared medium-high performance (average values are all above 3 on 5, see Figure 5.17) on most of the dimensions. In particular, the compliance with budget, the compliance with quality and the satisfaction of the local community are the dimensions on which NGOs have better performance. Since we guaranteed anonymity to the respondents, we do not think that respondents declared performance indicators significantly biased. More likely, our sample is composed of a majority of best-performer NGOs. As we have seen in the previous chapter, the knowledge and adoption of the tools is quite high, both of standard tools (e.g., WBS) and specific tools (e.g., logical framework). Moreover, it is important to keep performance over a minimum threshold, otherwise the project would fail. For instance, the satisfaction of the local community is directly related to the strategic goals that activate the project, the budget compliance since it is often not negotiable and thus it is a fixed constraint that can kill the project if it is not properly managed. Looking at Figure 5.17, we notice the economic sustainability after the end of the project. This evidence indicates the difficulty in guaranteeing that the short-term effort (i.e., the project) will continue to give positive effects in the future. Indeed, the possibility of guaranteeing, in the long term, the results of the project can depend on the availability of additional funding that enables what a project achieved to be extended and maintained. This, however, is not always feasible due to the variability characterizing funding means. The long-term difficulties are also apparent considering that obtaining long-term impact is another indicator that gains (slightly) lower results.

Even if the performance is quite high, we can still differentiate among those NGOs that have higher performance (called High Performers) and lower performance (called Low Performers). This classification can be defined according to both project-specific performance and long-term performance (i.e., long-term performance). By project performance, we refer to those measures that relate directly to the outcome of the project (i.e., compliance with the expected time, budget, and quality). Long-term performance refers to the impact in the long term of the project’s results (i.e., economic sustainability after the end of the project, ownership expansion to the local community, and the long-term project impact). Statistical analysis shows that there is a strong correlation between the different performance measures (all Pearson’s correlation coefficient are significant and higher than 0.30) and specifically between project performance and long-term performance. This evidence shows that NGOs that obtain high project performance also obtain high strategic performance.

In conclusion, we classified NGOs as high or low performers according to the average of both project and strategic results. Figure 5.18 provides the distribution of the sample according to this classification.

The respondents regarded as high performers (229 in total) are the ones that declare results above the average; while low performers (206 in total) are the ones that declare results under the average. There is only a small share of respondents that register high performance only on one of the two aspects, since, as previously mentioned, strategic and project results are strongly correlated.

Based on the described classification, we compare project management practices within these two groups. Figure 5.19 shows the different levels of adoption of the project management methodologies considered here for the two categories (results are provided based on the project performance classification only for briefness sake). The NGOs that register higher performances are more likely to manage their projects relying extensively on a wider range of methodologies and tools. The essential difference from this point of view relies only in the extent of adoption of more of these methodologies by high performers.

Figure 5.20 compares high and low performers on the use of project management tools. It can be noticed that the use of Gantt charts and progress reports is similar in both groups indicating that its use is common practices in NGOs. Differences, instead, appear for all the other tools, where high performers declare a stronger and more diffused use of such methodologies. The stronger differences can be found on the use of EVMS, RAM and scope management tools, which have been previously identified as some of the less-adopted methods on average. The gap between high and low performer tends to be smaller regarding well-known tools, while it becomes more significant when less-adopted techniques are considered. This indicates that high performers have a higher adoption of more complex tools, while limited difference can be found on simple methods.

5.4 Project Managers’ Evaluation of the Logical Framework (LF)

Within the research, particular importance is given to the LF. This tool is the most widely known and used inside the PCM framework. As described in Chapter 3, the LF is a 4x4 matrix that summarizes the project’s goals, activities, assumption, indicators, and sources of verification in order to measure and report the achievement of objectives.

Since the LF is widely employed, it is interesting to investigate how this tool is evaluated among the NGOs’ project managers and how extensively this tool satisfies the users’ expectations. First of all, NGOs were asked to provide indication of who is actively involved in the LF management (see Figure 5.21). It emerges clearly that the project manager is the most involved figure (more than 80% of sample declare the direct involvement of the project manager). However, the NGO staff also plays a significant role in supporting project managers in fulfilling this task (Figure 5.22).

The project’s donors are more likely to subscribe and approve the LF (see Figure 5.23), even if this is not a common practice (less than 60% of the sample asks donors to subscribe or approve the LF). Quite interestingly, the involvement of the local community is rather limited in any form.

Concerning the benefits of the use of the LF (Figure 5.24), it is interesting to observe that more than 60% of the project managers declare that they are the ones who benefit the most from the use of the LF, therefore they should be more than willing to adopt it. Nevertheless, it is significant that also the local communities benefit indirectly from the use of the LF, although they are not involved either in filling or subscribing/approving the LF. This suggests that the use of the LF has indeed a positive effect also on the final stakeholders and that a stronger involvement of them would likely contribute to better project performance and stronger success.

Overall, the LF evaluation, as well as the satisfaction related to its usage, is relatively high (positive average values are all above 3 which can be considered a reference point, see Figure 5.24). An important share of the project managers considers the LF an important tool for managing projects successfully, and relatively easy to be integrated with other project management tools. Moreover, the LF is reputed as moderately easy to use, even if some project managers say that the LF might be too simple and omits vital aspects of a project.

5.5 Conclusions

This research provides empirical evidence on several aspects of the project management in NGOs.

First of all, the use and knowledge of methodologies is concentrated on PCM and very limited knowledge of other guidelines can be found. This result is true for both high and low performers, suggesting that standard methodologies are not a prerequisite for superior performance as could have been expected.

A second consideration regards the use of project management tools. While some tools are significantly adopted (e.g., Gantt charts, progress report), others appear to be somehow neglected (e.g., WBS, RAM). The results highlight that NGOs are more likely to adopt simple techniques rather than focusing their attention on more structured and analytical methodologies. This consideration is not true, however, for all the considered NGOs: high performers, in fact, are more likely to adopt all the considered tools, compared to low performers. This indicates those organizations that invested more on these methods were capable to improve their performance, thus we argue that it is important to increase both the awareness and the knowledge of these tools among NGOs. In fact, the difference between the adoption of methodologies and tools of high versus low performers tends to be negligible when well-known techniques are considered, and it becomes more significant when less-adopted methods are taken into account. Therefore, we can conclude by arguing that a positive correlation exists between project management techniques employed and performances achieved.

A special consideration concerns the use and role of the logical framework. Most of the organizations devote their attention essentially to the logical framework. However, this tool, as reported by the literature and the guides analyzed in Chapter 4, is a tool to strategically frame a project, rather than a planning and controlling tool. Therefore, project managers should be trained on a broader set of tools, and further research is probably needed to integrate the logical framework with the other tools. Different elements in the end appear also to influence the adoption of standardized methodologies and tools among the considered NGOs. Their behavior varies significantly according to the context in which they operate and according to the possibility of properly structuring their organization. In some cases, NGOs operate in poorly structured contexts where involved people often don’t have a managerial background. This makes the adoption of structured methodologies more difficult and thus makes “standard” techniques less applicable. This consideration is particularly true in those contexts where organizations tend to be small and for those organizations that rarely manage complex projects.