MIKIPEDIA POLITICAL BLOG: DELPHI TECHNIQUE – DIALOGUE METHODS FOR UNDERSTANDING A PROBLEM BROADLY: INTEGRATING JUDGMENTS

Description

The Delphi technique is ‘a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem’ (Linstone and Turoff 1975:3). Furthermore, it is ‘a method for the systematic solicitation and collation of judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed with summarized information and feedback of opinions derived from earlier responses’ (Delbecq et al. 1975:10). It is used most frequently to integrate the judgments of a group of experts. A key feature of this technique, however, is that the respondents do not meet and their responses may be anonymous. We still consider it to be a dialogue method, however, because ‘conversation’ between the parties occurs, even though it is not face-to-face.
Three separate groups of actors are generally involved:

Respondent group: those whose judgments are obtained through completing the process
Staff group: those who design the initial questions, summarise the responses and prepare the questions for subsequent phases
Decision-makers: those wishing to receive a product such as a consensus position from experts, or a recommendation (adapted from Delbecq et al. 1975).

Although some flexibility exists in implementation, the core method, as described by Delbecq et al. (1975:11), is as follows:

First, the staff team in collaboration with decision makers develops an initial questionnaire and distributes it…to the respondent group. The respondents independently generate their ideas in answer to the first questionnaire and return it. The staff team then summarizes the responses to the first questionnaire and develops a feedback report along with the second set of questionnaires for the respondent group. Having received the feedback report, the respondents independently evaluate earlier responses. Respondents are asked to independently vote on priority ideas included in the second questionnaire and mail their responses back to the staff team. The staff team then develops a final summary and feedback report to the respondent group and decision makers.

Variations of this basic approach include:

whether the respondent group is anonymous
whether open-ended or structured questions are used to obtain information from the respondent group
whether the responses are collected in written form or verbally—for example, over the phone
how many iterations of questionnaires and feedback reports are used
what decision rules are used to aggregate the judgments of the respondent group.

The number of participants can range from a few to many hundreds. The larger the number of iterations employed, the closer to consensus will be the result. Written questionnaires can be in pencil-and-paper form or distributed and returned using electronic communication tools including email and the internet. Computer-based systems, using highly structured questionnaires, can produce real-time findings.

Examples of its use in research integration

1. The environment: developing an environmental plan for a university

What was the context for the integration, what was the integration aiming to achieve and who was intended to benefit?
Senior administrators at Dalhousie University in Halifax, Nova Scotia, Canada, were aware of a significant gap between the university’s environmental policies and their implementation. As a result, they resolved to develop an implementation plan that would be acceptable to all those who would be responsible for making it work.
Those responsible for developing the implementation plan used the Delphi technique

to consult with key representatives of the university community in order to generate ideas about the most desirable and feasible ways in which to incorporate the new Environmental Policy into the activities and structure of the university…Modifying a Delphi study for policy research can be used to generate ideas and provide decision-makers with the strongest arguments for and against different resolutions to an issue. (Wright 2006:763)

Who did the integration, how was it undertaken and what was integrated?
A panel of 28 individuals was selected, with equal numbers drawn from the identified key stakeholders: ‘students, staff, faculty, and administrators.’ A core feature was that the Delphi study participants would be anonymous to one another, as the Delphi technique was implemented by email between the panellists and the project managers, rather than through face-to-face discussion. This was considered important as it gave equal weight to each panellist’s judgments, avoiding problems that the power imbalances among the panellists (for example, between students and faculty) might otherwise create. No information was provided on what was integrated.
The Delphi questionnaires were distributed and responses received by email. Round one was the open-ended question: ‘After reading the Environmental Policy, what recommendations do you have to incorporate it into the activities and structure of Dalhousie University?’ A master list of 125 suggestions was developed from the responses. In round two, the participants were asked to review the master list from round one and rate each item for desirability and feasibility (separately) on a five-point Likert scale. The responses to the second round were analysed statistically for measures of central tendency and dispersion. The items were categorised as those that received consensus: a) for being desirable and feasible; b) for being desirable but not feasible; or c) rated as either not desirable or unsure. Each participant received a personalised questionnaire in round three, listing that person’s ratings in round two, along with the group responses. They were asked to reconsider their ratings and make any changes. In round three, the majority of participants modified two to five of their round two ratings.
What was the outcome of the integration?
The results of the Delphi technique study were used by the university managers as a key input to developing the Environmental Policy Implementation Plan. The features that made the Dephi technique useful were identified as anonymity, encouraging exploratory thought and developing innovative ideas, achieving consensus, serving as an educative tool about environmental issues and being a tool for empowerment (Wright 2006).

2. Public health: estimating the incidence of Salmonella poisoning

What was the context for the integration?
Despite food poisoning through food-borne Salmonella infection being an important public health problem, in the United Kingdom in the mid 1990s, official statistics were not able to provide an accurate estimate of the incidence of infections. It was agreed that the official data significantly underestimated the true incidence, but experts’ views differed about the level of underreporting.
What was the integration aiming to achieve and who was intended to benefit?
The Delphi technique was used by Henson (1997:197) to: a) ‘reconcile differences in expert opinion and provide more reliable estimates of the incidence of food-borne Salmonella’; and b) to identify expert opinion about the effectiveness of the available measures for control of the infection. This dialogue method was chosen because it was, in the view of the person who implemented the study,

a recognised technique for reconciling differences in group judgements where there is inherent uncertainty as to the actual state of the world. In this case, the group consists of experts on food-borne Salmonella in the UK. The aim is to generate data which may overcome acknowledged problems with published statistics. (Henson 1997:196)

Who did the integration, how was it undertaken and what was integrated?
The Delphi study was initiated by conducting a workshop in which seven experts in food-borne Salmonella infection examined the issues to be covered in the survey. They did so using the nominal group technique, discussed below. The workshop identified the precise wording to be used in the Delphi study questions.
Some 62 experts (their areas of expertise were not specified) in food-borne Salmonella infection, identified by workshop participants, were then invited to be part of the Delphi study, and 42 of them agreed to do so. Five Delphi rounds were conducted during a seven-month period, with three exploring the experts’ judgments of the incidence of infection and all of them exploring the effectiveness of control measures. This was done by means of questionnaires, but further details were not given.
The first question was: ‘What would you estimate to be the total number of persons ill due to infection with non-typhoid Salmonella in the UK from all sources (food and non-food), over the course of one year?’ The second asked what proportion of infections participants thought was food-borne and the third invited them to identify the proportion of cases by type of food. For each question, they were asked to advise how they produced their estimates and any difficulties they encountered in doing so. The results of the first and second rounds were fed back to participants, showing them the median, minimum and maximum responses from the whole panel and inviting them to revise their estimates of incidence.
In round one, participants were also asked to list the control strategies available for reducing the incidence of food-borne Salmonella infection. In round two, they were asked to refine the list and in round three the refined list was presented along with the question, ‘Taking each control strategy in turn, consider how effective it would be at reducing the total incidence of food-borne non-typhi Salmonella in the UK?’ This question was repeated in the fourth and fifth rounds, with the findings of the previous round fed back to participants.
What was the outcome of the integration?
An important outcome of the process was the narrowing of the range of estimates for the incidence of infection, as participants reflected on the median and range of responses to the incidence questions. Regarding the effectiveness of control measures, one approach (food irradiation) was identified by the panel as being particularly effective. Considerable disagreement remained, however, about which other measures were effective, even after three rounds considering this question. The author concludes that this is not really problematic as the Delphi study ‘provides a good summary measure of expert opinion in an area which is characterised by great uncertainty’ and ‘the spread of responses provides a good indication of the range within which we can expect to find the actual state of the world’ (Henson 1997:203).

3. Security: developing a new medical school curriculum addressing bio-terrorism

What was the context for the integration?
Since the 11 September 2001 attacks on the New York World Trade Centre and the Pentagon in the United States, responding to the medical sequelae of bio-terrorism and biological warfare incidents is no longer considered solely the province of emergency medicine specialists. Rather, it is seen as something that all healthcare providers need to be prepared to handle.
What was the integration aiming to achieve and who was intended to benefit?
Medical educators in the United States set out to develop new medical school curriculum guidelines relating to bio-terrorism so as to equip the next generation of medical graduates to be able to respond to this threat. They used an internet-based Delphi survey to identify the educational objectives to be covered by the curriculum guidelines (Coico et al. 2004).
The Delphi technique was chosen for this purpose because, in the views of those who wished to develop the new curriculum guidelines, it ‘can provide a relatively rapid means of gaining a consensus on complex issues’ (Coico et al. 2004:367).
What was being integrated?
This consensus came through the integration of the judgments of a group of experts in microbiology and immunology, who were engaged in medical education in US universities. Some 89 per cent of panellists had PhD degrees, 7 per cent were physicians and 77 per cent were involved in medical curriculum development. Two-thirds rated their expertise concerning bio-terrorism, biological warfare and bio-defence as ‘strong’ or ‘moderate’.
Who did the integration and how was it undertaken?
A total of 237 people were invited, by email, to join the Delphi panel and 64 (25 per cent) participated in one or more rounds. The Delphi process comprised three internet-based rounds using ‘a dynamic Web-based questionnaire’ (Coico et al. 2004:367). The responses were captured from the web server onto spreadsheets. Before the first round, participants provided demographic information including self-assessment of their expertise in bio-terrorism.
Previous workshop discussions had produced a list of six content-related curriculum categories for bio-terrorism teaching and learning: general issues, bio-defence, public health, infection control, infectious diseases and ‘weaponizable toxins’ (Coico et al. 2004:368). These were put to the participants in the first Delphi round, and they were asked to add knowledge, skills and attitude objectives to the list of educational objectives. They were also asked for suggestions about any content areas that seemed irrelevant to the project. In round two, the responses to round one were fed back to participants and they were asked to assess, for three identified levels of medical training, the relative importance of each objective. The results of round two were fed back to the panel in round three in the form of percentage endorsed figures, and they were asked to identify their top five curriculum objectives in each category. They were also asked to rate the usefulness of nine different methods of teaching/learning and assessment of bio-terrorism and bio-defence topics.
The products of round two were also passed to an independent expert committee to obtain their views. This separate, independent committee had members who were experts from other professions and disciplines concerned with the issues being addressed by the panel. Its function was to receive the panel’s findings and consider their implications.
What was the outcome of the integration?
Although the authors of the paper reporting on this project stated that they would have benefited from a higher participation rate, they felt that the Delphi technique ‘provided an opportunity to explore bioterrorism-related curriculum issues in depth’ (Coico et al. 2004:372). The outcome was the inclusion, in the US Medical Licensing Examination, of approximately one-third of the educational objectives identified through the Delphi study.

4. Technological innovation: developing professional association policies and practices for shifting from paper to electronic communications

What was the context for the integration?
The Institute of Electrical and Electronics Engineers (IEEE) describes itself as the world’s leading professional association for the advancement of technology, and the largest, with more than 365 000 members in more than 150 nations (<http://www.ieee.org/web/aboutus/home/index.html>). In the late 1990s, it identified the need to establish policies and procedures governing its transition from hard copy to electronic communication and dissemination of information within the institute and beyond it. Indeed, in 1996, it adopted the slogan ‘IEEE: networking the world’.
What was the integration aiming to achieve and who was intended to benefit?
The institute used the Delphi method to assess the benefits of and obstacles to its transition to electronic communications. It saw this method as a

technique that is considered appropriate when the research purpose is to glean and synthesize expert opinion about complex issues and to identify recommendations for addressing them. The technique is frequently used in exploratory research and in efforts aimed at technological forecasting, including technological trajectories and the impacts of technological change…Use of the method in this research project has allowed the researchers an opportunity to pool a wide range of expert opinion in order to arrive at a series of focused predictions that may guide the IEEE’s approach during this significant transition period. (Herkert and Nielsen 1998:80)

Who did the integration, how was it undertaken and what was integrated?
A pool of institute members—the exact number was not reported—was identified by the project managers and invited to participate in the study. They came from five areas: institute leadership and staff, institute technical activities representatives, institute regional activities representatives, customers and ‘informed others’. Forty agreed to participate in the study and 30 provided demographic information and responded to round one. (It is not clear if the Delphi questionnaires were distributed and returned electronically or in ‘pen-and-paper’ format.) In round one, participants were asked to assess:

the potential contribution of electronic communication and information dissemination in fulfilling the institute’s strategic planning goals and objectives that did not rely explicitly on the use of electronic media
the impact of electronic communication and information dissemination with respect to the five strategic planning goals and objectives that relied explicitly on the use of electronic media.

They were also invited to provide open-ended commentary on the benefits of and obstacles to the use of electronic media (Herkert and Nielsen 1998:82–4). A round one example question was:

Products and Services Objective: Make all IEEE information products and databases of value to members available in electronic form as quickly as possible.

I agree; making products and databases available in electronic

form as quickly as possible is a valuable objective.

       I disagree; making products and databases available in electronic

       form as quickly as possible is not a valuable objective.

       Discuss your answer in the space provided below.

In round two, the panellists were given a synthesis of the obstacles to the Institute’s increasing reliance on electronic communication derived from round one and were asked to identify the 10 obstacles that each respondent considered most problematic. In round three, they were provided with a list of the 11 major obstacles identified in round two and asked what actions the Institute should take to benefit maximally from electronic communications while avoiding its potential pitfalls. Content analysis was conducted on the responses to round three to identify the path forward.
What was the outcome of the integration?
This application of the Delphi technique resulted in the IEEE identifying six key factors affecting the adoption and use of electronic media:

characteristics of the IEEE as technology initiator;

characteristics of the potential individual adopter;

characteristics of the potential organizational adopter;

characteristics of the technology;

outcomes, and

characteristics of the contextual environment. (Herkert and Nielsen 1998:95–6)

This finding, combined with a content analysis of the panellists’ qualitative responses, enabled the investigators to develop a range of recommendations for consideration by the executives of the Institute to guide it in embracing electronic communication methods.

Commentary

The Delphi technique is generally implemented by means of pen-and-paper, email or web-based questionnaires, or by one-on-one interviewer–interviewee questionnaires. This means that it does not entail face-to-face dialogue. Instead, a ‘conversation’ occurs by means of responding to the questionnaires and sharing all the participants’ responses, one with another. What are missing are the additional communication cues—verbal and non-verbal—that occur in face-to-face dialogue. Here, the focus is on the contents of the message, the real wording, rather than the other features that constitute human communication. Nonetheless, we classify it as a dialogue method in that the iterations in the process have features similar to two-way communication in the face-to-face situation.
All four case examples applied the Delphi technique reasonably closely to the ‘ideal type’. The number of iterations used varied, with three of the examples utilising three rounds and another (the public health example) five rounds—an unusually large number. This also demonstrates how it can be implemented flexibly, depending on the topic, the participants, resource considerations, and so on.
Most commonly, the method is used with a group of peers: experts with relatively equal status, a more-or-less common knowledge base and a shared epistemology. This was the case with the public health and security examples, all the participants in which were experts. In contrast, the first example illustrated the participation of three stakeholder groups, none of which was particularly expert on the topic. The fourth example demonstrates the method’s use among a fairly diverse range of participants.
As a method of research integration, it is especially useful for complex problems about which uncertainty exists and for which expert judgment is needed to deal with this uncertainty. The problems are typically multifaceted and demand insights derived from different types of knowledge, experience and information. This means that the problem being addressed needs to be tightly defined and the questionnaires must deal explicitly with boundary issues. It is a highly task-oriented process, seeking answers to a tightly defined problem.
As the examples illustrate, the method is highly adaptable in terms of its contents. We are aware of at least one example of the method being given a title that reflects the contents being judged—namely, the ‘Ethical Delphi’ (Millar et al. 2006, 2007). This is not so much a methodological variant as the application of the standard Delphi technique to a particular content area—in this case, concerns about ethics and values.
This method, unlike those discussed above, relies very much on the people (the research integrators) who manage the process to make the syntheses and judgments. They develop the questions, score the responses and identify the conclusions, their validity and reliability and their utility. This allows for strong focus on the task (in contrast to unfacilitated face-to-face group processes where the focus can be readily diverted).

Origins and genealogy

The Delphi technique had its origins in the early 1950s’ Cold War between the United States and the Soviet Union, when the RAND Corporation was commissioned by the US Department of Defence ‘to apply expert opinion to the selection, from the point of view of a Soviet strategic planner, of an optimal US industrial target system and to the estimation of the number of A-bombs required to reduce the munitions output by a prescribed amount’ (Dalkey and Helmer 1963:458). Its title refers to the Delphic Oracle, reflecting the fact that it was originally used as a forecasting technique with respect to science and technology.
Since then, the Delphi technique has been used many thousands of times in diverse sectors addressing a huge array of questions. Although the originators did not situate it (in their 1963 paper) in any particular body of theory, subsequent scholars have attempted to do so. A wide range of traditions in Western philosophy has been invoked in this context, with one schema, presented by Mitroff and Turoff (1975), demonstrating that the method can be understood through the Lockean Inquiring System (the basis of much empirical science), the Leibnizian Inquiring System (the basis of much theoretical science), the Kantian Inquiring System (which combines both of these approaches) and the Singerian-Churchman Inquiring System.
Scholars have concluded that there is no single school of philosophy that best captures the theory underlying the Delphi technique (Mitroff and Turoff 1975).

MIKIPEDIA POLITICAL BLOG

THE BOSTON MARATHON BOMBING

Wednesday, August 21, 2013

DELPHI TECHNIQUE – DIALOGUE METHODS FOR UNDERSTANDING A PROBLEM BROADLY: INTEGRATING JUDGMENTS