The Delphi Technique
Dr. Muhammad Imran Yousuf
Division of Continuing Education,
University of Arid Agriculture, Rawalpindi
The Delphi technique is a group process used to survey and collect the opinions of experts on a particular subject. Linstone and Turoff (1975) provided a basic definition of the Delphi technique: “Delphi may be characterized as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem”. It has application whenever policies, plans, or ideas have to be based on informed judgment. This technique is useful where the opinions and judgments of experts and practitioners are needed but time, distance, and other factors make it unlikely or impossible for the panel to work together in the same physical location.
The Delphi technique, by definition, is a group process involving an interaction between the researcher and a group of identified experts on a specified topic, usually through a series of questionnaires. Delphi has been used to gain a consensus regarding future trends and projections using a systematic process of information gathering. This technique is useful where the opinions and judgments of experts and practitioners are necessary. It is especially appropriate when it is not possible to convene experts in one meeting. Skutsch and Hall (1973) identified the Delphi technique as a method for gaining judgments on complex matters where precise information is unavailable.
The technique was named after the ancient Greek oracle at Delphi from which prophecies were given (Koontz & O’Donnell, 1976). An oracle refers to a statement from someone of unquestioned wisdom and knowledge or of infallible authority (Funk & Wagnells, 1966). The Delphi technique was developed by Olaf Helmer and his associates at the Rand Corporation in the early 1950s when they were working on defense research. Rieger (1986) described Delphi’s development in five stages: (1) secrecy and obscurity, (2) novelty, (3) popularity, (4) scrutiny, and (5) continuity.
The first stage was secrecy, during which the Delphi technique was classified by the military. Delphi techniques were developed to gain consensus within a group of military experts on a very sensitive problem. This stage lasted from the early 1950s to the early 1960s, when it was declassified.
The second stage, novelty, lasted from the mid-1960s to the late 1960s. During this stage the technique was used primarily by corporate planners as a forecasting tool for industry and human services.
The third stage, popularity, lasted from the late 1960s to the mid-1970s. During this time 389 articles, papers, and reports appeared on the topic. Rieger (1986) reported that between the years of 1970 and 1974, 61 dissertations used the Delphi technique.
The fourth stage, scrutiny, began in 1975 with Sackman’s unexpected attack on the Delphi technique itself. The attack was not unchallenged. The “first thrust” of Sackman’s criticism was that the technique did not measure up to the psychometric standards of the American Psychological Association…. Sackman’s reasoning on this point was effectively challenged and refuted by Goldschmidt. Goldschmidt (1975) responded to Sackman’s criticism in an article entitled, “Scientific Inquiry or Political Critique? Remarks on Delphi Assessment, Expert Opinion, Forecasting, and Group Process by H. Sackman.” Rieger (1986) stated, “Members of the research community thus gave notice that Sackman should not necessarily be accepted as the final arbiter regarding Delphi’s scientific respectability”. Sackman’s second thrust criticized the indiscriminate execution of Delphi studies.
The fifth and final stage, continuity, is the Delphi’s present stage of development. Rieger (1986) identified 599 dissertations using the Delphi technique between 1975 and 1984, with 441 of them between 1980 and 1984.
Areas of application
Linstone and Turoff (1975) argued that Delphi has application in the following areas:
-ï€ Gathering current and historical data not accurately known or available.
-ï€ Evaluating possible budget allocations.
-ï€ Exploring urban and regional planning options.
-ï€ Planning university campus and curriculum development.
-ï€ Putting together an educational model.
-ï€ Delineating the pros and cons associated with potential policy options.
-ï€ Distinguishing and clarifying real and perceived human motivations.
-ï€ Exploring priorities of personal values, social goals, etc.
They further stated that one or more of the following leads one to use the Delphi Technique:
– The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis.
– The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to experience or expertise.
– More individuals are needed than can effectively interact in a face-to-face exchange.
– Time and cost make frequent group meetings unfeasible.
– A supplemental group communication process can increase the efficiency of face-to-face meetings.
Forms of Delphi
The original intent of Delphi was as a forecasting technique, designed to predict the likelihood of future events. Additional names have been given to this process. Dailey (1988) described it as an exploratory Delphi. Van Dijk (1990) called it a conventional Delphi. According to Dalkey (1972) the Delphi is a procedure that is a rapid and efficient way to “cream the tops of the heads” of a group of knowledgeable people. He further stated that a well-designed and properly managed Delphi could be a highly motivating environment for respondents.
A policy Delphi is one which seeks to generate the strongest possible opposing viewpoints on a policy issue from an expert panel. Rather than consensus, the emphasis is on identifying differing opinions and divergent responses through a process of debate carried out though the rounds of Delphi (Needham, 1990).
The policy Delphi is given other names also, such as focus Delphi and decision Delphi. A normative Delphi (also called a consensus Delphi), focuses on establishing what is desirable in the form of goals and priorities. It does not focus on speculating about what is probable within a given time frame in the future; instead it is an attempt to “… structure a set of properties which could be integrated into a normative future–properties based on the criterion of desirability rather than likelihood …” (Sutherland, 1975).
Most Delphi studies in educational settings are normative and are perceived as particularly useful. Rieger (1986) reported 83 percent of the dissertations completed during the 1981-1984 period which used the Delphi technique were of the normative type. He went on to state, “… it seems reasonable to claim that Delphi is continuing to be a much used tool in the search for answers to normative questions, especially in education areas, but also in other fields”.
The process for each type of Delphi is essentially the same; however, the purpose of a study determines the type of Delphi used. The Delphi’s process is similar to the nominal group technique (NGT), except Delphi does not require the physical presence of group members (Mitchell & Larson, 1987). An interaction process still takes place between the members of the group (Delphi panel) and the researcher, with the researcher acting as a facilitator.
The basic steps of the Delphi process were outlined by Pfeiffer (1968):
1. The first questionnaire which is sent to the panel of experts may ask for a list of opinions involving experiences and judgments, a list of predictions, and a list of recommended activities.
2. On the second round, a copy of the collective list is sent to each expert and the expert is asked to rate or evaluate each item by some criterion of importance.
3. The third questionnaire includes the list, the ratings indicated, and the consensus, if any. The experts are asked to either revise their opinions or discuss their reasons for not coming to consensus with the group.
Scheele (1975) illustrated a process where the opinions and judgments of people familiar with or associated with a subject and they listed a typical sequence of events in the Delphi process in six steps:
1. Identify the group members whose consensus opinions are sought. If the study goes beyond an intact group such that representatives must be selected, care must be taken to insure that all the various publics or positions are proportionately sampled.
2. Questionnaire One. Have each member generate a list of goals, concerns, or issues toward which consensus opinions are desired. Edit the results to a manageable summary of items presented in random order. Prepare the second questionnaire in an appropriate format for rating or ranking (Note: If an established or acceptable listing of such items already exists, this first step can be bypassed.).
3. Questionnaire Two. Have each member rate or rank the resulting items.
4. Questionnaire Three. Present the results of Questionnaire Two in the form of Questionnaire Three, showing the preliminary level of group consensus to each item. Where the individual differs substantially from the group, and chooses to remain so on Questionnaire Three, the respondent should provide a brief reason or explanation.
5. Questionnaire Four. The results of Questionnaire Three are presented in the form of Questionnaire Four, showing the new level of group consensus for each item and repeating the member’s latest rating or ranking, along with a listing by item of the major reasons members had for dissent from the prevailing group position. Each member rates or ranks each item for the third and final time, in light of the emerging pattern of group consensus and the reasons for dissent.
6. The results of Questionnaire Four are tabulated and presented as the final statement of group consensus.
Worthen and Sanders (1987) stated that this “iterative procedure can continue for several more rounds, but the payoff usually begins to diminish quickly after the third round”. Brooks (1979) included an additional step prior to beginning the procedure: assess the willingness of potential panel members to participate in the study.
Several steps, as identified by Brooks (1979), are involved in using the Delphi Technique:
1. Identifying the panel of experts.
2. Determining the willingness of individuals to serve on the panel.
3. Gathering individual input on the specific issue and then compiling it into basic statements.
4. Analyzing data from the panel.
5. Compiling information on a new questionnaire and sending to each panel member for review.
6. Analyzing the new input and returning to the panel members the distribution of the responses.
7. Asking each panel member to study the data and evaluate their own position based on the responses from the group. When individual responses vary significantly from that of the group norm, the individual is asked to provide a rationale for their differing viewpoint while limitations are placed on the length of the remarks in order to keep responses brief.
8. Analyzing the input, and sharing the minority supporting statements with the panel. Panel members are again asked to review their position and if not within a specified range, to justify the position with a brief statement.
Characteristics of Delphi Technique
Dalkey (1967) has identified the following basic characteristics of the Delphi technique:
1. Anonymity – – the use of questionnaires or other communication where expressed responses are not identified as being from specific members of the panel allows for anonymity.
2. Controlled feedback from the interaction – – Controlled feedback allows interaction with a large reduction in discord among panel members. Interaction consists of allowing interaction among group members in several stages, with the results of the previous stage summarized and group members asked to reevaluate their answers as compared to the thinking of the group.
3. Statistical group response – – the group opinion is defined as a statistical average of the final opinions of the individual members, with the opinion of every group member reflected in the final group response.
At the same time as Dalkey (1967) was identifying the basic characteristics of the Delphi technique, Helmer (1967) supported the validity and reliability of the technique as an acceptable method of data collection from an identified group. Further he said that Delphi Technique is efficient in both group decision making situations and in other areas where order of magnitude estimates are required (Helmer, 1983). A Delphi study carried to the extreme degree could be an expensive undertaking in both time and money on the part of the researcher and the respondents.
The Delphi technique is beneficial when other methods are not adequate or appropriate for data collection. It is particularly useful when
1. The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis.
2. The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to experience and expertise.
3. More individuals are needed than can effectively interact in a face-to-face exchange.
4. Time and cost make frequent group meetings infeasible.
5. The efficiency of face-to-face meetings can be increased by a supplemental group communication process.
6. Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured.
7. The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or by strength of personality (“bandwagon effect”). (Linstone & Turoff, 1975)
Helmer (1983) agreed that Delphi is a technique frequently used for eliciting consensus from within a group of experts that has application in reliability and has many advantages over other methods of using panel decision making. Helmer (1983) agrees with Linstone and Turoff (1975) in regards to the application of Dephi. Helmer (1983), Linstone and Turoff (1975), and Dalkey (1972) all found that one of the major advantages of using Delphi as a group response is that consensus will emerge with one representative opinion from the experts.
There are many additional advantages. The technique is simple to use. Advanced mathematical skills are not necessary for design, implementation, and analysis of a Delphi project. Because the Delphi provides confidentiality, many barriers to communication are overcome. Some of these barriers are reluctance to state unpopular views, to disagree with one’s associates, or to modify previously stated positions (Barnes, 1987).
It helps prevent a groupthink, as earlier mentioned, particularly with one or two dominant people. A major strength of the technique is the flexible, but limited, time parameters with which individuals have to respond to the questionnaires (Brooks, 1979). This flexibility allows individuals, who may be restricted by daily schedules and geographic location, the opportunity to respond at times available to them.
Delphi is not without limitations. The consensus reached in a Delphi may not be a true consensus; it may be a product of specious or manipulated consensus. A specious consensus does not contain the best judgment. Instead, it is a compromise position (Mitroff & Turoff, 1975).
Delphi appears to be a straightforward approach to doing research in the area of forecasting and for building consensus. Researchers, at first glance, think of Delphi as a simple technique that can be done easily. However, one must carefully consider the problems associated with Delphi before designing a Delphi study.
Linstone and Turoff (1976) suggested that there are five common reasons for Delphi to fail:
1. Imposing monitor views and preconceptions of a problem upon the respondent group by over specifying the structure of the Delphi and not allowing for contribution of other perspectives related to the problem.
2. Assuming that Delphi can be a surrogate for all other human communications in a given situation.
3. Poor techniques of summarizing and presenting the group response and ensuring common interpretations of the evaluation scales utilized in the exercise.
4. Ignoring and not exploring disagreement so that discouraged dissenters drop out and an artificial consensus is generated
5. Understanding the demanding nature of a Delphi and the fact that the respondents should be recognized as consultants and properly compensated for their time if the Delphi is not an integral part of their job function.
Delkey (1972) pointed out that Delphi is not sufficient to be a defining property for an uncertain question because the expert’s cultural bias can lead to similar answers to some questions which in fact are poorly known; or there could be an instance where the experts legitimately do not know the answer. According to Linstone and Turoff (1975), the “virtual” problems do not affect the utility of Delphi but rather how to select the respondent group.
Barnes (1987) has listed additional disadvantages of the technique:
1. Judgments are those of a select group of people and may not be representative;
2. Tendency to eliminate extreme positions and force a middle-of-the-road consensus;
3. More time consuming than the nominal group process;
4. Should not be viewed as a total solution;
5. Requires skill in written communication;
6. Requires adequate time and participant commitment (about 30 to 45 days to complete the entire process).
Fortune (1992) indicated that an additional reason for Delphi failure is that the panel members may not be able to see the vision or the “big picture” in which they are involved. This problem arises when the panel members chosen are so close to the problem that they cannot see the future.
An overriding factor in the selection of the Delphi technique is the appropriateness of the technique for a particular study. Linstone (1978) identified two circumstances where Delphi techniques are most appropriate:
(1) “the problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis” and
(2) “individuals who need to interact cannot be brought together in a face-to-face exchange because of time or cost constraints”.
The information obtained by the Delphi study is only as good as the experts who participate on the panel. Therefore, the composition of the panel relates to the validity of the results of the research (Spencer-Cooke, 1989). It is the panel’s opinions and judgments that are elicited and analyzed. Therefore, considerable thought most go into the selection of the panel.
Delphi techniques identify the reasons why there is a degree of disagreement among the experts and help to ascertain whether the nature of the disagreement is real or purely semantic (Helmer, 1983). Convergence, even when it happens, is not enough to validate the method because it should be convergence toward the correct value that counts (Helmer, 1983). Helmer (1983) stated that there are two reasons why relatively few experiments have been conducted to validate the predictive power of Delphi inquiries. One is that long-range forecasts cannot be verified until a sufficiently long time has elapsed. The other is that Delphi is a method pertaining to the utilization of expert opinions. Hence, to validate the method properly, experts would have to be used as laboratory subjects.
Dalkey and Helmer (1983) have well documented that statistically the Delphi Techniques tend to produce not only convergence but also that convergence is in the direction of the true value. Helmer (1983) pointed to the explicit evidence of the validity of the Delphi technique in producing relatively reliable forecasts.
Anderson, F. T. T. (1975). A modified Delphi study of the political feasibility of critical issues affecting educational reform in Maryland. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, Virginia.
Barnes, J. L. (1987). An international study of curricular organizers for the study of technology. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, Virginia.
Brooks, K. W. (1979). Delphi technique: Expanding applications. North Central Association Quarterly, 53, 377-385.
Cetron, M. J. (1969). Technological forecasting. New York: Gordon and Breach.
Dailey, A. L. (1988). Faculty consensus at a multi-campus college through Delphi. Community/Junior College Quarterly, 12, 21-26.
Dalkey, N. C. (1967). Delphi. Santa Monica, CA: The RAND Corporation.
Dalkey, N. C., & Helmer, O. (1962). An experimental application of the Delphi method to the use of experts (Report No. RM-727-PR) (Abridged). Santa Monica, CA: The RAND Corporation.
Flanders, F. B. (1988). Determining curriculum content for nursery/landscape course work in vocational agriculture for the 21st Century: A future’s study utilizing the Delphi technique. Doctoral Dissertation, University of Georgia.
Forsyth, D. R. (1990). Group dynamics (2nd ed.). Pacific Grove, CA: Brooks/Cole Publishing Company.
Fortune, J. C. (1992, May). [Interview with Dr. Jimmie C. Fortune, Professor at Virginia Tech]. Blacksburg, VA: Virginia Tech.
Goldschmidt, P. G. (1975). Scientific inquiry or political critique? Remarks on Delphi assessment, expert opinion, forecasting, and group process by H. Sackman. Technological Forecasting and Social Change, 7, 195-213.
Helmer, O. (1967). Systematic use of expert opinions (Report No. P-3721). Santa Monica, CA: The RAND Corporation.
Helmer, O. (1983). Looking forward: A guide to future research. Beverly Hills, CA: Sage.
Isaac, S., & Michael, W. B. (1981). Handbook in research and evaluation. San Diego, CA: EdITS Publishers.
Janis, I. L. (1972). Victims of groupthink. Boston: Houghton-Mifflin.
Koontz, H., & O’Donnell, C. (1976). Management: A systems and contingency analysis of managerial functions (6th ed.). New York: McGraw-Hill.
Linstone, H. A. (1978). The Delphi technique. In J. Fowlers (Ed.), Handbook of futures research (pp. 273-300). Westport, CT: Greenwood Press.
Linstone, H. A., & Turoff, M. (Eds. ). (1975). The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley.
Mitchell, T. R., & Larson, J. R., Jr. (Eds.). (1987). People in organizations: An introduction to organizational behavior. New York: McGraw-Hill Book Company.
Mitroff, I. I., & Turoff, M. (1975). Philosophical and methodological foundations of Delphi. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 17-35). Reading MA: Addison-Wesley Publishing Co.
Needham, R. D. (1990). Geographic: The policy Delphi: Purpose, structure, and application. The Canadian Geographer, 34(2), 133-142.
Pfeiffer, J. (1968). New look at education. Poughkeepsie, NY: Odyssey Press.
Reeves, G., & Jauch, L. R. (1978). Curriculum development through Delphi. Research in Higher Education, (8), 157-68.
Rieger, W. G. (1986). Directions in Delphi developments: Dissertations and their quality. Technological Forecasting and Social Change, 29, 195-204.
Scheele. (1975). Consumerism comes to Delphi. Technological Forecasting and Social Change, 7, 215-219.
Skutsch, M., & Hall, D. (1973). Delphi: Potential uses in education planning. Project Simu-School: Chicago component. Chicago, IL: Chicago Board of Education, Illinois Department of Facility Planning.
Skutsch, M., & Hall, D. (1973). Delphi: Potential uses in education planning. In J. L. Barnes, (1987). An international study of curricular organizers for the study of technology. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, Virginia.
Spencer-Cooke, B. (1989). Conditions of participation in rural, non-formal education programmes: A Delphi study. Educational Media International, 26(2), 115-124.
Sutherland, J. W. (1975). Architecting the future: A Delphi-based paradigm for normative system-building. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications. Reading, MA: Addison-Wesley.
Uhl, N. P. (Ed). (1983). Using research for strategic planning. San Francisco: Jossey-Bass.
Van Dijk, J. A. G. M. (1990). Delphi questionnaires versus individual and group interviews: A comparison case. Technological Forecasting and Social Change, 37, 293-304.
Weatherman, R., & Swenson, K. (1974). Delphi technique: Futurism in education. In J. L. Barnes, (1987). An international study of curricular organizers for the study of technology (pp. 56-69). Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, Virginia.
Worthen, B. R., & Sanders, J. R. (1987). Educational evaluation: Alternative approaches and practical guidelines. New York: Longman.