examination credibility tree goal

Expert methods of analysis in the study of socio-economic and political systems occupy a special place in the display and assessment of social processes.

In general, two groups of expert assessments can be distinguished: individual and collective:

Individual expert methods used for forecasting in relatively narrow areas of science and practice. They are based on the use of expert opinions, independent of each other. The information that the customer receives from the expert is unique and focused on a problem of a local nature.

1. Method of garlands and associations.

The scope of application of this method can be a complex problem situation, poorly studied, regarding which there is no established opinion. The method can only be implemented under the condition; a lot of preliminary work, consisting in an in-depth study of the properties of analogues of the object under study, the feasibility of their involvement, the formation of the psychological attitude of experts, etc.

2. Method of paired comparisons.

Based on a simple comparison, the expert of alternative options from which he must choose the most preferable. The method makes it possible to take into account the equivalence or fundamental incomparability of the presented alternatives, and therefore they are excluded from the analysis. During such a comparison, the expert not only selects best options, but also formulates criteria that allow such a choice to be made, emphasizing the properties and characteristics of the selected alternative.

3. Method of preference vectors.

When used, the expert is presented with the entire set of alternative options being evaluated and is asked to indicate for each of them how many alternative options he is ahead of. The obtained information can be represented as a vector, one of whose components is the number of alternatives that are superior to the first, the second component is the number of alternatives that are superior to the second, etc. This method can also be used during collective examination, presenting a collective view on the relationship between the alternatives under consideration.

4. Method of focal objects.

Distinctive feature this method is that it completely excludes quantitative approach and is aimed mainly at concentrating the researcher’s attention on the so-called focal object, which serves as the basis for comparison with the objects that make up the field of directed random search.

5. Individual expert survey.

Possible in the form of an interview or in the form of an analysis of expert assessments.

The interview method means a conversation between the customer and the expert, during which the customer, in accordance with the developed program, poses questions to the expert, the answers to which are significant for achieving program goals. Analysis of expert assessments involves individual filling out by an expert (experts) of a form developed by the customer, based on the results of which a comprehensive analysis of the problem situation is carried out and possible ways in her decision. The expert presents his considerations in the form of a separate document, drawn up on the basis of a thorough examination of the object.

6. Midpoint method.

Used for large quantities alternative solutions. To do this, two alternative solutions are formulated, one of which is the least preferable, the second the most. After this, the expert is asked to select a third alternative, the assessment of which is located between the values ​​of the first and second alternatives. The procedure ends when the comparative preference of all alternative options participating in the examination is determined.

7. Churchman-Ackoff method.

In accordance with this method, all alternative options are ranked by preference, and the expert assigns quantitative ratings to each of them. The advantage of this method is that experts allow adjustments during the discussion of alternative options. If one option is preferred over another, then their values ​​are summed up.

8. Lottery method.

In accordance with the method, available alternatives are distributed in descending order of preference.

Collective expert assessments used when predicting objects and processes of an interdisciplinary nature.

Collective methods are the most effective in terms of achieving maximum objectivity of expert assessment, since they involve the use of a wide and representative range of specialists. In general, methods for organizing collective idea generation can be divided into several types.

1. Brainstorming.

The main goal of brainstorming is to stimulate the creative process of generating ideas, which is possible in a group discussion. The method allows, in an uncertain situation, to develop maximum quantity possible solutions, focusing the participants' attention on the problem under discussion. Characteristic feature This method is a procedure for separating the stage of generating ideas from the stage of their evaluation. The main advantage of the brainstorming method is its focus on finding non-standard solutions, capable of being realized with the most open and free mode of discussion of the problem. This mode of discussion allows not only to identify possible directions in solving current problems, but also to form a group of people with high creative abilities, which is extremely important for an organization of any type.

2. Methodology of courts.

The essence of the technique is to organize a discussion of a problem situation in a group, when one side develops independent proposals, and the other gives criticism to all these proposals. The use of this technique reproduces a court scenario in which there is a competition between the prosecution and the defense. The purpose of the methodology is to identify the most reasoned and optimal solution obtained during a thorough examination.

A characteristic feature of the courts’ methodology is its role-playing, which allows the participants in the discussion to express themselves most fully in the organizational process, realizing not only their own place in it, but also the place of other services and job functions.

3. Black box method.

The main advantage of the “black box” method is to minimize the possible influence of stakeholders on making ineffective decisions. The method aims to identify a special analytical center, which makes conclusions exclusively on the developments of independent experts who evaluate the prospects for making decisions on a certain list of problems.

4. Heuristic forecasting method.

The use of this method involves the involvement of highly specialized experts who, based on pre-developed questionnaires and tables, must develop general model the object under study. Thus, the heuristic forecasting method is a method for obtaining and specialized processing of forecast estimates of an object through a systematic survey of highly qualified specialists (experts) in a narrow field of science, technology or production. Forecast expert assessments reflect the individual judgment of a specialist regarding the prospects for the development of his field and are based on the mobilization of professional experience and intuition.

5. Synectic method.

One of the advantages of the synectic method is the ability to achieve consolidation between the various levels of management involved in the process of making an expert assessment. The use of the method is advisable due to the fact that during the adoption of expert assessments, problems are discussed between representatives of the same level, which allows them to speak frankly and balancedly.

In this case, it is necessary to take into account the opinion of not only representatives of senior management levels, who can often have very general idea about the object of evaluation, but also ordinary performers, whose knowledge about the object is very valuable and practically significant.

The synectic method is significant in the practice of Japanese enterprises, where the factor of personnel involvement in decision making is given great value. Thanks to the consolidation of interests between expert levels, not only social stability is achieved, but unique way awareness and resolution of current problems.

6. Diary method.

A characteristic feature of this method is its impersonality and extreme objectivity. The implementation of the method is possible only if there are documentary sources, on which the examination report is exclusively carried out. Most often, the method is used in a management structure with a strictly regulated system job descriptions. The object of expert assessment is, as a rule, the shift log, instructional documents, on the basis of which recommendations are developed for improving the activities of enterprises.

7. “Delphi method”

It is the most popular of the expert methods, and its popularity is directly related to the heuristic capabilities of the method itself, which allow solving complex complex problems.

The essence of the method is a consistent survey of experts’ opinions on a problem of interest to the organizers of the examination.

The method involves a series of interviews with experts who do not have the opportunity to enter into direct contact with each other and receive information about the conclusions of others only from their written reports. The purpose of the method is to make an objective and accurate assessment of existing alternatives in order to make optimal and socially acceptable decisions.

Expert methods are used in solving prognostic, analytical and design problems associated with informality and lack of certainty in ideas about organizational and economic objects. The essence of this method: experts carry out an intuitive-logical analysis of the problem with a qualitative assessment of the judgment and formal processing of the results. Features of the expert assessment method: the need for a scientifically based organization of examination, the use of quantitative methods to assess the qualitative judgments of experts.

Expert method can be used to determine forecasts for the development of objects; when determining goals and objectives, alternative distribution of resources; when making decisions under conditions of uncertainty and risk.

1st stage using this method - forming a group of experts. Properties that are necessary for a specialist to be included in the expert group.

Competence (degree of qualification in a certain field of knowledge);

Creativity (ability to solve creative problems);

Analytical and broad-mindedness; constructiveness (the ability to form concrete proposals);

Self-criticism of the expert; attitude towards examination.

To form expert groups, testing, documentation and other methods can be used.

Test method consists in the fact that, based on the developed tests, possible candidates are examined, and a group is formed based on the results of the answers.

Documentation method- selection of experts based on their objective characteristics, which are contained in their personal documents (work experience, position, academic degree, number of publications, etc.).

Method of appointment- determination by the head of a group of experts from among the employees. The main disadvantage of this method: the opinion of employees may be consistent, but erroneous, expressing the official position of the organization on this issue (“school effect”). The results of the examination in this case are of interest mainly only for internal use.

2nd stage applications expert method- carrying out an examination. This stage begins with choosing a method for interviewing experts. There are individual, group and Delphic methods.

At individual way from each expert, through questionnaires or interviews, assessments are obtained that are independent of the opinions of others. Then, after their generalization and processing, the overall resulting assessment is determined. It is rational to use individual expertise when it is necessary to develop a point forecast of the condition of an object, when ranking a set of objects, and in other cases when the most important qualities of an expert are his competence and constructiveness.

Group the method involves obtaining a summary assessment or a common decision from all experts at once through joint discussion. Its use is advisable when searching for unconventional solutions, when assessing the characteristics of little-studied objects, i.e. if necessary, obtain a creative solution. A group survey can be carried out through discussions, meetings, conferences, and brainstorming.

Delphi method synthesizes a number of positive features of individual and group examinations. Experts independently express their opinions in writing. The most important component of the method is carefully developed survey programs carried out in several rounds, and the regulation of questions in each subsequent round. At the end of each round, the group of examination organizers analyzes the responses received, summarizes them and prepares a certificate-bulletin based on the results of the round, the text of which is familiarized to all experts. In this case, the information in the certificate is anonymous. During the repeated survey, experts receive questions that clarify the initial answers and formulated conclusions, taking into account the results of the previous round. In the third round, experts are told on which points there is a common opinion; experts who expressed a different opinion from others are asked to justify it. The fourth, most often the last, round repeats the procedure of the third. Thus, the area of ​​divergence of opinions is narrowed and a common solution is developed. The advantage of the Delphic method is that it reduces or completely eliminates such psychological factors as ostentatious conviction, reluctance to refuse to publicly express one’s opinion, and the influence of authority.

3 th stage expert methods - processing survey results. To ensure the possibility of formal processing of examination results, a numerical system is required that describes the properties of objects and the relationships between them using quantitative parameters (various scales of names (classifications), orders, intervals, relationships, differences).

The naming scale is used to describe the belonging of an object to certain classes. Order scale - to measure the ordering of objects according to one or a number of characteristics (rank scale). Interval scale - to display the magnitude of differences between the properties of objects. Ratio scale - to reflect the relationship between the properties of objects, for example, their weight. Difference scale - if necessary, determine how much one object is superior to another in one or more characteristics.

The choice of scale is determined by the objectives of the examination, the characteristics of the object, and the capabilities of the group.

When processing examination results, the choice is important measurement method. The most commonly used methods: ranking, paired comparison, direct assessment, sequential comparison.

The regulations must meet following requirements: provide sufficient variety of wording; unity of the structure of the formulation (for example, the formulation must consistently answer the questions: what is necessary? over what (with what)? for what?) The resulting formulations must sufficiently fully reflect their most important content, i.e. have significant capacity; the formulation should be done in such a way as to exclude discrepancies.

The problems of improving expert technologies are associated with the development of the following areas: the formation of an expert commission, the organization and conduct of examinations based on the use of modern methods, the use of multi-criteria assessments when interpreting the results.

25. HEURISTIC METHODS

Heuristic methods are based on logic, intuition and experience of decision makers (DMs). These methods allow you to “catch” and use these processes when developing alternatives. Depending on the approach used, heuristic methods are divided into formal heuristic and informal heuristic.

The basis of formal heuristic methods is the formalization of techniques for solving complex problems by a person by modeling it thought processes. They include the method of evolutionary modeling, labyrinth methods, etc.

Evolutionary modeling assumes the presence of initial experience in the process of adopting SD. This experiential, informational material is needed to run the evolutionary model. Based on existing experience, several solution options are being developed that allow us to approach eliminating the problem and achieving the solution goal from different positions. Each option is examined based on predetermined criteria. In test mode, the original, reference solution (“parent”) is changed randomly, resulting in the production of a “child” (generated idea). If the “descendant” is worse than the “parent”, it is discarded, and through another mutation a new “descendant” is born. If the “child” is better, then the “parent” is discarded, and the “child” takes its place, and the procedure is repeated again. The main advantage of this method: the ability to use computer technology, which allows you to search quite quickly. However, you cannot get an outstanding, non-standard, creative solution.

Labyrinth methods are based on a step-by-step search followed by an assessment of the possible continuation of the path to eliminating the problem. If the direction is “dead-end”, a return to the starting point occurs, and the process is repeated again until a path for further movement is found.

Conceptual modeling is based on collecting initial information when analyzing the situation and building a structural model that allows you to isolate the most important elements relationships. The main means to achieve the goal is a structuring method based on the principle of decomposition (separation).

The basis of informal heuristic methods is the management of human intellectual activity. The need for such management is due to the peculiarities of his thinking (informality, ability to generalize, to navigate in an uncertain situation, tendency to dissipation and loss of information). To enhance the first three properties and neutralize the last two, psychointellectual generation of ideas is used.

The process of solving complex problems using idea generation is carried out in the form of a targeted, controlled conversation-discussion between two direct participants: the leader and the decider. The presenter puts before resolving issues, on which the decider must express his judgment. A discussion ensues around these judgments. Opponents and experts can be assigned to help the presenter. The opponents' task is to criticize the decision maker's judgments and involve him in the discussion. The task of the experts is to help the presenter evaluate judgments and outline the consequences of further discussion. Several possible schemes for organizing idea generation sessions:

By the number of leaders: poly-control (several leaders), mono-control (one leader), auto-generation (no leader);

By the number of decisive ones: unilateral schemes (one decisive), multilateral schemes (many decisive);

By type of contact: with direct contact (in one room), with indirect contact (through technical means).

Conditions for ensuring targeted idea generation:

It is necessary to ensure psychological comfort (creating a comfortable workplace, high spirits and a feeling of relaxedness in the decision maker);

Ensure structuring of the process of finding a solution (develop a psychoheuristic program containing a list of issues discussed, goals of discussion and recommendations);

Create information and technical support systems.

As a result of generation, a set of data should be obtained that makes up the main information array or field of possible solutions.

A number of methods of psychological activation are based on the concept of psycho-intellectual generation of ideas. The choice of generation method is made depending on the nature of the task. When solving urgent issues the best methods may be direct brainstorming or business games. In inventive creativity - types of brainstorming and synectic methods. In prognostic tasks - questionnaire methods, morphological analysis, etc.

The nominal group technique is based on the principle of limitations on interpersonal communication, so all members of the group gathered to develop a solution initially present their proposals in in writing independently, independently of others. Then each participant reports the essence of his project, the presented options are considered by group members (without discussion or criticism), and only after that each group member, independently of the others, presents in writing a ranking of the ideas considered. The proposals that receive the highest score are accepted as the basis for the decision. The peculiarity of this method and its advantage - despite working together members of the group, there is no restriction of individual thinking.

The smash attack technique can be used to detect flaws, erroneous conclusions, and conclusions in a study that is in the process of being completed. The meeting is attended by up to 50 people who are familiar with the working document that is the subject of discussion in advance. All participants speak in turn. The task of each speaker is to discover what is possible more disadvantages of work. The merits of the work and ways to eliminate the shortcomings are not mentioned. The duration of one performance is 1-3 minutes; it is forbidden to repeat shortcomings noted by other participants. Sometimes it is advisable to hold a discussion in two circles, providing the opportunity for repeated presentations by those who wish to clarify their comments.

The smash attack is a valid means of negative analysis. Therefore, the authors of the document under discussion should not be present at the discussion. Organizing a smash attack is similar to a direct brainstorming attack.

Synectic methods are based on the method of brainstorming, carried out by a special group of specialists distinguished by their flexibility of thinking and broad outlook. Such groups, accumulating techniques and work experience, achieve positive results in finding new technical solutions.

Synectics- this is a method of forecasting by analogy, transferring conclusions regarding certain characteristics from one subject to another. At the first stage, the leader sets the task and answers questions from group members. In the second stage, each participant proposes their own ideas, trying to look at problems from different points of view, thus overcoming “psychological inertia.” The leader selects one of them and forms its essence (key ideas). On the third _ participants search for analogies to the key idea using known facts from other areas of knowledge. In the fourth, the leader tries to apply some of the analogies and ideas proposed by group members to the task at hand. Next, an expert assessment of the chosen direction is carried out, and if the conclusion is positive, the work continues until a decision is made.

Morphological methods when developing alternatives, it is an approach to establishing typical logical connections and interdependence. Most often, this group of methods is used to identify possible options for technical, economic, and organizational solutions.

Under construction morphological table. The entire process of developing solution options in the table is presented in the area of ​​three fields. The first field - information - serves to designate all operations of collecting, pre-processing, storing and transmitting information. The analytical field represents operations related to the analysis of information and the selection of alternatives. In the target - conclusions, targets, limitations and requirements for the solution are formed. All operations occurring during the development and decision-making process are traced.

In practice, none of the methods is used in isolation from the others, obtaining effective solution is possible only through an appropriate combination of them.

Expert method.

1.4. Threshold values ​​are determined based on the opinions of expert groups.

The advantage of the method is simplicity and flexibility, since the assessment procedure allows you to analyze heterogeneous indicators in any quantity.

However, the assessment results largely depend on the qualifications and experience of the appraiser. Therefore, the defining moment and main problem This method is the selection of competent employees and/or the search for experts who will be tasked with assessing potential.

Expert method - This is a set of logical and mathematical-statistical methods and procedures associated with the activities of experts in processing the information necessary for analysis and decision-making.

IN practical activities on studying and forecasting demand method expert assessments can be used to solve the following problems: 1) Development of medium- and long-term forecasts of the group structure of demand for consumer goods. 2) Forecasting the intra-group (in the expanded assortment) structure of demand for the upcoming business year. 3) Identification of groups of potential consumers. 4) Assessment of the amount of unsatisfied demand by groups, types and varieties of goods.

An expert is a competent person who has deep knowledge about the subject or object of research.

How to form a group of experts? At the very first stage of selection, it is advisable to use two criteria as criteria: occupation and work experience in the profile of interest to us. If necessary, the level, nature of education, and age are also taken into account. The central criterion for selecting experts is their competence. To determine this, two methods are applicable, with varying degrees of accuracy: self-assessment of experts and collective assessment of the authority of experts.

The simplest and most convenient form of self-assessment of experts is a cumulative index, calculated based on the experts’ assessment of their knowledge, experience and abilities on a ranking scale with the positions “high”, “medium” and “low”. In this case, the first position is assigned the numerical value “1”, the second – “0.5”, the third – “0”. In this case, the cumulative index - the coefficient of the expert’s competence level is calculated using the formula.

The competence level coefficient ranges from 0 to 1. Usually, it is customary to include in a group of experts those with a competence index of at least the average of 0.5 and above –1.

The collective assessment method is used to form a group of experts in the case where they have an understanding of each other as specialists. This situation is typical for scientists, creative figures, politicians, and economists.

Forecast. It most clearly demonstrates the difference between expert assessment and information obtained as a result of a mass survey. It lies in the desire for consistency, uniformity of judgments and assessments expressed by experts. Indeed, is it possible to use for practical purposes, say, the opinion of thirty experts if they include 5-7 mutually exclusive prognostic assessments? Further, the larger the population of respondents, the higher the reliability of data in a mass survey, as well as some average statistical indicators. In principle, predictive expert assessment is feasible for any social processes and phenomena.

In applied sociology, a number of methods of expert surveys have been developed that are used to obtain a predictive assessment. At the same time, it is appropriate to note that some technical and methodological techniques widely used in mass surveys lose their significance when surveying such a specific audience as experts. As a rule, mass surveys are anonymous. In expert surveys, this loses its meaning, because experts must be fully aware of the tasks that are being solved during the study with their help. Therefore, there is no need to use indirect or control questions, tests or any other techniques in an expert questionnaire aimed at identifying the “hidden” positions of the respondent. Moreover, the use of such techniques can cause significant damage to the quality of expert assessment. An expert in the full sense of the word is an active participant in scientific research. And an attempt to hide the purpose of the study from him, thus turning him into a passive source of information, is fraught with the loss of his trust in the organizers of the study.

The main tool for an expert survey is a questionnaire or interview form developed using a special program.

Unlike a mass survey, the program of a predicted survey of experts is not so detailed and is predominantly conceptual in nature. First of all, it clearly formulates the phenomenon to be predicted and provides possible options for its outcome in the form of hypotheses.

Quite often used in applied sociology is such a method of expert forecasting as the “Dolphin technique”. It consists in developing consistent opinions by repeating a survey of the same experts many times. After the first survey and generalization of the results, its results are communicated to the participants of the expert group. Then a repeat survey is conducted, during which the experts either confirm their point of view or change the assessment in accordance with the majority opinion. This cycle contains 3-4 passes. During such a procedure, an assessment is developed, but the researcher, of course, should not ignore the opinion of those who, after repeated surveys, remained at their point of view.

Assessing the degree of reliability of the results of mass surveys. In production management decisions With the help of sociological research, the question often arises about the reliability of mass research and, accordingly, about the legitimacy of the conclusions formulated on their basis. In short, we are talking about assessing the competence of the opinions expressed by respondents.

For this purpose, an expert questionnaire is compiled, which includes mainly closed questions, identical in structure to the questions formulated in the respondent’s questionnaire. The expert’s task is, taking into account the objective situation and factors of interest to the researcher, to give an unbiased, comprehensively balanced assessment on the questions posed.

Certification of team members. IN recent years In the practice of studying the state of ideological and educational work, a type of expert assessment method called certification is widely used. In this case, team leaders take on the role of experts, public organizations or a special certification commission.

STEPS:

1. Statement of the problem

2. Technology for organizing expert analysis

3. Formation of an expert commission

4. Organization of an expert survey

5. Formal methods for describing object preferences

6. Formal methods for determining preferences

7. Mathematical methods processing of examination results

8. Assessing expert agreement

During management production system Situations constantly arise when managers different levels(from master to minister) are faced with the need to choose one of several possible options actions. Development and decision-making is a key procedure in the activities of a manager, which determines the entire further course of the management process, especially the final result of management activities.

Many factors that determine or influence the choice of solution are, by their nature, not amenable to quantitative characteristics, while others practically cannot be measured. All this made it necessary to develop special methods, facilitating the choice of management decisions in complex technical, organizational, economic problems (methods of operations research, expert assessments, etc.).

1.4. Ranking as method of collective expert assessments. The essence of the procedure is as follows: experts (specialists in the field under study) are asked to rank (arrange phenomena in ascending (descending) order of importance) of certain phenomena (price-forming factors). In this case, each phenomenon (factor) is assigned its own rank (ordinal place of the phenomenon (factor) in general sequence factors). Ranks are usually denoted by ordinal numbers natural series(1, 2, 3, 4……). In this case, rank 1 is assigned to the most significant factor.

The sum of ranks assigned by an expert for all factors can be calculated using the following formula:

ri – rank assigned to the i-th factor;

m – number of studied phenomena (factors).

The sums of ranks assigned by each expert must be equal. If an expert assigns equal ranks to two (three and so on) different phenomena, i.e. considers two phenomena (factors) to be equivalent (related ranks), then the sums of ranks assigned by each expert will not coincide

which in turn does not allow further calculations.

To avoid violation of the above equality, the appraiser conducting the examination needs to recalculate such ranks into the so-called standardized ranks. The calculation is carried out by dividing the sum of places occupied by related ranks by their number. An example of the calculation is presented in Table 1.

As a result, we obtain a normal ranking for which the main ranking conditions (1) are met.

Next, for each phenomenon, the sum of the ranks assigned to each expert is calculated. The phenomenon (factor) that received the smallest sum of ranks is assigned a corresponding rank of 1, etc. If at this stage for some phenomena the sums of ranks coincide, then it is necessary to carry out the procedure of standardizing the ranks again, but according to the ranks obtained by summing up the expert assessments. An example of ranking is presented in Table 2. It should be noted that the number of experts (d) must be at least 1 greater than the number of factors under study (m).

Factors
Expert								sum
Standardized ranks
		3,5	3,5
			4,5		4,5
			4,5		4,5
Sum of ranks (r1)		23,5	31,5
Factor ranks (R)
Standardized final ranks	1,5			1,5		6,5	6,5

Obtaining ranks allows for further calculations of weighting coefficients for each pricing factor when making adjustments in qualimetric models and other methods of the comparative approach. However, the final conclusion about the possibility of using the collective opinion of experts is possible after determining the consistency of expert opinions. The consistency of opinions is checked using the coefficient of concordance (agreement). The coefficient is calculated using the following formula:

Where,
m – number of factors being assessed;
d – number of experts;
Ri – sum of ranks for the i-th factor (phenomenon);

The closer the coefficient value is to 1, the higher the consistency. The acceptability of the obtained expert assessments is determined by comparing the significance criterion x2fact=d*(m-1)*W with the table value x2tab. with degrees of freedom (m-1) and a given probability (for example: Po = 0.05, P=1-Po). If the value obtained by calculation is equal to or higher than the table one, then the concordance coefficient is significant and with a reliability of 0.95, the opinions of the experts are consistent. Table values ​​of the significance criterion are presented in Table 3.

m-1	P0		m-2	P0
0,05	0,01	0,001	0,05		0,001
	3,84	6,63	10,83		26,3		39,25
	5,99	9,21	13,81		27,59	33,41	40,79
	7,81		16,27		28,87	34,8	42,31
	9,49	13,28	18,46		30,14	36,19	43,82
	11,07	15,09	20,52		31,41	37,57	45,31
	12,59	16,81	22,46		32,67	38,93	46,8
	14,07	18,47	24,32		33,92	40,29	48,27
	15,51	20,09	26,12		35,17	41,63	49,73
	16,92	21,67	27,88		36,41	42,98	51,18
	18,31	23,21	29,59		37,65	44,31	52,62
	19,67	24,72	31,26		38,88	45,64	54,05
	21,03	26,22	32,91		40,11	46,96	55,48
	22,37	27,69	34,53		41,34	48,28	56,89
	23,68	29,14	36,12		42,56	49,59	58,3
		30,58	37,7		43,77	50,89	59,7

x2fact = 44.016 > x2table = 12.59

Thus, the expert opinions given in our example are quite consistent and can be used in further calculations.

A weighting factor can be assigned to each factor using the following formula:

ri – final factor rank.
In our example, the weights would be distributed as follows (Table 4).

Factors
								sum
Rank	1,5			1,5		6,5	6,5
Weight	0,232	0,179	0,143	0,232	0,107	0,054	0,054

The presented method of distributing weights does not pretend to be objective, and has a number of disadvantages, in particular: the distribution of weights is based on the ranking of factors, which in turn allows factors to be distributed only with a given interval, i.e. the method gives the answer that factor “A” is more important than “B”, and “B” is more important than “C”. However, according to this, factor “A” is twice as important as factor “C,” which may not be true.

As a rule, expert methods are most widely used in the study of control systems. How scientific way The expert method was developed relatively recently and for the first time it was called “Delphi”. Subsequently, other similar methods were developed, based on expert assessments. At first, expert methods were used mainly to solve problems related to forecasting in the field of science and technology, and then they began to be used in other areas, including management.

The essence of expert methods, both when solving problems of CS research and when using them in decision-making practice in other fields of science, technology, and management, lies in averaging in various ways the opinions (judgments) of expert specialists on the issues under consideration.

The most common expert methods for classification based on preference assessment at present when making management decisions are the following:

• rank method;
• direct assessment method;
• comparison method.

The latter method includes two varieties:

paired comparison and sequential comparison.

In principle, each of them has much in common, and the only difference is that the assessment (measurement) of the studied system management objects is carried out in different ways. Moreover, each method has certain advantages and disadvantages.

The commonality of each method lies in the sequence of procedures for their use. These include:

• organization of expert assessment;
• collecting expert opinions;
• processing the results of expert opinions.

Practice shows that reducing subjectivity and, accordingly, increasing the objectivity of the results of using expert methods significantly depends on compliance with the rules of organization, preparation and conduct of expert work. This especially depends, first of all, on the organization of expert assessment, the appointment of a person responsible for organizing and conducting expert assessment work, as well as on the formation of expert commissions.

For general management of expert work, a chairman of the expert commission should be appointed. The commission consists of two groups: working and expert.

The working group is headed by its leader (organizer). His subordination includes technical workers who carry out technical work on preparing materials for the work of experts, working out the results of the experts’ work, etc.

The expert group includes experts - specialists in the problems being solved.

The formation of the expert panel is carried out by the head (organizer) working group. In this case, a number of sequential activities are performed:

• problem statement and definition of the group’s area of ​​activity;
• compiling a preliminary list of experts - specialists in the field of activity under consideration;
• analysis of the qualitative composition of the preliminary list of experts and clarification of the list;
• obtaining expert consent to participate in the work;
• compilation of the final list of the expert group.

The number of experts in the expert group depends on many factors and conditions. In particular, on the importance of the problem being solved, available capabilities, etc. In most cases, it is determined minimally required quantity experts, which often becomes the most important condition for determining the number of invited experts.

The selection of specific experts is carried out based on an analysis of the quality of each of the proposed experts. Various methods are used for this purpose:

• assessment of candidate experts based on statistical analysis of the results of past activities as experts on organizational design problems;
• collective assessment of the candidate expert as a specialist in this field;
• self-assessment of the candidate expert;
• analytical determination of the competence of candidate experts.

Very often several methods are used simultaneously. For example, methods of self-assessment and collective assessment of the qualities of the proposed expert. This approach makes it possible to reasonably select experts with the necessary qualities. However, it should be recognized that the method of assessing past performance seems to be more objective than the methods of self-assessment and collective assessment.

Regardless of the chosen method of assessing the qualities of candidates, experts must in all cases satisfy the following requirements:

• professional competence in the field of design of organizational systems;
• creativity (ability to solve creative problems);
• scientific intuition;
• interest in the objective results of expert work;
• efficiency (composure, ability to switch from one type of activity to another, communication, independence of judgment, motivation of actions);
• objectivity;
• nonconformism.

Gathering expert opinions involves resolving the following issues:

• determining the place and time for collecting opinions;
• determining the form and methodology for collecting opinions;
• determining the number of rounds of collecting opinions;
• determination of the composition and content of the documentation;
• determining the procedure for recording the results of expert opinions in documents.

It is very important to determine the form for collecting expert opinions. Among all the known forms of collecting opinions, we can note individual, collective and mixed, i.e. These forms differ primarily in the factor of expert participation in the work (individual or collective). Each of these forms has a number of varieties:

• survey;
• interviewing;
• discussion;
• brainstorming;
• meeting;
• business game.

They all have their own advantages and disadvantages. In many cases of organizational design, each of these varieties is used together, which often gives great effect and objectivity. This approach to collecting expert opinions, that is, when a mixed form is used, is used in cases of some uncertainty about the problem, disagreement of individual opinions, or disagreement of experts during a collective discussion.

At the same time, most often in the practice of designing organizational systems, questionnaires are used, which allows experts to collect their opinions with less labor, but in terms of time, collecting opinions when using this type is longer.

Typically, the questionnaire development process includes:

• determining the form and content of an appeal to an expert;
• choosing the type of questions;
• formulation of questions;
• presentation of the information necessary for the expert;
• development of the questionnaire form.

It is interesting to choose the types of questions, among which the most used in recent years are the so-called fan, closed and open types(fan - presupposes one answer from a number of answers presented in advance in the questionnaire; closed - “yes”, “no”, “I don’t know”; open - a question the answer to which can be given in any form).

When questioning experts, it is very important to formulate questions correctly, simply and unambiguously, briefly and at the same time with the necessary completeness in the questionnaires, and in the text explanatory note indicate what exactly is required from the expert.

To answer questions, that is, to make a decision by each expert, objective and (or) subjective measurements of the object in question are carried out in an explicit or implicit form. In subjective measurement, experts, as a rule, use one of the previously mentioned most commonly used methods (ranks, direct assessment, comparisons).

Using the rank method, the expert performs ranking (ordering) of the studied objects of the organizational system depending on their relative importance (preference). In this case, usually the most preferred object is assigned rank 1, and the least preferred one is assigned the last rank, equal in absolute value to the number of objects being ordered. This ordering becomes more accurate with fewer objects of study and vice versa.

Thus, this method makes it possible to determine the place of the object under study among other CS objects. The advantage of the rank method is its simplicity. The disadvantages are:

• the inability to rank with sufficient accuracy the number of objects whose number exceeds 15-20;
• does not answer the question of how far in importance the objects under study are from each other.

This method is used in the practice of studying control systems, despite its simplicity, quite rarely.

The direct assessment method is the ordering of the objects under study (for example, when selecting parameters to compile a parametric model) depending on their importance by assigning points to each of them. In this case, the most important object is assigned (evaluated) greatest number points on the accepted scale. The most common rating scale range is from 0 to 1, 0 to 5, 0 to 10, 0 to 100. In the simplest case, the rating can be 0 or 1. Sometimes the rating is carried out in verbal form. For example, “very important”, “important”, “unimportant”, etc., which is also sometimes translated into a point scale (3, 2, 1, respectively) for greater convenience in processing survey results.

The use of this method is used only if the experts are confident that they are fully informed about the properties of the object under study, which often does not happen.

The comparison method is carried out, as mentioned earlier, by pairwise comparison and sequential comparison.

In pairwise comparison, the expert compares the objects under study according to their importance in pairs, identifying the most important one in each pair of objects. The expert presents all possible pairs of objects in the form of a record of each combination (object I - object 2, object 2 - object 3, etc.) or in the form of a matrix.

As a result of comparing the objects in each pair, the expert expresses an opinion about the importance of one or another object, that is, he gives preference to one of them. Sometimes experts come to the conclusion that each of the objects in the pair is equivalent. Ordering in each pair of objects, of course, does not immediately provide ordering of all the objects under consideration, so subsequent processing of the comparison results is necessary. It is most convenient to carry out pairwise comparisons and their processing using matrices as a tool.

In some cases, with a large number of objects under study, the results of paired comparisons are influenced by psychological factors, that is, preference sometimes goes not to the object that is actually preferable to others, but to the one that is written first in the list of pairs or is located in the matrix above the one being compared. Therefore, sometimes, to exclude psychological influence, a double pair comparison is carried out, that is, a pair comparison is made again, but only with the reverse arrangement of objects and, accordingly, the objects in each pair.

The method of paired comparisons is very simple and it allows you to study a larger number of objects (compared, for example, with the rank method) and with greater accuracy.

The essence of the sequential comparison method is as follows. The expert arranges all studied objects in order of their importance (like the rank method). Previously, each of the objects is assigned a certain amount points, for example, on a scale from 0 to I (as an assessment method). Moreover, the most important object is given a score equal to I, and all others are given in decreasing order of their importance, that is, from I to 0. Next, the expert decides whether the importance of an object with rank I will be greater than the sum of the points of all other objects. If so, then the score of the first object increases until this condition is met, and if not, then the expert reduces this value to such a numerical value that it becomes less than the sum of the scores of all other objects.

The values ​​of the ratings of the second, third and subsequent objects in importance are determined sequentially in the same way as the rating of the first most important object.

The sequential comparison method is the most labor-intensive for experts. This is especially noticeable when the number of objects being studied is more than six or seven.

The processing of collected expert opinions is carried out both quantitatively (numerical data) and qualitatively (substantive information). Various methods are used for this. It should be noted that in the presence of numerical data, to resolve issues with sufficient information material, methods of averaging expert judgments are mainly used. However, even with available numerical data, but if there is insufficient information on the issue at hand (which often happens when designing a control system), along with quantitative methods for processing expert data, methods of qualitative analysis and synthesis are also used.

When using the considered expert methods (ranks, etc.), the opinions of experts often do not completely coincide. Therefore, it is necessary to quantitatively evaluate the degree of consistency of expert opinions and determine the reasons for the discrepancy in judgments. The measure of consistency is, of course, determined on the basis of statistical data from the entire group of experts.

To assess the measure of consistency of expert opinions, as a rule, concordance coefficients are used - dispersion and entropy.

The dispersion coefficient of concordance takes values ​​from 0 to 1. At 0, there is no agreement between expert opinions; at 1, there is complete agreement. If the dispersion coefficient of concordance is greater than 0.5, then the consistency is usually considered sufficient.

The entropy coefficient of concordance (otherwise called the coefficient of agreement) also varies from 0 to 1 and also with a larger value of the coefficient of agreement - a greater measure of consistency.

In cases where expert opinions differ slightly, the above coefficients approximately give the same measure of agreement. However, if there are significant differences in the opinions of experts, then the values ​​of the coefficients will differ significantly. Thus, joint analysis of coefficients allows us to objectively determine the degree of consistency of expert opinions.

The application of all the considered expert methods, despite their shortcomings, shows their effectiveness in research and design of control systems. Moreover, the greatest effect is achieved by using several methods simultaneously.

A type of expert method can be classified as sociological, which is based on a survey, collection and analysis of the opinions of respondents (for example, actual or potential consumers). Such a survey and collection of opinions is usually carried out in in writing- distribution of questionnaires or orally (at conferences, auctions, exhibitions, educational institutions, etc.). When using this method, you should also apply scientifically based survey methods, mathematical principles of collecting and processing information.

Processing expert and sociological data and calculating consistency measures require labor-intensive calculations. Therefore, it should be used more widely when collecting and processing the results of expert and sociological information. computer equipment. There are opportunities for this, since the automation of conducting and processing the results of this kind of data has become the subject of the creation of a number of software products.

I. Astashkina, V. Mishin

Control systems research,

Expert research method

Expert and analytical research methods

The main methods included in the group of expert analytical ones are the following methods (Table 3.3):

· expert;

· examinations;

· diagnostic;

· graphic.

Let's take a closer look at these methods.

The expert method operates with intuitive information transformed by imagination and this is most suitable for MIS, since this area of ​​knowledge is dominated mainly by quality criteria performance assessments. This research method should be applied to all types of management. There are several modifications of the method, but the Delphi method is most often used. The structuring of the expert research method (basic categories) is shown in Fig. 4.2.

Rice. 4.2. Structuring the expert research method (main categories)

The most significant area of ​​application of the expert research method seems to be the area of ​​correctness of delimitation of areas of responsibility (sectors of responsibility or functions) between managers and divisions horizontally. Particularly important in this regard are knowledge of the principles of developing questionnaires, the use of hidden, indirect questions, etc.

Predictive expert assessments reflect the individual judgment of specialists regarding the prospects for the development of the object of study and are based on the mobilization of professional experience and intuition (Meskon, 1993).

Conditions of use intuitive (expert) research method:

1) lack of sufficiently representative and reliable statistics on the characteristics of the object;

2) uncertainty of the state of the external environment;

3) medium- and long-term forecasting of new markets, objects of new areas of industry, strongly influenced by discoveries in fundamental sciences (for example, the microbiological industry, quantum electronics, etc.);

4) lack of time or funds allocated for research and decision-making, lack of opportunities to use other suitable methods in this regard;

5) force majeure situations.

Degree of reliability of the examination established by the absolute frequency with which the expert's assessment is ultimately confirmed by subsequent events.

Requirements for an expert include the following provisions:

1) high level general erudition;

2) the expert’s assessments must be stable over time and transitive;

3) availability additional information about predicted signs only improves the expert’s assessment;

4) the expert must have certain practical and (or) research experience and be a recognized specialist in this field of knowledge;

5) the expert must have a psychological orientation for the future;

6) it must be capable of adequately reflecting the development trends of the object under study;

7) the expert should not be interested in the specific result of the study.

When characterizing experts, it should be borne in mind that as a result of developing assessments, there may be two types of errors:

1) systematic,

2) random.

To correct systematic errors, you can apply correction factors or use specially designed training games. Random errors vary from one expert assessment to another and are characterized by the amount of dispersion. In research practice, it is very difficult to evaluate them.

Special training games are a means of training experts.

Organization of expert work forms must be programmed or unprogrammed, and the expert’s activities can be carried out orally (interview) or in writing (answers to questions from special tables of expert assessments or free presentation on a given topic) (Sarkisyan, 1977).

Organization and stimulation of the expert’s work consists in developing heuristic techniques and methods that facilitate the search for expert assessment, legal norms that guarantee the expert registration of priority in authorship. It is important to non-disclose all scientific and technical ideas put forward by the expert in the process of work, forms of moral, professional and material interest of the expert in expert assessments, organizational forms of the expert’s work (inclusion in the work plan, etc.) (Meskon, 1993).

When solving the problem of forming an expert group, it is extremely important to identify and stabilize a workable network of experts. Method for stabilizing an expert network is as follows (Dobrov, 1969). Based on an analysis of the literature on the problem under study, any specialist who has several publications in this field is selected. They ask him to name the 10 most competent, in his opinion, specialists on this problem. Next, they simultaneously turn to each of the ten named specialists with a request to indicate the 10 most important of their fellow scientists. From the resulting list of specialists, the initial 10 are deleted, and the rest are sent letters containing the above request.

This procedure is repeated until none of the newly named specialists can add new names to the list of experts, ᴛ.ᴇ. until the network of experts stabilizes. The resulting network of experts can be considered a general population of specialists competent in the field of research of the problem. At the same time, due to a number of practical limitations, it turns out to be inappropriate to involve all specialists in the examination. For this reason, it is extremely important to form a representative sample from the general population of experts (Bestuzhev-Lada, 1982). This issue is resolved within the framework of each specific method.

In order to minimize research costs, they strive to involve a minimum number of experts N min, provided that the error of the research result is no more than b, where 0

Nmin = 0.5(3/b + 5). (4.1)

In this case, stabilization of the average assessment of the predicted characteristic should be observed. The achievement of this stabilization is evidenced by the fact that the inclusion or exclusion of an expert from the group does not change the relative assessment of the desired value by more than b.

A questionnaire for surveying specialists is the most important tool for expert forecasting. Preparation and conduct of the examination includes the development of questionnaires containing a set of questions on the object of study. Structurally, the set of questions in the questionnaire should be logically related to the central task of the examination. The content of the questions is determined by the specifics of the research object and its methodology. Τᴀᴋᴎᴍ ᴏϬᴩᴀᴈᴏᴍ, system of questions in the questionnaire must meet the following requirements (Glushchenko, 1997).

1) formulation in generally accepted terms;

2) exclusion of any semantic uncertainty;

3) focus on ensuring the achievement of research goals;

4) compliance with the structure of the research object;

5) ensuring a uniform and unambiguous interpretation of the survey results.

The form of the survey questions is:

· open and closed;

· direct and indirect.

The most commonly used method is interviews and analytical expert assessments.

Interview method involves a conversation with an expert, during which the researcher, in accordance with a pre-developed program, poses questions to the expert regarding the prospects for the development of the object under study. The success of such an assessment largely depends on the psychological ability of the expert to give impromptu conclusions on various issues, incl. fundamental issues (Sarkisyan, 1969). A well-known disadvantage of this method is the significant psychological pressure on the expert.

Methods of collective expert assessments are based on the principles of identifying the collective opinion of experts on the prospects for the development of the forecast object.

At the root of the use of these methods is the hypothesis that experts have the ability to assess with a sufficient degree of reliability the importance and significance of the problem under study. There are a large number of modifications of collective expert assessment methods, among which the following methods are the most popular:

· round table;

· ʼʼDelphiʼʼ;

· software forecasting;

· heuristic forecasting;

· collective generation of ideas.

The collection and processing of individual expert opinions is carried out based on the following principles (Glushchenko, 1997):

1) questions in the questionnaires are posed in such a way that it is possible to give a quantitative description of the experts’ answers;

2) the survey of experts is carried out in several rounds, during which questions and answers are increasingly refined;

3) all interviewed experts are familiarized with the results of the survey after each round;

4) experts justify assessments and opinions that deviate from the majority opinion;

5) statistical processing of responses is carried out sequentially from work to work in order to obtain general characteristics.

Moreover, the prevailing judgment of specialists on any issue is revealed in an environment that excludes their direct debate among themselves, but at the same time allows them to periodically weigh their judgments taking into account the answers and arguments of their colleagues. Revision and the possibility of changing one’s previous assessments based on clarification of the considerations of each of the experts and the subsequent analysis by each participant of the totality of reasons presented by the experts stimulate respondents to take into account factors that they at first tended to omit as insignificant.

Delphi method of expert assessments was developed by the American researcher O. Helmer to solve complex strategic problems in order to obtain wider sources of extremely scarce information about the future, to completely eliminate the subjective factor in judgments and assessments of the future, to stimulate the ways of thinking of specialists by creating a special information system with feedback, to eliminate interference in the exchange of information between specialists, pressure from authority and other forms of pressure, to ensure increased reliability of forecasts through special procedures for quantitative assessment of expert opinions and their machine processing.

In contrast to the scenario method, the Delphi method involves preliminary familiarization of the involved experts with the situation using some model: such a model should be both a strict mathematical model, for example, an econometric model of economic development, and an informal description of the process, for example, a scenario. In systems analysis, the main form of the model, which is subject to improvement and saturation with information using expert assessments, is, as a rule, a tree of goals.

The specialist is asked to evaluate the structure of the model as a whole and make proposals for including unaccounted connections in it. In this case, a questionnaire method is used with unified forms of questions, answers and assessments. The results of each stage of the survey and the systematization of its results are again brought to the attention of all experts, which allows them to further adjust their judgments based on the newly received information. The information obtained is further compared and analyzed based on the use of data by quantitative assessment experts.

There are a number of procedures for averaging and objectifying the opinions of experts, as well as a number of procedures for assessing the authority and relative weight of the opinions of the experts themselves, for example, their mutual assessment of competence in any field. In cases where objective data are not available, which applies especially to information about the future, the Delphi method seems to be the most reliable means of obtaining comprehensive and at the same time sufficiently reliable data. In the MIS, the Delphi method is used at stage VI (Table 3.1) to assess the current state of those factors that cannot be directly quantified (for example, assessing modern social factors influencing the formation of goals), at stage VII in assessing one of the most important methods for obtaining and processing forecast information.

Morphological analysis- a means of studying all possible combinations of organizational solutions, for example, those proposed for the implementation of individual management functions. If we write down all the functions in a column, and then next to each function we indicate, line by line, all possible options for its implementation, we will obtain a morphological matrix (Table 4.1). The idea behind this method is essentially to break a complex problem into small subtasks that are easier to solve individually. And functions are nothing more than a set of tasks, works and procedures.

Table 4.1

Morphological matrix of ways to perform management functions

Legend:

F – function;

n – serial number of the function;

P – method of implementing the function;

k – serial number of the method of implementing the function.

The most famous varieties of the method are:

method of systematic field coverage (MSPP);

method of denial and construction (DEC);

morphological box method (MBM), etc.

Sociological research are widely used in practice to study problems in the information-behavioral and structural-functional subsystems associated with specialists and managers, in choosing the direction of action, increasing involvement in the affairs of the organization, interest in the implementation of plans, etc. For example, you can find out whether there are any functions, tasks, works and procedures undistributed between departments, managers or specialists.

Sociological research is carried out by collecting and processing information about the needs and interests of the organization’s specialists, the nature of interpersonal and intergroup relationships, the type of organizational culture that has developed under the influence of the modern composition of personnel in the organization, and other factors. For these purposes use:

· interview;

· questionnaires;

observation and self-observation;

· study of documents;

· study of factors of group behavior, etc.

The results obtained during the study make it possible to formulate measures and actions to improve the efficiency of the functioning of the IPP, SFP and their elements.

Scripting method is a means of primary ordering of the problem and a means of obtaining and collecting information about the relationship of the problem with other problems and about possible and likely directions for future development. The method is named after the well-known and carefully developed technology of script writing. Based on an iterative sequence of scenario construction in order to select a suitable control alternative (Fig. 4.3).

Principles, underlying the scripting method:

· consistent resolution of uncertainty (iterative process);

· research into development scenarios.

Purpose of the method– obtaining a scientifically based forecast for making management decisions.

Scenario- ϶ᴛᴏ a hypothetical picture of the sequential development in time and space of events that make up the evolution of control objects.

Rice. 4.3. Script Method Logic

When constructing scenarios, the following concepts are used:

1) internal and external factors:

· internal factors – the internal structure of the system, the relationship between its elements, the patterns of its development;

· external factors – interaction of the system with a wider class system (macro- and microenvironment).

2) scenario parameters - unpredictable factors affecting the system;

3) limitation (each system is within the framework that limits its state - equifinality):

· natural (determined by the natural environment in which the system is located, for example, climatic conditions, etc.);

· normative (related to the legal environment and all possible norms of behavior - culture, customs, traditions, morality, religion, ethics).

4) scenario indicator – a parameter critical to the limit states of the control object.
Posted on ref.rf
This is a parameter that limits the state of the system in certain directions (which threshold cannot be crossed so as not to cause destruction of the system);

5) procedure of the scenario method - determination of the guaranteed forecast, ᴛ.ᴇ. boundaries beyond which the development of the system cannot go beyond for objective reasons.

Scenarios can be used at a variety of stages of the MIS, when it is necessary to collect and organize very heterogeneous and unstructured information. But the main area of ​​application of this tool is stage 1 of system analysis - problem analysis and stage VII - forecast and analysis of future conditions (Table 3.1).

Expert research method - concept and types. Classification and features of the category "Expert Research Method" 2017, 2018.