Performance Evaluation with PROMETHEE GDSS and GAIA : A Study on the ITA-SAT Satellite Project

The increased complexity of organizational contexts in the end of the 20th century and beginning of the 21st century has resulted in the increment of the variables number to be considered during a performance evaluation. Therefore, many professionals began to search methods that would enable the simultaneous analysis of multiple criteria. Thus, this paper has investigated the application of a multicriteria approach to evaluate the performance in the development of subsystems during the development of complex products, by using the PROMETHEE GDSS and the GAIA methods. To achieve this objective, a study on the ITA-SAT satellite project was performed, in which a perception analysis of the members of the project regarding the development of each subsystem was carried out. Once these perceptions were captured, the proposed approach was used to identify subsystems with less satisfactory performances. Finally, the results were analyzed and brought to discussion at a meeting with managers and project members with a feedback to the conclusions of this study.


MCDA
Most decision-related problems consist of issues involving contradictory multiple criteria, which represent one of the characteristics of a problem approached by MCDA.Since there is rarely one action that can simultaneously present the best performance in the considered criteria, the MCDA can criteria (Zeleny, 1982).
It involves a set of methods that can be found in literature with different names, such as: Multi-Attribute Decision Making (MADM), Multi-Criteria Decision Making (MCDM), Multi-Objective Decision Making (MODM) or Multiple Objective Decision Aiding (MODA) (Gomes and Gomes, 2001).
According to Roy (1996), MCDA can be used to face four kinds of decision-related issues, as follows: among the ones being taken into account; into classes, according to the performance; according to the performance; problem by describing it.methods PROMETHEE GDSS (Macharis et al., 1998) and GAIA (Hayes et al., 2009), which were used in this study.

PROMETHEE GDSS
The PROMETHEE GDSS belongs to the outranking methods, also understood as subordination, prevalence, or outranking.Such methods aim at building an outranking relation to represent the preferences of the decision-makers and to According to Le Téno and Mareschal (1998), the basic principle of outranking methods is that if one action performs better than another in most of the criteria and it does not presbe the chosen one.Besides, these methods consider that small impact on the decision-maker (Vincke, 1992).
According to Silva et al. (2010), one of the advantages of the PROMETHEE method (I, II, III, IV, V, VI, and GDSS) is how each criterion is assessed by the preference functions.This leads to a more reliable understanding of perception differences among the decision-makers at evaluating each action, and it also does not allow an unlimited compensation of major advantages between the actions.
For Brans and Mareschal (2005), two kinds of information are necessary to use the PROMETHEE method: between criteria and within each criterion.
Information between criteria is comprised of the weight assigned to each of them.According to Macharis et al. (2004), with the PROMETHEE method, it is assumed that decisionmakers are capable of properly evaluating the criteria weights without using any additional methods when the number of criteria is not high, in a way that the sum of weights is equal to one.
However, information within each criterion refers to the preference functions related to each criterion and its parameters (Chart 1), in which "q" represents the threshold of indifference, "s" represents the threshold of strict preference, and " " represents a value between "q" and "s" (Brans and Mareschal, 2005).
ence functions for each criterion (as well as the parameters), the calculation of deviation modules d k (a i , a j ) between judgments f k (a i ) and f k (a j ) for each pair of actions "a i " and "a j " wee performed, according to each considered criterion "k" and to the equation: d k (a i , a j ) = f k (a i ) -f k (a j ).The results of such equa- et al. (2010), for each value resulting from d k (a i , a j ), a preference function is used that translates the difference between the evaluations obtained by two actions "a i " and "a j " over a criterion k in a value of p k (d k ), which ranges tions are called intensities of preference, and they are the basis with which an action "a i " is preferred over an "a j ", considering k of each criterion "k" for the "n Belderrain, M.C.N., Gonçalves, T.J. Chart 1. Judgment scale.

Verbal scale Numeric scale
Very good (VG) 5
, a m a a , a m a a represents the intensity with which an action is chosen over which an action is overcome by others.et al., 2011).
GDSS can be conducted.According to Macharis et al. (1998), in the PROMETHEE each of them, as demonstrated in Fig. 1 for four decisionmakers and seven actions.
resulting in the group evaluation (Brans and Mareschal, 2005).

GAIA
One of the advantages to use the PROMETHEE family methods is the possibility of interpreting geometrically the results by the GAIA method (Mareschal and Brans, 1988).criterion "l" is calculated according to Eq. 5, where "a i " represents the action "i lated (Hayes et al., 2009).
row, an action.
Using the biplot method (Kohler and Luniak, 2005) in , the actions and criteria are projected in a plane shaped In GAIA plane, actions are represented by points and criteria by vectors.Moreover, the vector of weights W( 1 2 n ) in the k-dimensional space can be projected on the GAIA plane, as demonstrated in Fig. 2. The projection of the weight vector on the GAIA plane is called According to Alencar and Almeida (2010), the GAIA aspects concerning the criteria and about the impact of decision-makers one the issue.Brans and Mareschal (2005) state that the analysis of the GAIA plane helps at understanding the structure of the problem: Performance Evaluation with PROMETHEE GDSS and GAIA: A Study on the ITA-SAT Satellite Project indicate more variable criteria; directions near to that of the decision stick; by vectors in orthogonal directions; one another; and are represented by points located in the direction of the vector concerning the criterion in question.
In order to meet the objectives of this study, this approach was applied to evaluate the development of the ITA-SAT satellite project.

ITA-SAT SATELLITE PROJECT
The ITA-SAT satellite project is part of the pluri-annual plan of "Desenvolvimento e Lançamento de Satélites Tecnológicos de Pequeno Porte", concerning the development and launch of small technological satellites.It was created by Action number and test technological innovations of satellites and to enhance concerning micro and nano-satellites.
This activity involves AEB, the National Institute for Space Research (INPE), and the Technological Institute of Aeronautics (ITA).Other upper education institutions may be associated with the project by means of a cooperation term with ITA.
The ITA-SAT project was subdivided into stages, according to the Standards of the European Space Agency: (0) mission each stage can be observed in the standard ECSS-M-ST-10C of the European Space Agency (ESA) (ESA, 2009).
The ITA-SAT project began in 2009, and stages 0, A and B are concluded, with their respective revisions, Mission (PRR), System Requirements Review (SRR), and Preliminary Design Review (PDR).Details may also be seen in the standard ECSS-M-ST-10C of the ESA.
During the performance of this study, the ITA-SAT satellite project was facing many problems, including project.The types of activities that should be performed were not previously known, so the scope was periodically more knowledge on the product that was being designed.Concerning deadlines and objectives, the pace of development in the project was different in several subsystems, and many activities were on hold because they depended on the conclusion of others, which were not concluded on time.Also, the communication and documentation of the of the knowledge that resulted from an activity was lost when a member of the team left the project.Such problems equipment in the ITA-SAT satellite project.(Sato et al., 2011) 0 W G A IA pl an e Figure 2. Vector of weights projected on GAIA plane.(Brans and Mareschal, 1994).
ITA-SAT satellite project in order to search for ways to improve, solve, or ease those issues.

Identi cation and analysis of the decision conte t
The ITA-SAT satellite project, whose preliminary of subsystems for attitude control (AC), on-board computer (OBC), potency (PT), thermal control (TC), structure (ST), and telecommand and telemetry (TT).
The functions of these subsystems, according to Larson and Werts (1999) apud Sato et al.

AC
It controls the direction the satellite should be pointed at, since it is necessary that some of its instruments are pointed at

OBC
It controls internal communication, the satellite, and on-board data processing.Two types of data are processed by the active subsystems, temperature etc., which are used by the ground segment to work on the subsystems' function.Scienstudies, such as images and radiation measurements.Due to the different nature of the activities in subsystem OBC, we divided the subsystem into OBC -Software (OBC-S) and Hardware (OBS-H).

PT
It generates, stores, conditions, and distributes energy for the satellite subsystems.

TC
It ensures that all subsystems and components of the satelproject.

ST
It represents the mechanical connection between the different subsystems and components in the satellite.It should be able to sustain the efforts during launch and at the moment of uncoupling the launch vehicle, consisting of a surface on which the equipment is put together.Also, it should provide protection against radiation, it should have a coupling interface with the launch vehicle, besides being in charge of grounding for thermal and electrical ends.

TT
and the ground control station.As to decision-makers, four representatives were selected among the participants of the project.The general manager considered them as those who had more knowledge on the subject.
Structuring the problem and the multicriteria model lite, many traditional management methods established in literature, such as those mentioned in the Project Management Body of Knowledge (PMBOK) (PMI, 2008), were not used in this project.Therefore, no development indicators were prepared when this research was carried out.
Furthermore, the indicators used to assess each subsystem were obtained by literature analyses on management development indicators of projects.We have chosen to use the ones described in PMBOK and those by Terribili Filho (2010).

Time management
necessary resources, to develop, and to control the project timetable.
Communication effectiveness make sure the information is properly generated, collected,

Risk management
well as response planning, monitoring, and risk control.
Integration management combine, gather, and coordinate the processes of different subsystems that compose the project.

Cost management
estimate, budget, and control costs in order to conclude the Quality management that the needs for which the project was initiated could be met.
Among the mentioned criteria, we observed that since this the necessary technologies should be developed.This fact did not allow us to predict the aspects related to cost or to the quality of the project, since we could not tell which type of Because of that, the cost management criteria, as well as the In this study, we tried to evaluate the performance based on the perception of the project's team, using a judgment scale et al.
good), and it is detailed in Chart 1.
In order to determine the preference functions, the ones demonstrated in Chart 2 were presented to the decisionmakers, and the usual function was considered as the most adequate for the used criteria and by the decision makers in this study.During the interviews, they said that, according to the adopted scale, any difference between judgments was the preference functions, each decision-maker was asked to ponder the criteria.Besides, the general manager was asked to ponder the knowledge of each decision-maker concerning the

Individual analysis
With the objective of evaluating the development performance of the project's subsystems, individual evaluations with the perceptions of each decision-maker were PROMETHEE method, the verbal and numeric scales in Chart 1 were presented.Afterwards, every decision-maker was asked to think of an alternative that represented, in their Based on this hypothetical alternative, each decision-maker had to answer how much better they considered the ITA-SAT satellite project in the evaluated criterion.Many of them answered they had pictured the ITA-SAT project as a 'very poor' alternative, thus obtaining one in the numeric scale.In other occasions, they said the ITA-SAT project performed twice, three times or four times better than the very poor hypothetical alternative.It is important to mention that no decision-maker considered the performance of the ITA-SAT thetical alternative.With these questions, the evaluations of the decision-makers about the performance of each subsys-After collecting judgments, the subsystems were individually assessed with the software D-Sight (Macharis et al., 2010).each decision-maker DM i (i=1,2,3,4) are demonstrated in Table 1, with the order of priority of each subsystem.It is number corresponding to the order of priority was repeated for the tied subsystems.
Each decision-maker analysis concerning the GAIA plane is presented in Chart 3. In these planes, vectors represented the actions; and the vector with the circular For decision-maker 1 (DM 1 ), the analysis of the GAIA plane showed that criteria with more heterogeneous judgments were: integration management (IM), communication effectiveness (CE), and time management (TM).Also, time management (TM) and communication effectiveness (CE) were observed.This means that according to DM 1 , the criterion did not have good results in the last two ones, and vice-versa.

Subsystems
1 as related to resource allocation.The subsystems have limited resources to invest in managerial aspects related to each criteas to the adoption of managerial evaluations; this is why the performances differ in the subgroups.pointed to the right.Besides, the subsystems AC, ST and PT are always located in an opposite direction to that of the (judgments of lower value) and, consequently, higher criticality, which can be observed in three out of the four orders mate directions, as well as those referring to CE and TM, criteria are compared.
In relation to decision-maker 3, we observed the presence of two groups of criteria with the same judgments, and those , se q d s 1, se d s that the planes by decision-makers 2, 3 and 4 are very similar.The groups of criteria observed in these planes lead to the conclusion that decision-makers 3 and 4 associated RM with TM, which also brings similarities between CE and IM.

Global analysis
After the individual evaluations, the global analysis was of each decision-maker presented in Table 1 were used to make of PROMETHEE II were used with data in this performance resulting in the group evaluation.
It is worth to mention that subsystems AC, ST and PT were considered as the most critical ones in the global evaluation, which is in accordance with many of the results subsystems, a sensitivity analysis was performed with the Chart 3. GAIA plane of decision-makers (i=1, 2, 3,4).intervals; in between, the weight values of each decisionmaker can vary without changing the priority order of the subsystems.As demonstrated in Table 3, the priority order is not even changed after 25% alterations (or more) in the pondering of the decision-makers, which makes the priority of the subsystems more acceptable for this study.
Chart 4 presents the GAIA plane of the group decision, between DM 1 and the others.As seen, DM 1 is the same that had aspect was seen as an indication that this decision-maker knew more about the project and, consequently, his/her judgment was different from the others, since the general manager pondered it as being the most important one (according to Chart A4).makers, we noticed that DM 1 presented the three subsystems ST, AC and PT as those with the worst performances, while decision-makers 2 and 3 consider subsystems AC, PT and TT maker 4, who presents (because of ties) the four subsystems previously presented (AC, ST and TT) as those with the worst performances.Such discrepancy was discussed in a workshop with the general manager, with the decision-makers, and the other members of the team.The consensus was that the considerations of DM 1 were more coherent to the reality of the project.It is important to say that the three subsystems regarded as having the worst performance by the group of decision-makers coincided with the choice of DM 1 .
With the objective of addressing the actions and improving the performance within the subsystems that had the worst performance, a complementary analysis took place to identify considerations of each decision-maker in relation to each indicator and subsystem were added, as presented in Table 4. AC, in relation to the criterion TM, the judgments of decision-makers 1, 2, 3 and 4 in this criterion were added, and, With this procedure, the priority orders to perform the improvement actions in each subsystem were obtained.The most critic situations were in subsystems AC and PT, concerning indicator CE, and in the subsystem ST, the indicator RM.With this result, the auditing of the indicators that presented worse performance in the subsystems was proposed to the general manager.Chart 4. GAIA plane of the group of decision-makers.

FINAL CONSIDERATIONS
This study aimed at investigating the application of PROMETHEE GDSS and GAIA methods for the performance evaluation in the subsystems of the ITA-SAT project.Based on the study, advantages were observed as to the use of such methods.PROMETHEE GDSS enabled the higher adherence of the team to the obtained results, due to the participation of different decision-makers, each one representing different interests inside the project. of preference functions, the behavior of the values given by decision-makers in relation to different performances presented by a criterion can be shaped in a way to get better accuracy as to the characteristics, which are really valuable for the decision-makers.
preference between decision-makers and criteria, which can be used to check a consensus among them.to be essential when the results were presented during a meeting for the other project members.Therefore, it was questioned if there was the participation of members with Even though the PROMETHEE GDSS method does not restrict the number of decision-makers for the performance of judgments, their number can be limited due to the lack of accessibility of the facilitator and/or because of their Therefore, the general manager was asked to point out the most important decision-makers, indicating the four ones mentioned in this study.
In order to assist the general manager in facing the problems of the project, improvement actions were proposed on the subsystems and indicators that presented worse performances.Such actions consisted of creating or improv-ment plan is revised and updated; updating the management plan, in order to involve the representatives of the project's subsystems; implementing mechanisms of learned lessons; consolidating a system of periodic evaluation/auditing; developing methods to make sure each member of the team is being instructed as to the activities to be performed, and, the project have feedback on their performance.
In relation to the limitations of this study, it is important to say that we did not consider the criteria CM and QM, which mance ranking of the subsystems.In this sense, it is clear that the results obtained in this study do not completely show the general performance of each subsystem, but their performance in relation to the criteria TM, CE, RM and IM.Besides, the adopted scale uses qualitative levels (concerning the following which possibly resulted in the imprecision of judgments.
Finally, the suggestion is that future researchers perform studies with the objective of handling the presented limitations, their judgments concerning cost and quality indicators in innovation projects, be it with the construction of scale of judg-Furthermore, it is possible to investigate the use of other MCDA methods to evaluate the performance of each subsystem.

Performance
Evaluation with PROMETHEE GDSS and GAIA: A Study on the ITA-SAT Satellite Project Chart 2. Preference functions used by the PROMETHEE methods.
. Technol.Manag., São José dos Campos, Vol.4,No 3, pp.381-392, Jul.-Sep., 2012 referring to RM were similar to those of TM; the ones referring to CE are the same as the judgments of IM.Besides, two TT, ST and AC, and the second one is comprised of TC and OBC-H.Something similar happened in relation to the GAIA plane of decision-maker 4, with similar judgments for two criteria (RM and TM) and two groups of action with the same second PT and AC.

Performance
Evaluation with PROMETHEE GDSS and GAIA: A Study on the ITA-SAT Satellite

Table 3 .
Range of variation of the weights.