Throughout the systems development life cycle, the analyst, management, and users have been evaluating the evolving information systems and networks to give feedback for their eventual improvement. Evaluation is also called for following system implementation.
Evaluation Techniques
In recognition that the ongoing evaluation of information systems and networks is important, many evaluation techniques have been devised. These techniques include cost-benefit analysis (as discussed in Chapter “Project Management“); models that attempt to estimate the value of a decision based on the effects of revised information using information theory, simulation, or Bayesian statistics; user evaluations that emphasize implementation problems and user involvement; and information system utility approaches that examine the properties of information.
Each type of evaluation serves a different purpose and has inherent drawbacks. Cost-benefit analysis may be difficult to apply, because information systems provide information about objectives for the first time, making it impossible to compare performance before and after implementation of the system or distributed network. The revised decision evaluation approach presents difficulty, because all variables involved with the design, development, and implementation of the information system cannot be calculated or quantified. The user involvement approach yields some insight for new projects by providing a checklist of potentially dysfunctional behavior by various organizational members, but it stresses implementation over other aspects of IS design. The information system utility approach to evaluation can be more comprehensive than the others if it is expanded and systematically applied.
The Information System Utility Approach
The information system utility approach for evaluating information systems can be a comprehensive and fruitful technique for measuring the success of a developed system. It also can serve as a guide in the development of any future projects the analyst might undertake.
Utilities of information include possession, form, place, and time. To evaluate the information system comprehensively, these utilities must be expanded to include actualization utility and goal utility. Then the utilities can be seen to address adequately the questions of who (possession), what (form), where (place), when (time), how (actualization), and why (goal).An example of this information utility approach can be seen in the evaluation of a blood inventory system in the figure illustration below.
An information system can be evaluated as successful if it possesses all six of these utilities. If the system module is judged as “poor” in providing one of the utilities, the entire module will be destined to fail. A partial or “fair” attainment of a utility will result in a partially successful module. If the information system module is judged as “good” in providing every utility, the module is a success.
Possession utility answers the question of who should receive output, or, in other words, who should be responsible for making decisions. Information has no value in the hands of someone who lacks the power to make improvements in the system or someone who lacks the ability to use the information productively.
Form utility answers the question of what kind of output is distributed to the decision maker. The documents must be useful for a particular decision maker in terms of the document’s format and the jargon used. Acronyms and column headings must be meaningful to the user. Furthermore, information itself must be in an appropriate form. For example, the user should not have to divide one number by another to obtain a ratio. Instead, a ratio should be calculated and prominently displayed. At the other extreme is the presentation of too much irrelevant data. Information overload certainly decreases the value of an information system.
Place utility answers the question of where the information is distributed. Information must be delivered to the location where the decision is made. More detailed reports or previous management reports should be filed or stored to facilitate future access.
Time utility
answers the question of when information is delivered. Information must arrive before a decision is made. Late information has no utility. At the other extreme is the delivery of information too far in advance of the decision. Reports may become inaccurate or may be forgotten if delivered prematurely.
Actualization utility involves how the information is introduced and used by the decision maker. First, the information system has value if it possesses the ability to be implemented. Second, actualization utility implies that an information system has value if it is maintained after its designers depart, or if a one-time use of the information system obtains satisfactory and long-lasting results.
Goal utility answers the “why” of information systems by asking whether the output has value in helping the organization obtain its objectives. The goal of the information system must not only be in line with the goals of decision makers, but it must also reflect their priorities.
