Modeling Engineering Information with Machine Learning

Status of Machine Learning as Tools for Engineering Design

Machine Learning Programs as Modeling Tools

The modeling task assigned to ML is different from the automated knowledge generation or knowledge reorganization. The difference is not merely terminological. The use of ML programs for automated knowledge construction or reorganization involves many implicit and unwarranted assumptions about the nature of data and the way it was pre-processed before learning could work, and the rationale behind using ML programs is often hidden (Reich et al, 1993). In contrast, viewing ML programs as simply methods for modeling data by summarizing it into pre-defined structures such as production rules or decision trees, in the same way that statistical methods are viewed, lets us use them for whatever practical purpose they are capable of supporting.

The modeling role of ML involves: (1) obtaining structured information from some source; (2) pre-processing this information to suite the modeling task; (3) selecting and applying ML programs to the data for creating models summarizing it; and (4) analyzing the models to better understand the nature of the information and potentially its source. Aside from trivial cases, models, in general, have several important properties (Subrahmanian et al, 1993): (1) They are never perfect; thus, their results must be critically analyzed and put into perspective. (2) Models never display a complete view of the object they model; thus, better understanding of the object being modeled can be obtained from modeling it in different ways. (3) Different models of the same object may be conflicting. This is not critical if we remember the first property. Finally, (4) simple models can be as effective as complex models for some purposes; thus, it is important to understand the functionalities of different models and their cost in order to make effective modeling choices.

The Ingredients of a Modeling ML System

A modeling ML system is an environment that facilitates the use of ML programs for modeling information for practical purposes. There are several ingredients that such a system must have.
(1) Information management facility that handles information preparation, restructuring, and maintenance.
(2) Generic learning tasks. Since we would like to have diverse modeling capabilities, the system must include several ML programs. Each ML program can perform one or more primitive learning tasks such as calculating attribute dependencies, creating decision rules or decision tree, or pruning a tree. These tasks are referred to as generic learning tasks (Reich and Fenves, 1989).
(3) Integrative infrastructure. The system must have an infrastructure that (1) integrates the ML programs and some mechanisms for accessing external software needed for using the models learned for practical engineering purposes or merely for analyzing or evaluating them; and (2) allows specifying data exchange protocols without enforcing a single approach or representation. It is understood that this flexibility may result in irreversible translations or information loses when transferring information between different ML programs, and that some translations may require human intervention. Nevertheless, this is necessary since no single unified representation could support the complete learning process.

Using a Modeling ML System: Contextual ML modeling

In addition to the above difficulty, the particular learning tasks that are most helpful to address the questions at hand need to be identified. Brief experience has shown that it is hard to automatically select generic learning tasks for solving a particular learning problem. Rather, this selection requires significant understanding of ML programs and experience in their practical use. Such experience needs to be accumulated and maintained for future modeling activities in a systematic manner by the aforementioned integrative infrastructure. The process of ML modeling discussed above and depicted in Figure 1 is referred to as Contextualized ML Modeling (CMLM) (Reich, 1991b).

Figure 1: Contextualized machine learning modeling

Example ML Modeling Projects

Two good examples demonstrating the importance of the modeling role of ML are: (1) the modeling of existing decision procedures (DP) that facilitates their evaluation or redesign (Reich et al, 1995); and (2) the learning of knowledge from engineering databases (Reich et al, 1994).

Modeling Decision Procedures (MODEP)

In modeling decision procedures (MODEP), ML programs are used to model DPs. Examples for training ML programs are generated by simulations of the DP being studied. In addition, knowledge about the DP, which can vary from little (in the case of a ``black box'' DP) to significant (in the case of a ``glass box'' DP), influence the types of ML modeling that are performed. One of our projects involves the modeling of a DP called CHOICE for selecting among mathematical models that simulate groundwater contaminant transport processes. The modeling employed CN2, FOIL, and IND. The use of different programs' parameters created different models of the original DP, thus facilitated its evaluation and uncovered flaws that were subsequently corrected. The new DP was then subjected to the same modeling procedure.

The project led to the following observations. (1) ML programs have limitations even for addressing tasks simpler than their intended role as automated knowledge generation or reorganization tools. Nevertheless, (2) ML programs can be applied to practical problems if the nature of the application is carefully selected to match their limited scope of applicability by following CMLM. This may require being intimate with the details of ML principles and programs and being creative in their operation. In contrast, (3) ML programs are hardly usable to engineers unfamiliar with such detailed knowledge of ML.

This study was instrumental in understanding that the practical use of ML techniques requires an infrastructure for the manipulation of different ML programs in creative ways and the accumulation of information about the past uses of ML programs. The plan is to develop an infrastructure and a collection of tools (ML and others) that can be used while the DP is being developed and not only after its development has terminated.

Learning from Databases

Engineering databases contain significant information such as historical data on product performance and maintenance decisions that can assist in future engineering design and management decisions. The major impediment to using this information in practice is the problem of making the knowledge embedded in the information explicit for engineers. While the development of new database technologies can provide efficient means for storing and retrieving information, it cannot help engineers in comprehending it.

Knowledge in databases is hard to model by ML programs. The information in databases may have errors of different kinds. Information that is critical for learning may be missing, and redundant or useless information may dominate the data. The information can come from several sources and thus not be consistent in the way terminology is used. The integration infrastructure must support the reconciliation of such terminological discrepancies and, as before, allows for flexible integration of ML programs and other utility tools. We plan to use n-dim (Levy et al, 1993) as the integrative infrastructure and as generic learning tasks include several borrowed ML programs (e.g., CN2, FOIL, IND), as well as in-house developments (e.g., Ecobweb and others presently under development).

We have started working on learning bridge management knowledge from bridge management databases for the purpose of supporting decision making. An example scenario for this purpose is shown in Figure 2 (Reich et al, 1994). Other studies we are working on include the learning from a building cost database (with Retik, A., Strathclyde U., Scotland) and the learning from production management databases (with Karni, R., Technion, Israel).

Figure 2: Extracting knowledge from bridge management databases

Modeling database information with ML has other benefits that can be realized even before assisting in engineering decision making. For example, learned models can help the design of database schema by or uncover its evolution by detecting dependencies or ``causal'' influences between attributes; characteristic and classification rules can be extracted from data and subsequently assist in maintaining data integrity and data entry by warning users when data suspected of being incorrect is entered. These benefits can provide feedback to those who design databases on the kind of data needed to be captured and its structuring.

Why is it critical for engineering design to address ML modeling?

If one looks at the many recent publications on quality design one sees that the key to quality design is information quality. Engineering design is based on exploiting significant amount of data for making informed decisions about product development. It is only through the modeling of this information that quality engineering design is achieved. Modeling information with ML programs is a capability that can become practical with the right integration of available technology into usable tools. It has much better chances to impact design before any other method developed in ML in design research mature.

There is another reason to engage in ML modeling that is related to engineering design research and practice. Any infrastructure for engineering design such as n-dim, will accumulate significant information over time. Without such modeling the information in the system will become unmanageable. Thus, the relationship between an integrated infrastructure and ML programs is one of continual reinforcement.

None of the tasks we mentioned, the modeling of decision pocedures or learning from databases is trivial; ML modeling is complex. However, in adopting the modeling role of ML programs we become explicitly aware of their properties by virtue of them being models. In particular, models are imperfect, incomplete, may be conflicting and even wrong albeit practically effective. This awarness then leads us to a systematic study of the practical utility of different programs in different learning contexts.

References

Levy, S., Subrahmanian, E., Konda, S. L., Coyne, R. F., Westerberg, A. W., and Reich, Y. (1993). ``An overview of the n-dim environment.'' Technical Report EDRC-05-65-93, Engineering Design Research Center, Carnegie Mellon University, Pittsburgh, PA.

Reich, Y. and Fenves, S. J. (1989). ``Integration of generic learning tasks.'' Technical Report EDRC 12-28-89, Engineering Design Research Center, Carnegie Mellon University, Pittsburgh, PA.

Reich, Y. and Fenves, S. J. (1992). ``Inductive learning of synthesis knowledge.'' International Journal of Expert Systems: Research and Applications, 5(4):275--297.

Reich, Y. and Fenves, S. J. (1995). ``A system that learns to design cable-stayed bridges.'' Journal of Structural Engineering, ASCE, 121(7):1090--1100.

Reich, Y., Konda, S., Levy, S. N., Monarch, I., and Subrahmanian, E. (1993). ``New roles for machine learning in design.'' Artificial Intelligence in Engineering, 8(3):165--181.

Reich, Y., Fenves, S. J., and Subrahmanian, E. (1994). ``Flexible extraction of practical knowledge from bridge databases.'' In Proceedings of the First Congress on Computing in Civil Engineering (Washington, DC), pages 1014--1021, New York, NY, American Society of Civil Engineers.

Reich, Y., Medina, M., Shieh, T.-Y., and Jacobs, T. (1995). ``Modeling and debugging engineering decision procedures with machine learning.'' Journal of Computing in Civil Engineering. (in press).

Reich, Y. (1991a). ``Design knowledge acquisition: Task analysis and a partial implementation.'' Knowledge Acquisition, 3(3):237--254.

Reich, Y. (1991b). ``Designing integrated learning systems for engineering design.'' In Birnbaum, L. and Collins, G. C., editors, Proceedings of The Eight International Workshop on Machine Learning (Evanston, IL), pages 635--639, San Mateo, CA, Morgan Kaufmann.

Reich, Y. (1993a). ``The development of Bridger: A methodological study of research on the use of machine learning in design.'' Artificial Intelligence in Engineering, 8(3):217--231.

Reich, Y. (1993b). ``Towards practical machine learning techniques.'' In Proceedings of the First Congress on Computing in Civil Engineering (Washington, DC), pages 885--892, New York, NY, ASCE.

Reich, Y. (1995). ``Measuring the value of knowledge.'' International Journal of Human-Computer Studies, 42(1):3--30.

Subrahmanian, E., Konda, S. L., Levy, S. N., Reich, Y., Westerberg, A. W., and Monarch, I. A. (1993). ``Equations aren't enough: Informal modeling in design.'' Artificial Intelligence in Engineering Design, Analysis, and Manufacturing, 7(4):257--274.