In this paper, we present an architecture and a formal framework to be used for systematic benchmarking of monitoring and diagnostic systems and for producing comparable performance assessments of different diagnostic technologies. The framework defines a number of standardized specifications, which include a fault catalog, a library of modular test scenarios, and a common protocol for gathering and processing diagnostic data. At the center of the framework are 13 benchmarking metric definitions. The calculation of metrics is illustrated on a probabilistic model-based diagnosis algorithm utilizing Bayesian reasoning techniques. The diagnosed system is a real-world electrical power system, namely the Advanced Diagnostics and Prognostics Testbed (ADAPT) developed and located at the NASA Ames Research Center. The proposed benchmarking approach shows how to generate realistic diagnostic data sets for large-scale, complex engineering systems, and how the generated experimental data can be used to enable “apples to apples” assessments of the effectiveness of different diagnostic and monitoring algorithms.

Diagnostic inference involves the detection of anomalous system behavior and the identification of its cause, possibly down to a failed unit or to a parameter of a failed unit. Traditional approaches to solving this problem include expert/rule-based, model-based, and data-driven methods. Each approach (and various techniques within each approach) use different representations of the knowledge required to perform the diagnosis. The sensor data is expected to be combined with these internal representations to produce the diagnosis result. In spite of the availability of various diagnosis technologies, there have been only minimal efforts to develop a standardized software framework to run, evaluate, and compare different diagnosis technologies on the same system. This paper presents a framework that defines a standardized representation of the system knowledge, the sensor data, and the form of the diagnosis results – and provides a run-time architecture that can execute diagnosis algorithms, send sensor data to the algorithms at appropriate time steps from a variety of sources (including the actual physical system), and collect resulting diagnoses. We also define a set of metrics that can be used to evaluate and compare the performance of the algorithms, and provide software to calculate the metrics.

In performance maintenance in large, complex systems, sensor information from sub-components tends to be readily available, and can be used to make predictions about the system's health and diagnose possible anomalies. However, existing methods can only use predictions of individual component anomalies to guess at systemic problems, not accurately estimate the magnitude of the problem, nor prescribe good solutions. Since physical complex systems usually have well-defined semantics of operation, we here propose using anomaly detection techniques drawn from data mining in conjunction with an automated theorem prover working on a domain-specific knowledge base to perform systemic anomalydetection on complex systems. For clarity of presentation, the remaining content of this submission is presented compactly in Fig 1.

Complex engineering systems require efficient fault diagnosis methodologies, but centralized ap- proaches do not scale well, and this motivates the development of distributed solutions. This work presents an event-based approach for distributed diagnosis of abrupt parametric faults in continuous systems, by using the structural model decompo- sition capabilities provided by Possible Conflicts. We develop a distributed diagnosis algorithm that uses residuals, computed by extending Possible Conflicts, to build local event-based diagnosers based on global diagnosability analysis that gen- erate globally correct local diagnosis results. The proposed approach is applied to a multi-tank sys- tem, and results demonstrate an improvement in the design of local diagnosers. Since local diag- nosers use only a subset of the residuals, and use subsystem models to compute residuals (instead of the global system model), the local diagnosers are more efficient than previously developed dis- tributed approaches.

We present the third implementation of a framework created jointly by NASA Ames Research Center, Palo Alto Research Center, and Delft University of Technology to com- pare and evaluate diagnosis algorithms (DAs). This year‟s competition, DXC‟11, introduces a software track in addition to the industrial and synthetic tracks of previous competitions. A total of eleven DAs competed in the three tracks. The paper describes the systems, diag- nostic problems of the tracks, fault scenarios, evaluation metrics, participating DAs, results and analysis.

The lecture of the [Stanford-IVHM lecture series](https://dashlink.arc.nasa.gov/group/stanford-ivhm-lecture-series/) will give an overview of the approaches in building diagnostic solutions for networks and complex systems. The conventional rule-based approach and the top-down analysis will be compared with other innovative solutions based on information modeling and codebook correlation. One specific solution pioneered by research done in Columbia University and later implemented by SMARTS/EMC will be presented in more detail as an example of a consistent approach to diagnostics. Speaker: Yuri Rabover, Ph.D. VMTurbo Dr. Yuri Rabover, is a co-founder and Director of Product Strategy of VMTurbo, a startup in a stealth mode. Prior to VMTurbo Yuri spent 12 years working for SMARTS as director of engineering, product management and technology partnership. After EMC acquired SMARTS for $275M in 2005, Yuri was managing the Advanced Solution Group in the EMC Corporate CTO Office developing prototypes and proof of concepts of new innovative solutions. He is a seasoned technologist, strategist and researcher in the wide area of system, network and storage management with more than 20 years of industry and academia experience.

Diagnosis and prognosis are necessary tasks for system re- configuration and fault-adaptive control in complex systems. Diagnosis consists of detection, isolation and identification of faults, while prognosis consists of prediction of the remain- ing useful life of systems. This paper presents a novel inte- grated framework for model-based distributed diagnosis and prognosis, where system decomposition is used to enable the diagnosis and prognosis tasks to be performed in a distributed way. We show how different submodels can be automati- cally constructed to solve the local diagnosis and prognosis problems. We illustrate our approach using a simulated four- wheeled rover for different fault scenarios. Our experiments show that our approach correctly performs distributed fault diagnosis and prognosis in an efficient and robust manner.

Diagnosability is an important issue in the design of diagnostic systems, because it helps identify whether sufficient information is available to distinguish all the faults. Diagnosability of hybrid systems, however, is challenging, because mode transitions may occur during fault isolation. We present an event-based framework for hybrid systems diagnosis based on a qualitative abstraction of measurement deviations from nominal behavior. We derive event-based fault models that describe the possible measurement deviations sequences due to faults, which, coupled with the mode transition structure of the system, are used to automatically synthesize an event-based diagnoser for hybrid systems. We introduce notions of diagnosability for hybrid systems and show how the event-based diagnoser can be used to verify the diagnosability of the system. We apply our diagnosability analysis scheme to a real-world electrical power distribution system.

Diagnosis and prognosis are necessary tasks for system reconfiguration and fault-adaptive control in complex systems. Diagnosis consists of detec- tion, isolation and identification of faults, while prognosis consists of prediction of the remain- ing useful life of systems. This paper presents an integrated model-based distributed diagnosis and prognosis framework, where system decomposi- tion is used to perform the diagnosis and prog- nosis tasks in a distributed way. We show how different submodels can be automatically con- structed to solve the local diagnosis and prog- nosis problems. We illustrate our approach us- ing a simulated four-wheeled rover for different fault scenarios. Our experiments show that our approach correctly performs fault diagnosis and prognosis in a robust manner.

Multiple fault diagnosis is a difficult problem for dynamic systems. Due to fault masking, compensation, and relative time of fault occurrence, multiple faults can manifest in many different ways as observable fault signature sequences. This decreases diagnosability of multiple faults, and therefore leads to a loss in effectiveness of the fault isolation step. We develop a qualitative, event-based, multiple fault isolation framework, and derive several notions of multiple fault diagnosability. We show that using Possible Conflicts, a model decomposition technique that decouples faults from residuals, we can significantly improve the diagnosability of multiple faults compared to an approach using a single global model. We demonstrate these concepts and provide results using a multi-tank system as a case study.