A framework to compare and evaluate diagnosis algorithms (DAs) has been created jointly by NASA Ames Research Center and PARC. In this paper, we present the first concrete implementation of this framework as a competition called DXC’09. The goal of this competition was to evaluate and compare DAs in a common platform and to determine a winner based on diagnosis results. 12 DAs (model-based and otherwise) competed in this first year of the competition in 3 tracks that included industrial and synthetic systems. Specifically, the participants provided algorithms that communicated with the run-time architecture to receive scenario data and return diagnostic results. These algorithms were run on extended scenario data sets (different from sample set) to compute a set of pre-defined metrics. A ranking scheme based on weighted metrics was used to declare winners. This paper presents the systems used in DXC’09, description of faults and data sets, a listing of participating DAs, the metrics and results computed from running the DAs, and a superficial analysis of the results.

We present the third implementation of a framework created jointly by NASA Ames Research Center, Palo Alto Research Center, and Delft University of Technology to com- pare and evaluate diagnosis algorithms (DAs). This year‟s competition, DXC‟11, introduces a software track in addition to the industrial and synthetic tracks of previous competitions. A total of eleven DAs competed in the three tracks. The paper describes the systems, diag- nostic problems of the tracks, fault scenarios, evaluation metrics, participating DAs, results and analysis.

We present the third implementation of a framework created jointly by NASA Ames Research Center, Palo Alto Research Center, and Delft University of Technology to compare and evaluate diagnosis algorithms (DAs). This year's competition, DXC'11, introduces a software track in addition to the industrial and synthetic tracks of previous competitions. A total of eleven DAs competed in the three tracks. The paper describes the systems, diagnostic problems of the tracks, fault scenarios, evaluation metrics, participating DAs, results and analysis.

Slides presented at DXC'10 session of 21st International Workshop on Principles of Diagnosis

The DXC Framework is a collection of programs and APIs for running and evaluating diagnostic algorithms (DAs). It is complementary to system XML catalogs and datasets. A DXC Framework installer for Windows may also be downloaded. Competitors must submit DAs that can communicate with and be executed by the DXC Framework. It was designed primarily for the Diagnostic Competition and future competitions, however, it is lightweight and reusable in any context where data from one or more sources needs to be communicated to, processed by, and diagnosed by reasoners. It is distributed under the LGPL. Contents: APIs- The Java and C++ DXC APIs provide the classes and functionality necessary for data exchange. Scenario Execution Software - A scenario is an interval of time in which system data is sent to a diagnostic algorithm. StandaloneSDS is used to execute one scenario on its own and send the results to a file. ScenarioLoader is for executing a large collection of scenarios with one or more DAs in an automated fashion (as is necessary to perform benchmarking of algorithms). Example Diagnostic Algorithms - Two minimal example DAs are provided (along with source code) to demonstrate proper use of the APIs--one in C++, the other in Java. #Updates to the DXC Framework: This DxcFramework update (DxcFramework-1.1.zip) provides several important bugfixes and new features (see below). Your questions, feedback and suggestions have been very helpful in improving the software--please continue sending them. A DXC Framework installer update for Windows (DxcFramework-1.1.msi) may also be downloaded. DxcFramework 1.1 Changelog: * Added Evaluator. The Evaluator takes the output generated by scenario execution (stored in ./Scenarios/Results/) and calculates DA evaluation metrics. See README for more information. Cpu time and peak memory usage are currently not reported in Linux, this will change. * Added SystemCatalog class for basic XML catalog parsing and validation. * Included system catalogs and example scenarios for one system from each track. * New StandaloneSDS options: -t: specify wait time (in seconds) to keep listening for diagnoses after scenario end. -v: validate scenarios against corresponding system catalog (Not available in JavaSSDS). Catalogs must be placed in $DXC_HOME/SystemCatalogs, unless a different directory is specified in Dxc.cfg SYSTEMCAT_DIR. * Added JavaStandaloneSDS execution script, now just run "JavaStandaloneSDS" * Scenario timing changes. DA startup timeout now 20 seconds. Added 20 second post-scenario wait period to allow DAs to send final diagnosis and exit. * Scenario timing values can now be edited in Dxc.cfg (values are in milliseconds). Current values reflect our planned competition scenario conditions. Competitors will be notified of any changes to these values. #Evaluator Update Please follow these instructions to update the Evaluator in the Framework (see Evaluator.zip below): 1. Make sure your version of DXC Framework is the most recent (1.1). Source release users, ensure that Java code is built. 2. Move the copy of Evaluator.jar contained in $DXC_HOME/Bin to Evaluator.jar.backup. 3. Save new Evaluator.jar to $DXC_HOME/Bin. 4. Choose desired config file and save to $DXC_HOME (top-level directory), renaming to evalCfg.xml. Use evalCfg.xml.industrial for industrial track metrics/scoring, evalCfg.xml.synthetic for synthetic. 5. Run "Evaluator > results.txt" (or similar) at the command line. Evaluator will calculate metrics and scores for results files it finds in $DXC_HOME/Scenarios/Results.

Slides presented at DXC'11 session of 22nd International Workshop on Principles of Diagnosis

Sample data, including nominal and faulty scenarios, for Diagnostic Problems I and II of the Second International Diagnostic Competition. Three file formats are provided, tab-delimited .txt files, Matlab .mat files, and tab-delimited .scn files. The scenario (.scn) files are read by the DXC framework. See the Second International Diagnostic Competition project page for more information.

The DXC Framework is a collection of programs and APIs for running and evaluating diagnostic algorithms (DAs) under identical experimental conditions. It is complementary to system XML catalogs and data sets. The Oracle is new to DXC'10 and provides a querying capability to the DAs to determine the correct actions for given faults. Because the oracle uses proprietary NASA software, it is provided only in binary form. Thus the framework is available as a windows binary installer, linux source files + binary oracle, or windows source files + binary oracle. Please see the included READMEs and example DAs for more information.

Diagnostic inference involves the detection of anomalous system behavior and the identification of its cause, possibly down to a failed unit or to a parameter of a failed unit. Traditional approaches to solving this problem include expert/rule-based, model-based, and data-driven methods. Each approach (and various techniques within each approach) use different representations of the knowledge required to perform the diagnosis. The sensor data is expected to be combined with these internal representations to produce the diagnosis result. In spite of the availability of various diagnosis technologies, there have been only minimal efforts to develop a standardized software framework to run, evaluate, and compare different diagnosis technologies on the same system. This paper presents a framework that defines a standardized representation of the system knowledge, the sensor data, and the form of the diagnosis results – and provides a run-time architecture that can execute diagnosis algorithms, send sensor data to the algorithms at appropriate time steps from a variety of sources (including the actual physical system), and collect resulting diagnoses. We also define a set of metrics that can be used to evaluate and compare the performance of the algorithms, and provide software to calculate the metrics.

The DXC Framework is a collection of programs and APIs for running and evaluating diagnostic algorithms (DAs) under identical experimental conditions. It is complementary to system XML catalogs and data sets. The Oracle for DXC'11 is the same as DXC'10 and provides a querying capability to the DAs to determine the correct actions for given faults. Because the Oracle uses proprietary NASA software, it is provided only in binary form. Thus the framework is available as a windows binary installer, linux source files + binary oracle, or windows source files + binary oracle. Please see the included READMEs and example DAs for more information.