This page elaborates on the usage of MobiBench.
All software related to MobiBench can be found at its github repo.
Contact: William Van Woensel
This section lists all of the MobiBench code projects.
All projects are packaged as Maven projects (*), and available on GitHub.
MobiBenchWebService is a Java EE project, and performs the following functions:
Converting RDF data & SPIN rules into other formats.
Extensibility To plugin a new resource format, create a new converter class implementing the uniform interface; i.e., RuleConverter or DataConverter (to convert SPIN rules to another format, the SPIN2 class can be used as superclass). Then, add the class to a configuration file (spin2s.txt / rdf2s.txt), used by the web service to dynamically load converter class definitions at startup.
Performing OWL2 RL ruleset selections.
Extensibility Supporting a new selection criterium depends on its requirements. In case an a priori analysis of the ontology is required, you can create a new subclass of the DomainBasedSelection class. Else, simply add a new subfolder under the (MobiBenchEngineJS)/www/res/owl/owl2rl/ folder, which keeps a list of rules and axioms to be removed, replaced or added (see existing files for examples).
Pre-processing certain n-ary rules, so they are executable by general-purpose rule engines.
Extensibility To support a new pre-processing mechanism, create a new subclass of the PreProcessor class. In case the mechanism requires ontology analysis, OntologyBasedPreProcessor can be subclassed, which features utility methods for that purpose.
The wvw.mobibench.service.servlet.MobiBenchServlet class listens for incoming POST requests and delegates them to the corresponding component. E.g., to support additional pre-processing options, or non-default selection methods, extend this class to listen for corresponding requests.
The file (MobiBenchEngineJS)/www/js/libs/webservice_utils.js defines utility functions to contact the webservice from the local benchmark engine (thus acting as a local "proxy"). This file will also need to be extended to support additional selection / pre-processing / etc. methods.
MobiBenchEngineJS is the core JavaScript MobiBench framework, which includes the core benchmarking code, JavaScript reasoners and their plugins, and currently available benchmark resources. Currently, there are two platform deployments: MobiBenchEngineNative (Android) and MobiBenchEnginePc (PC). Note that MobiBenchEngineNative is located under MobiBenchEngineJS/platforms/android.
Platform-specific JavaScript libraries and plugins are placed under (MobiBenchEngineJS)/www/js/platforms (e.g., www/js/platforms/android). Typically, platform-specific JavaScript code references platform-specific (Android, Java) code behind-the-scenes: e.g., to store data persistently, logs messages, makes HTTP requests, invokes platform-specific reasoning engines, etc. For MobiBenchEngineNative, JavaScript-to-Java communication is made possible using the JavaScript Cordova communication bridge (cordova.exec()); for MobiBenchEnginePc, this is done by using JDK8 Nashorn's support for invoking Java methods from JavaScript (see here for a short tutorial).
The method of loading any JavaScript libraries and plugins depends on the platform: for Android, this is done by the www/index.html file (which is opened by Apache Cordova); for PC, this done by the www/js/platforms/pc/init.js file (which is loaded by custom code, using the JDK8 Nashorn engine). To load additional JavaScript code, these files thus need to be extended.
MobiBenchEngineNative is self-contained, and includes the Javascript MobiBench framework under assets/www. (Therefore, it may be easier to setup some sort of file-syncing between MobiBenchEngineJS/www and MobiBenchEngineNative/assets/www to make sure the JavaScript code remains synchronized.) Currently, MobiBenchEnginePc refers to the MobiBench JavaScript framework code in MobiBenchEngineJS (and thus requires that project to be separately available). Both projects contain platform-specific code to e.g., respond to platform-specific calls from JavaScript (see before). Both re-use a shared code-base, MobiBenchEngineCore, which includes e.g., configuration, result and timer Java classes, the reasoning engine super-class, etc.
We note that, at least currently, we mostly use the PC deployment for debugging/verification purposes (the JDK8 Nashorn JavaScript output is *very* detailed), but it could just as well be used for benchmarking. However, in light of this, we cannot guarantee that all reasoners will work on PC (e.g., RDFStore-JS seems to have some issues).
Separate automation options exist for MobiBenchEngineNative and MobiBenchEnginePc.
MobiBenchAutoPc automates running benchmarks using MobiBenchEnginePc. This project simply references the MobiBenchEnginePc project and invokes the MobiBenchEnginePc.execBenchmark method.
MobiBenchAutoDevice: MobiBencAutoDeviceServer and MobiBenchAutoDeviceClient implement a remote automation solution for benchmarking on a mobile (Android) device. MobiBenchAutoDeviceServer is an Android project deployed on the device. The MobiBenchAutoDeviceClient Java client is deployed on a server or PC, and sends a set of generated benchmarks as HTTP requests to the MobiBenchAutoDeviceServer, which invokes the MobiBenchEngineNative (using Android Intents) to perform the requested benchmarks, and returns the results. The advantage of this setup is that it doesn't require a re-deployment on the Android device for each new set of benchmarks, but merely simply re-running the Java client. Also, it allows for remote benchmarking, e.g., in case the developer does not have physical access to the device. Clearly, this solution requires the MobiBenchEngineNative app to be installed on the device.
Both projects re-use a shared codebase, MobiBenchAutoCore, which includes e.g., shared configuration classes, a factory to generate sets of benchmark configurations, based on a user-supplied automation configuration (UserConfigFactory); and a factory to generate JSON serializations of such benchmark configurations (JsonConfigFactory).
MobiBenchAnalysis is a Java project, and performs analysis tasks, including: aggregating performance results & heap dumps into summary CSV files; and comparing performance times of two reasoning configurations (e.g., different OWL2 RL subsets), and output both the individual (i.e., per benchmark ontology) and total (i.e., aggregated) differences in performance. This tool is also available as a command line utility (run "jar mb-analysis.jar -help" for available options).
MobiBenchUtils contains a range of miscellaneous utility functions, including the comparison of service matching, OWL2 RL / DL reasoning output, and pre-processing output; and extracting OWL2 RL test suite files.
Dependency utility projects include utils, rdf-utils, library-utils and cmd-utils.
Below, we list the Android-ported source code, including dependencies & test code, for the supported OWL reasoners and rule engines. Each test project also includes notes (NOTES.txt) on the porting process. (PC versions may be obtained from the respective developers.)
This section is meant for developers who want to plugin new engines into MobiBench.
Due to the large amounts of dependencies of typical reasoning engines, as well as conflicts between these dependencies, we support packaging engines and their libraries as self-contained .jar files, which can be separately loaded at runtime without requiring re-compilation. The concrete packaging process depends on the platform (PC or Android; see below). This packaging has already been done for all currently supported reasoners and for both platforms.
Alternatively, an option for Android is to directly include engines and their libraries in the app, requiring a re-compilation each time. Choosing between these two options requires uncommenting the corresponding loadEngine method in the wvw.mobile.plugins.engine.NativeEnginesService class (option 1: method on line 252; option 2: method on line 256).
Below, we elaborate on each option.
At runtime, given a request for a particular reasoner, the (MobiBenchEngineNative)/wvw.mobibench.loader.android.AndroidDependencyLoader class dynamically loads the jar file from the (MobiBenchEngineNative)/assets/libs/ folder. We note that this option is courtesy of an Android Developers Blog post, which also served as the inspiration for the PC packaging process.
At runtime, given a request for a particular reasoner, the (MobiBenchEnginePc)/wvw.mobibench.loader.pc.PcDependencyLoader class dynamically loads the jar file from the (MobiBenchEnginePc)/libs/ folder.
We note that, the dependencies for AndroJena, ELK and Hermit are currently the Android versions. But, these may be easily replaced by more up-to-date dependencies (which could however require changing some parts in the Java plugin).
This option simply involves putting the reasoner's ReasoningEngine subclass as a Java file under the wvw.mobibench.engine package folder (and, if needed, its direct superclass), and including its dependencies on the project's build path. These subclasses are already present under the package folder as .txt files (MobiBenchEngineCore) - simply rename the desired one to a .java file. Reasoner dependencies can be found in their respective projects.
Instructions
This section gives instructions on how to use the MobiBench framework.
Setup: configure the (MobiBenchWebService)/src/main/resources/res/config.properties file, including the local path for storing files with relative paths (storagePath).
Deployment: the MobiBenchWebService project needs to be deployed using a server with a servlet container(e.g., Apache Tomcat). The web service needs to be deployed first, since it performs supporting functions for the MobiBench framework.
Setup: in case automation tools are not being used (see below), the (MobiBenchEngineJS)/www/js/config.js file needs to be configured with the IP address and port of MobiBenchWebService. Further, the benchmark.js file supplies several configuration parameters (see lines 3-17), such as whether conformance should be automatically tested, log output should be persisted, etc. Some of these options can also be set via the benchmark configuration (e.g., see Code 1).
Further:
Deployment:
Performance results, together with generated inferences and log output (if configured), will be sent to the web service during and/or after the benchmarks. Heap dumps will be stored on the device itself, under Android/data/benchmarks/dumps/. Performance result files and heap dumps are named after the ID from the benchmark configuration (see Code 1).
In MobiBenchAutoCore, update the wvw.mobibench.auto.AutoBenchmark.run method to execute the desired type of benchmark (e.g., run_OWL2RL method). In wvw.mobibench.auto.factory.user.DefaultUserConfigFactory, update the desired method (e.g., user_OWL2RL) with an automation configuration (see existing methods for examples). Each run_* (e.g., run_OWL2RL) method invokes the appropriate user factory method (e.g., user_OWL2RL) to generate sets of benchmark configurations. Update wvw.mobibench.auto.AutoBenchmark.remotePath to point towards (MobiBenchEngineJS)/www/res/owl/ on your machine.
MobiBenchAutoDevice: first, run the MobiBenchAutoDeviceServer project on the Android device. The app should display an IP address on the local WiFi network, on which the device benchmark server is accessible. Copy this address to (MobiBenchAutoDeviceClient)/wvw.mobibench.auto.AutoBenchmarkDeviceClient.deviceAddress. Then, run the MobiBenchAutoDeviceClient project as a Java application, with the wvw.mobibench.auto.AutoBenchmark.run setup (see above) to execute the desired benchmarks. Note that the MobiBenchEngineNative project first needs to be installed on the device.
MobiBenchAutoPc: simply run this project as a Java application. (Regrettably, it seems that the JDK8 Nashorn engine cannot be stopped; so after all benchmarks are done, the process needs to be stopped manually).
To execute benchmarks programmatically, call the execBenchmark JavaScript function (www/js/benchmark.js) with a configuration object. Below, we show an example:
config: {
engine: 'androjena',
nrRuns: 10,
warmupRun: true,
dumpHeap: true,
reasoning: {
task: 'ontology_inference',
mechanism: {
ontology_inference: {
type: 'owl2rl',
dependency: 'rule_inference'
},
rule_inference: {
mainFlow: 'frequent',
subFlow: 'load_data_exec_rules'
}
}
},
resources: {
ontology: {
path: 'res/owl/data/0.nt',
type:'data', format:'RDF', syntax:'N-TRIPLE'
},
owl2rl : {
axioms: {
path: 'res/owl/owl2rl/full/axioms.nt',
type:'data', format:'RDF', syntax:'N-TRIPLE'
},
rules: {
path: 'res/owl/owl2rl/full/rules.spin',
type: 'rules', format: 'SPIN'
},
selections: [ 'inf-inst', 'entailed' ]
},
confPath: 'res/owl/conf/ontology_inference/owl2rl/ore-small/0/full.nt'
id: '...'
}
}
This object specifies the unique engine ID; the number of experiment runs, possibly including a "warmup" run (which will not count towards the collected metrics); and whether memory usage should be measured (dumpHeap). Further, the reasoning part indicates the high-level reasoning task (i.e., ontology_inference) and concrete implementation (i.e., owl2rl; called "mechanism"). Any dependent task is listed as well (i.e., rule_inference), including the main and sub process flows to be followed.
The resources section supplies information on the ontologies, service descriptions / user goals, data and (potentially) rulesets used in the benchmark. In this case, an ontology and OWL2 RL resources are listed (including the axioms and ruleset, as well as ruleset selections to be applied). In addition, the ontology section can indicate the expected output of the reasoning task (confPath) which allows for automated conformance checking.
Finally, the id field indicates the name of the file in which performance results (and the heap dump) will be stored. Any identifier may be used here.
To check which configurations (e.g., reasoning task, mechanism, subflow) are supported by each engine, check the JavaScript / Java plugin (for Java, check the createEngineConfig method).
To deal with an abundance of benchmark results, MobiBenchAnalysis supports automatically aggregating benchmark results, including performance times and memory heap sizes, into a CSV file. This file lists the performance results per configuration; including process flow, reasoning task and implementation (or "mechanism"; e.g., OWL2 RL vs. built-in OWL reasoning), applied rule subsets (for OWL2 RL), potential engine-specific options, and dataset (potentially including scale). Secondly, MobiBenchAnalysis supports outputting the individual (i.e., per benchmark dataset) and total (i.e., aggregated) differences in performance between two reasoning configurations, thus greatly facilitating the comparison of different reasoning setups.
This tool is also available as a command line utility (the .jar file can be found under the MobiBenchAnalysis project). Run "jar mb-analysis.jar -help" for available options.
The functionality of the MobiBenchWebService can also be accessed separately; e.g., to separately convert benchmark resources or perform ruleset selections. The webservice.js script (in MobiBenchEngineJS/www/js/) contains code snippets to access various functions of MobiBenchWebService. For instance:
// first, retrieve rules from file
var rules = retrieveResource("res/owl/owl2rl/full/rules.spin");
// then, convert rules via Web service
convert(rules, 'rules', {
to : 'Datalog' // convert to this format
}, function(convRules) {
console.log("converted rules:\n" + convRules); // outputs converted rules
});
The (MobiBenchEngineJS)/www/webservice.html page executes the webservice.js script. Important: loading this page requires deploying the relevant MobiBenchEngineJS code (www/ folder) on the same server as MobiBenchWebService (e.g., localhost), and accessing the webservice.html file via the browser. Else, a cross-origin resource-sharing (CORS) exception will be raised by the browser.
Below are some pointers on how to import and run the MobiBench projects on Eclipse.
(*) The Android projects are not packaged as Maven projects, and have their dependencies directly in their libs/ folder. This is because of difficulties that Eclipse has with Android Maven projects: when putting all plugins under <pluginManagement> in pom.xml (as recommended when faced with a "plugin lifecycle .." error), Eclipse complains that "apk" packaging is not supported. There are other working solutions out there, but instead of forcing people to deal with this issue we opted to make these "regular" Android projects.