Comments on: Exaggerated Claims and the Importance of Hands-on Evaluation

By: ILOG BRMS Blogs » Blog Archive » The Good, The Bad and The Ugly - Rule Engine Benchmarks

Thu, 19 Jun 2008 13:31:24 +0000

[...] for BRMS are fairly well documented, and there seems to be a general consensus that they should not be used for reliable product performance comparison, however they keep cropping up! Even in [...]

By: Adrian Walker

Adrian Walker — Sat, 22 Apr 2006 19:13:01 +0000

Actually, having an agreed synstax for rules is only part of the story. Different rule engines can produce different results from the same set of rules.

In our work on the Internet Business Logic system, we publish a model theoretic specification of what our rule engine is supposed to do. We also publish a simplified version of the algorithm for the rule engine.

There's more about this in
"Understandability and Semantic Interoperability of Diverse Rules Systems", http://www.w3.org/2004/12/rules-ws/paper/19

Thanks in advance for comments on the paper, or on the online system. (Shared use of the system is free.)

By: Dietrich Kappe

Dietrich Kappe — Mon, 10 Apr 2006 18:06:18 +0000

Sorry about the name mixup, Peter. I must have been watching old episodes of Hollywood Squares. I’ve fixed it in the post.

By: Peter Lin

Peter Lin — Sun, 09 Apr 2006 16:51:57 +0000

Actually, my name is Peter Lin. In my bias thinking there are two classes of tests one should perform when evaluating rule engines.

The first group is worse case scenario, because there are times an application needs cross product joins.

The second group tests network topology. By network topology, I mean different network size. The size of a RETE network can be measure across 3 dimensions: node count, width and depth.

Since runtime performance is primarily the result of the network topology, one can roughly estimate the performance by looking at a few sample rules. The approach I tend to take is to look at a dozen or so rules that represent the basic types of rules an application will use. Once I have a decent idea of the types of rules and their general structure, it's fairly easy to predict the results. The next step I take is to generate a decent set of rules and run several dozen benchmarks. I like to run series of benchmark to determine the scalability as a factor of rule count and dataset size.

I'm planning to design and implement a set of benchmarks to measure rule engine performance using RuleML as the language. I've been working with one of the founders of RuleML, and over the last few years we've been thinking about this issue. Take for example Manners benchmark publish miranker. It's valid to run Manners in forward and backward chaining, because an application should employ both techniques.

The dangerous thing is to say manners in backward chaining is an "apples-to-apples" comparison to forward chaining RETE. In general, a good backward chaining engine should be 30-50x faster than RETE, because backward chaining use the first best match approach. Rather than build up lots of activations in the agenda, it takes the best match and fires it immediately. This means backward chaining execution of manners isn't doing cross product matching.

I've seen cases where a system really needs to do cross product matching. In those cases, RETE will beat non-RETE implementations by 100x or greater.

I'm hoping by middle of next year I will have a set of 20 test scenarios written in OMG Production RuleML that all engines can use for comparison.

peter