Exaggerated Claims and the Importance of Hands-on Evaluation
Anyone who has worked with more than one commercial rule engine knows that there are some significant performance differences between the various engines. In some cases the performance issues are so severe that you end up scratching your head, looking at the Gartner magic quadrant and trying to figure out how they put some of these turkeys where they did.
With Business Rules Engines we are, unfortunately, in a marketplace where information asymmetry and switching costs are the order of the day. Vendors don’t like to have their rule engines compared head-to-head and they use everything from custom rule description formats to alternate rule evaluation algorithms to rule repositories to legal restriction on making any performance measures public in order to make those comparisons difficult.
As a result, you see all sorts of claims from vendors about how nice their super secret algorithms perform. Peter Lin regularly takes on some of the more outrageous claims and, as best as he can, given the lack of information from the vendor, attempts to debunk them. His latest bit of skeptical wisdom, about the HyperRETE algorithm from EWA Systems, can be found here.
Ideally, we would have a standard rule description format, something like RuleML for example, and would use the rule engine as a standard execution environment with a standard set of API’s, etc. Vendors could differentiate their product offering by providing nice IDE and Rule Repository functionality or souping up their rule engine to provide better performance. Imagine Java IDE vendors clouding the issue by each shipping their own, non-standard JVM’s, tightly integrated with the IDE. If you want to run apps developed with their IDE, you need to use their engine and if you want to use their engine, you need to use their IDE.
In reality, Rule Engines should be a commodity by now. It’s just the lack of sophistication by IT managers that prevents them from evaluating BRE technology just like they would OS or database software.
So, what is a product manager to do? My advice is to evaluate as many rule engines as you can. If you need inferencing, i.e. the ability to chain together conclusions across a large fact base, make sure to evaluate RETE engines and make sure those engine are really really RETE. The best way to test the engines for performance and scalability is to build a prototype that replicates the rules and facts you are going to process. The academic benchmarks like Manners are all well and good, but if you are going to write lots of rules that involve 20 conditionals or the existential quantifier ("there are at least 3 instances of a bad credit reference"), you’re better off testing using that. If you’re expecting to chase a million customers through the rule engine every day, you want to know if you’ll need 4 or 400 CPU’s, so make sure to use enough representative data so you can accurately estimate performance.
If, instead, you swallow a vendors song and dance and fancy GUI interface, you deserve what you get. So, do your homework.
