A Framework for the evaluation of specification miners based on finite state machines (supplementary material)

This page provides supplementary material for our paper on evaluating specification miners:

  • The 32 reference specifications as dot files and as pdf files.
  • The implementation of our precision and recall metrics for finite state machines.
Feel free to use this material for your research, in particular, for evaluating your specification miner!


Reference Specifications

API classdot filepdf file
java.net.DatagramSocketdotpdf
java.net.MultiCastSocketdotpdf
java.net.Socketdotpdf
java.net.URLConnectiondotpdf
java.net.URLdotpdf
java.util.ArrayDequedotpdf
java.util.ArrayListdotpdf
java.util.Collectiondotpdf
java.util.Dequedotpdf
java.util.EnumMapdotpdf
java.util.EnumSetdotpdf
java.util.Formatterdotpdf
java.util.HashMapdotpdf
java.util.HashSetdotpdf
java.util.Hashtabledotpdf
java.util.IdentityHashMapdotpdf
java.util.LinkedHashMapdotpdf
java.util.LinkedHashSetdotpdf
java.util.LinkedListdotpdf
java.util.Listdotpdf
java.util.Mapdotpdf
java.util.NavigableMapdotpdf
java.util.NavigableSetdotpdf
java.util.PriorityQueuedotpdf
java.util.Queuedotpdf
java.util.Setdotpdf
java.util.SortedMapdotpdf
java.util.SortedSetdotpdf
java.util.StringTokenizerdotpdf
java.util.TreeMapdotpdf
java.util.TreeSetdotpdf
java.util.WeakHashMap dotpdf

You can also download all dot files and all pdf files at once.


Implementation of Precision and Recall Metrics

Your can download an implementation of the precision and recall metrics. It comes with a simple Java file that demonstrates how to create two finite state machines and how to compute the precision/recall of one with respect to the other. The demo uses the finite state machines from Figure 1 of our ICSM'10 paper. The implementation is in Scala and builds upon Anders Møller's automaton library. The above download contains all requirements and has been tested with Java 1.7.0.

After downloading and unpacking, run the demo with:
java -cp lib/scala-library.jar:lib/automaton.jar:lib/fsmEval.jar:bin/ FSMEvalDemo