An important part of the work carried out in mOeX is experimental. We plan to develop a modular open source platform for setting up, running and publishing experiments. It will be developed with three properties in mind: (reusability), so that others can develop their experiments with it and improve it, even beyond the project; (repeatability), so that experiments could be routinely repeated and independently reproduced; and (accountability), with the idea of publishing our results (positive or negative) continuously and supporting publications of these results seamlessly. In experimental sciences, laboratory logbooks are compulsory. Because our experiments are made in silico, maintaining a logbook is assumed to be easier because it can be filled automatically, for large parts.
We have started developing a prototypical environment for cultural knowledge evolution which supports our work. Its goal is to ease performing, controlling, sharing, and repeating experiments. In a first instalment, this software is based on off-the-shelve technologies: available semantic web technologies (RDF, OWL and alignments) are used for representing knowledge; git is used for tracking software versions; a wiki is used as an experiments logbook. This software combo allows to spare development time while filling minimally its purposes.
The current software is made of a Java API describing populations, agents, environments, knowledge models, game, experiments and loggers, in which we have implemented the necessary classes for the experiments in [euzenat2014c]. The wiki provides a systematic layout of the reports, e.g., author, hypothesis to be tested, version of the software used, raw results, etc. This is highly facilitated by the systematic description of cultural experiments and their results. The wiki provides accounting support since every change is logged. Using this minimal platform allows us to learn about what is necessary in such platform before eventually deciding, provided that we have sufficient resources, to develop more ambitious software.
We consider building on existing open-source software such as Sumatra [davison2012a] (an open source framework for logging, repeating and documenting experiments), Jupyter Notebook (another framework for documenting repeatable experiments), NetLogo [wilensky2015a] or Gama [grignard2013a] for multi-agent simulation or the researchobject model [hettne2014a] (a model for experimental research production based on semantic web technologies).
However, our effort concentrates on the specific developments required for supporting cultural evolution experiments: describing standardised experiments covering the variations that we want to implement, performing large-scale controllable experiments and collecting directly data so that it can feed logbook entries automatically. These entries should be complete enough to repeat experiments automatically and run them again with different initial conditions.
We experimented with container technology in order to embed our experiments and offer to others to run them easily. To that extent, we developed scripts to specify, run, and analyse experiments. In particular, these scripts are able to generate a Docker container specification that can perform experiments in the same conditions or with updated software. The documentation of the experiments is also eased by this process.
Lazy lavender development site: https://gitlab.inria.fr/moex/lazylav
| References |