History

OMF Main Page » OMF User Guide »

Experiment Life-cycle - Usage Overview

This page provides a general overview of the steps involved in a complete experiment cycle with OMF, i.e. developing an experiment, deploying it, and accessing its result.

If you are a new user, we suggest that you first read through the OMF Introduction page to get familiar with the different OMF names and entities, which are used below.

The following Figure 1 illustrates the usage of OMF in the case when you are using OMF together with OML to collect your experiment measurement (as mentioned on the ''about OMF'' page it is possible not to do so).

Figure 1. Usage Overview, the different steps involved in using OMF to evaluate your prototype (e.g. application, kernel module) on an OMF-enalbed testbed.

1. Step 1: The application, kernel module, or software to evaluate

In this 1st step, you develop the software you would like to evaluate.

Basically this is the prototype of your research idea, algorithm, scheme, etc... It can be either a user-space applications (e.g. your new mobile peer-to-peer services) or a system-space software (e.g. a modified kernel module driver implementing your new ad-hoc routing algorithm). You may use any type of programming tools or languages to develop such a software, as long as the resulting binary is executable on one of the baseline OS installed on the resources you would like to use. Most of the time that baseline OS is some flavour of Linux (note that different testbeds may provide/support different OS images).

Alternatively, you can use (or modify and recompile) an existing software. For example, this might be the case if you are interesting in evaluating the bandwidth/loss characteristics of a particular network medium.

Optional:

If you want to use the OML framework to systematically collect measurements when running your experiment, then you will need to include OML Measurement Points within your software. This is an easy task (i.e. adding few lines of code to your software) and it is explained in details on the OML wiki pages. We highly recommend using OML to collect measurements on OMF-enabled testbeds.

2. Step 2: Developing the experiment

In this 2nd step, you develop the experiment you would like to execute.

First, you need to write an OMF Application Definition for the software, which you would like to evaluate. Basically, this Application definition can be viewed as an interface that makes OMF entities aware of your application, i.e. which parameters it does accept, which measurement points it does provide, or where to find the package/archive files that contains your software. Many examples of Application Definitions can be found in the basic tutorials and the advanced ones

Second, you need to write an OMF Experiment Description (ED).

The first part of the ED describes the set of resources required by an experiment, and their corresponding configurations. Such as:
  • a PC-based wireless node, with its wireless network interface in ad-hoc mode, using channel 1 and essid foo
  • your application X, launched with parameter bar=1 and collecting data from measurement points foobar
The second part of the ED describes the set of actions to perform in order to realize that experiment. Such as:
  • when all applications are installed on all the nodes, do a list of tasks
  • when a node is reaching a specific state do another list of tasks

An ED is written using the OMF Experiment Description Language (OEDL). Examples of Experiment Descriptions can be found on the basic tutorial and the advanced tutorial pages.

3. Step 3: Running the experiment

In this step, you execute your experiment on an OMF-enabled testbed. This involves:

  • Getting an account on an OMF-enabled testbed (or install OMF on your own testbed...). A list of sites with OMF-enabled testbed is available on our deployment page. Currently, each of these sites has its own registration policies and procedures. As part of the ongoing federation effort, a current task aims at unifying these different procedures.
  • Making a reservation for some resources on this OMF-enabled testbed. Again here different institutions will have different access and reservation policies and procedures. For example, WINLAB uses a web-based scheduler to share the use of testbed resources among users.
  • Accessing the console for that testbed during your allocated reservation.

For example, assuming you have registered as the user 'alice' at NICTA, and made a reservation for all the resources on the 'sandbox1' testbed at NICTA.

You will then access the console of the 'sandbox1' testbed via SSH using the following command:

ssh alice@sandbox1.dynhost.nicta.com.au

After successful authentication, you will be logged in to the console of 'sanbox1'.

  • Uploading your Experiment Description, application definition, and potential software package to the console (using tools such as 'scp'). Of course, this step is not necessary, if you developed your experiment and appliation(s) on the testbed console itself.

For example, if your experiment description is named "myExp123.rb", you will run it using the command:

omf exec myExp123

For more information on the 'omf exec' command, please refer to the OMF tool manual pages.

Note: as part of the ongoing federation effort, a task is currently looking at allowing users to run experiment directly from their own machine, i.e. invoking the OMF Experiment Controller from their own computer.

4. Step 4: Accessing and exploiting the results

In this 4th step, you access the measurements that were collected during the execution of your experiment, and analyse them. This assumes that you did create/include OML Measurement Points within your software at the above Step 1, and that you explicitly required data to be collected from that Measurement Point in your ED at the above Step 2.

During the experiment execution, your measurement are collected by the OML clients on your experiment resources and forwarded to the OML Collection Server. The Collection Server stores the measurement in a unique database with the same ID as your experiment. This is explained in more detail in the OML section on the OMF introduction page.

Using your experiment ID (returned by the 'omf exec' command from Step 3), you can then directly run queries on your measurement database or request a full dump/copy of it, via the 'result' service of the Aggregate Manager of the testbed where you ran the experiment. An example of this is described in details in the basic tutorial pages.

You can then import the replies from your queries, or the full database dump, to your own choice of tools (Matlab, Excel, Gnuplot, etc...) for further processing and plotting.

5. Step 5: Experimenting further...

After analysing the result of your experiment, you may decide to:
  • run some more trials of the same experiment (back to Step 3)
  • vary some parameters of your experiments (back to Step 2)
  • change the type of measurement you would like to collect and the pre-processing to apply to them (back to Step 2)
  • change your software implementation to fix bugs or extend capabilities (back to Step 1)
  • add some additional Measurement Points to your software (back to Step 1)

6. Next...

OMF-Overview-Usage.png (94.5 kB) Thierry Rakotoarivelo, 21/10/2009 11:46 am