Multiple Runs with Open Grid Engine

Warning

This tutorial is old and hasn’t been updated in a while. While the general process is accurate, the program names and arguments may have changed. It is recommended that you understand how to run multiple scenarios on your local computer first. Please refer to Multiple Runs on a Local Computer first.

Running single simulations is helpful for directly observing autonomous behaviors. However, if you want to statistically compare various algorithms, you will need to generate hundreds or thousands of runs. Luckily, Scrimmage ships with scripts that enable it to run on a cluster or a single computer running Open Grid Engine.

  1. Install dependencies on Ubuntu:

    $ sudo apt-get install python-pip python-scipy python-pandas

    Install Python Pandas:

    $ sudo pip install pyDOE

  2. Now you need to configure Open Grid Engine (OGE) to run on your local machine. Follow the instructions at the following website to configure Open Grid Engine for a single computer: https://scidom.wordpress.com/2012/01/18/sge-on-single-pc/ . Start at “Part 1: Installation of SGE” and you can finish before “Part 3: Elementary Test Runs on SGE.”

    Note

    When you run sudo qmon the GUIs icons may not appear. To fix this problem, execute the follow commands in a terminal:

    $ mkdir -p $HOME/icons/Qmon
$ cd $HOME/icons/Qmon
$ for f in /usr/share/gridengine/pixmaps/*; do ln -sf $f $(basename ${f%.xpm}); done

  3. Submit 10 simulations to Open Grid Engine:

    $ cd ~/scrimmage

$ python scrimmage/scripts/generate_scenarios.py \
   --ranges scrimmage/ranges/test-1.xml \
   --num_runs=10 \
   scrimmage-sasc-plugins/missions/capture_the_flag-cpp.xml \
   doe_cache_dir

This will create 10 variations of the capture_the_flag-cpp mission by varying parameters found in the test-1.xml file. The mission files and text output from each mission will be saved in doe_cache_dir. The ranges file is optional.

  1. You can observe the status of the submitted jobs:

    $ qstat

    You can delete a running job:

    $ qdel <job-ID>

    Also, you can use the sudo qmon GUI to monitor and delete jobs in the “Job Control” section.

  2. The simulation results are stored in the log_dir location. After the simulation is complete, you can use a number of tools to process the simulation results. Run the aggregate-runs program to count the number of victories between the teams in the simulations based on the score weights defined in an XML file:

    $ aggregate-runs -d ~/swarm-log \
-w ~/scrimmage/scrimmage-sasc-plugins/config/weights/sasc.xml

    The -w flag specifies the XML score weights file to use. The aggregate-runs program creates text files at ~/swarm-log/aggregate/wins that specify the scenarios in which each team won.

  3. The filter-runs program reads the generated text files and allows for easy playback of each type of scenario.:

    $ filter-runs ~/swarm-log

    Running filter-runs will print output similar to the following:

    ====================================================
Choose an outcome number:
----------------------------------------------------
Number             Name            Win Count
----------------------------------------------------
[0]                draw            1
[1]                team_1          7
[2]                team_2          3
>>

    To view the simulation runs where team_1 was the winner, type the number 1, hit enter, and follow the remaing prompts to step through the playback of each simulation.

The generate_scenarios.sh generates multiple scenarios from a single base scenario through a sampling process called “Latin Hypercube Sampling” (https://en.wikipedia.org/wiki/Latin_hypercube_sampling). While, a base scenario file is provided, the LHS algorithm needs to know how many and which parameters can be varied in the base scenario. The “ranges” file defines the parameters that should be varied in the base mission file and defines minimum and maximum values for each parameter. An example can be helpful. If we look at the test-1.xml ranges file located at ~/scrimmage/scrimmage/ranges/, we see several XML tags that have related values in a base scenario file. For example, we see:

<x low="0" high="2000" type="float"/>

This line tells the LHS algorithm that the x tag in the base mission file can be varied from 0 to 2000 and that the x tag is of type float. Most parameters in the base mission file can be varied in a similar way.