Create a Metrics Plugin

A Metrics plugin is a fairly abstract plugin that allows a developer to keep track of continuous variables in the simulation or count messages generated by entities and Entity Interaction plugins. At the end of a simulation, the SCRIMMAGE simulation controller requests the team_metrics_ member variable from a Metrics plugin and writes the value for the team’s metrics to a CSV file for later processing. This is how SCRIMMAGE is able to determine the success of an algorithm across a large number of scenarios. A common use of the Metrics plugin interface is to count collisions generated by the Entity Interaction plugins. To create a Metrics plugin, enter the following at the terminal:

$ cd /path/to/scrimmage/scripts
$ ./generate-plugin.sh metrics MyMetrics ~/scrimmage/my-scrimmage-plugins

Build the metrics plugin:

$ cd ~/scrimmage/my-scrimmage-plugins/build
$ cmake ..
$ make

Examine Metrics Plugin

The init method can be used to save variables from the plugin’s XML file. The automatically generated plugin counts the number of ground collision messages that are generated by the Entity Interaction plugin, so in this init method, a subscriber for the GroundCollision message is setup.

1
2
3
4
void MyMetrics::init(std::map<std::string,std::string> &params)
{
    sub_ground_collision_ = create_subscriber("GroundCollision");
}

The step_metrics method is also simple in that it processes the ground collision messages that are generated and counts them.

1
2
3
4
5
6
7
bool MyMetrics::step_metrics(double t, double dt)
{
    for (auto msg : sub_ground_collision_->msgs<sc::Message<sm::GroundCollision>>()) {
        scores_[msg->data.entity_id()].increment_ground_collisions();
    }
    return true;
}

The step_metrics method makes use of the helper class, Score, which is defined in MyMetrics.h. The Score class is a class that allows the user to increment, set, and get counts of various variables or metrics. Also, the Score class computes the weighted score based on the number of counts of a metric and the associated weight value defined in the plugin’s XML file. For example, if the plugin’s XML file contains the tag:

<ground_collisions>-1.0</ground_collisions>

and there were 5 ground collisions. The score for that team would be -1.0 times 5, which is -5. In this example, the Score class is used to first compute single entity metrics, which are then aggregated to a team Score at the end of the simulation.

When the simulation is complete, the simulation controller calls the calc_team_scores method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
void MyMetrics::calc_team_scores()
{
    for (auto &kv : scores_) {
        Score &score = kv.second;

        int team_id = (*team_lookup_)[kv.first];

        // Create the score, if necessary
        if (team_coll_scores_.count(team_id) == 0) {
            Score score;
            score.set_weights(params_);
            team_coll_scores_[team_id] = score;
        }
        team_coll_scores_[team_id].add_ground_collisions(score.ground_collisions());
    }

    for (auto &kv : team_coll_scores_) {
        int team_id = kv.first;
        Score &score = kv.second;
        team_metrics_[team_id]["ground_coll"] = score.ground_collisions();
        team_scores_[team_id] = score.score();
    }

    // list the headers we want put in the csv file
    headers_.push_back("ground_coll");
}

It is the responsibility of the Metrics plugin to fill in the team_scores_ map, which is a map of team IDs that point to weighted team scores. Also, the team_metrics_ map has to be filled in, which is a map of team IDs, which point to metric names (as strings) with their values. The Metrics plugin developer also has to provide header names for the CSV file is generated. This is done by appending to the headers_ list. The strings in the headers_ should match those in the team_metrics_.

Finally, the simulation controller will call the Metrics plugin’s print_team_summaries method to display scores to the terminal.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
void MyMetrics::print_team_summaries()
{
    for (std::map<int, Score>::iterator it = team_coll_scores_.begin();
         it != team_coll_scores_.end(); it++) {

        cout << "Score: " << it->second.score() << endl;
        cout << "Ground Collisions: " << it->second.ground_collisions() << endl;
        cout << sc::generate_chars("-",70) << endl;
    }
}

Note

The Metrics plugin’s “score” functionality will probably be augmented in the future since it’s sort of an awkward interface right now. Take a look at the SimpleCollisionMetrics plugin in SCRIMMAGE core for a more complete example.