Metrics

Both RunGames and ParameterSearch (see Running Games, Parameter Tuning) allow bespoke reporting of matters of interest. In most cases the MetricsGameListener can be used as is; and new game-specific Metrics can be added quite easily.

The example below shows the outline format of the JSON definition. (This uses the GeneralGameResultListener.json example in data/listeners.) Providing this file to the listener argument in RunGames will automatically log the required data.

{
  "class": "evaluation.listeners.MetricsGameListener",
  "args":  [
	{"enum" : "evaluation.metrics.IDataLogger$ReportDestination", "value" : "ToFile"},
    [
      {"enum": "evaluation.metrics.IDataLogger$ReportType", "value": "RawData"},
      {"enum": "evaluation.metrics.IDataLogger$ReportType", "value": "Summary"}
    ],
    [
      {"class": "evaluation.metrics.GameMetrics$FinalScore" },
      {"class": "evaluation.metrics.GameMetrics$OrdinalPosition" },
      {"class": "evaluation.metrics.GameMetrics$PlayerType" },
      {"class": "evaluation.metrics.GameMetrics$Winner" },
      {"class": "evaluation.metrics.GameMetrics$Actions" }

  ]]
}

The key sections to be defined are:

Description Class Values
Where to record Data IDataLogger$ReportDestination ToFile, ToConsole, ToBoth. The file location is then determined by the destDir parameter for both RunGames and ParameterSearch
What type of records IDataLogger$ReportType RawData, Summary, Plot, RawDataPerEvent. RawData will produce one file for each of the Metrics, with all the values; Summary generates summary statistcis for each (min, max, mean, sd etc.), and Plot generates graphs for each, which can be helpful. RawDataPerEvent is useful if you want to extract raw data for processing externally (with Python or R data analysis scripts for example). This will create one file for each event (GAME_OVER, ACTION_CHOSEN, etc.) with one record for each matching event, and one column for each corresponding Metric.
The data items to report any implementation of AbstractMetric In the example above this uses some generically useful fields - the final score of each player, their ordinal positions, the agents used (PlayerType), the Actions they took at each decision, and the winner of each game.

Each Metric has a default set of related events. In the example above GameMetrics$FinalScore and Winner are triggered only by a GAME_OVER event, while GameMetrics$Actions is triggered by an ACTION_CHOSEN event. There is another metric, GameMetrics$GameScore, not included above, which is triggered by all of GAME_OVER, ACTION_CHOSEN and ROUND_OVER; if that is the level of granularity you need. General game metrics can be found in evaluation.metrics.GameMetrics; there are some metric on MCTS statistics in players.mcts.MCTSMetrics', and a corresponding default JSON definition in data/listeners’.

The possible Events are defined by evaluation.metrics.Event

  • ABOUT_TO_START
  • GAME_OVER
  • ROUND_OVER
  • TURN_OVER
  • ACTION_CHOSEN : immediately after an Action is chosen, but before it is implemented
  • ACTION_TAKEN : immediately after an Action has been implemented and the game state advanced
  • GAME_EVENT : a game-specific event that is worthy of notice and is not linked directly to a player action

Default generation of these event has been added to the core framework at appropriate places (mostly in StandardForwardModel), and they can be overridden as required for specific games.

GAME_EVENT is the one event that is not implemented in any default part of the framework. It is designed to be used in a game when an event occurs that is important to log to understand how the game is going. One example might be if a player is knocked out of the game on another player’s turn. Generating a GAME_EVENT with appropriate detail text will ensure that this is logged appropriately.

Bespoke IGameListeners

For most cases the Metrics system above is sufficient, but bespoke implementations of the IGameListener interface can listen to a Game (with a classic Observer pattern) and do anything required on receipt of a game event by implementing the onEvent and onGameEvent methods (see class documentation for details).

IStateFeatureVector / IActionFeatureVector

These interfaces take a state, or a (state, action) pair and generate a numeric array (double[]). In contrast to the AbstractMetric interface, these are useful for output that is designed to be machine-readable, and to project any state or action into an n-dimensional vector. Examples of IStateFeatureVector are implemented currently for Dots and Boxes, Love Letter and Dominion.

Two useful IGameListener implementations are provided to generate game trajectories at the desired level of granularity: StateFeatureListener and ActionFeatureListener. These can be configured with a specific IStatisticLogger, GameEvent to define when a snapshot should be recorded, and the I[State|Action]FeatureVector that defines the snapshot vector.

When using RunGames, this can all be defined easily in a JSON config file as above. An example is below:

{
    "class" : "utilities.StateFeatureListener",
    "args" : [
        { "class" : "games.dotsboxes.DBStateFeatures" },
        { 
            "enum" : "evaluation.metrics.Event$GameEvent", 
            "value" : "ACTION_TAKEN"
        },
        false
    ]
}

In this example we want a StateFeatureListener (the class in the top-level). We then list the args that the constructor takes. In this case there are three arguments: IStateFeatureVector, GameEvent, boolean. The values for each of these are specified in the top-level args array (inside the […]) - and must be in the same order as the relevant class constructor.

This is recursive to allow quite complex details to be configured. At the second level of nesting in this example we specify:

  • The implementation of IStateFeatureVector to use is DBStateFeatures, which has a no-arg constructor.
  • We wish to record data every time an action is taken (any one of the valid GameEvent constants is valid, so we could have chosen to record at the end of a player’s turn for example).
  • Finally, the false indicates that we want to record data for every player at each event (setting this to true would only record data for the current player at that point in the game).

The valid JSON tags are:

  • class. The full package path of the class to be used.
  • args. An array of the values to use in the class constructor. These must be in the order they appear in the class constructor.
  • enum. A special case for when the class is an enum. This replaces the use of class.
  • value. Provides the value to use for an emum, and only valid in this situation.

The JSON values can then be any of a String, Integer, Double, Boolean.