Metrics

MetricsManager

class nupic.frameworks.opf.predictionmetricsmanager.MetricsManager(metricSpecs, fieldInfo, inferenceType)

This is a class to handle the computation of metrics properly. This class takes in an inferenceType, and it assumes that it is associcated with a single model

getMetricDetails(metricLabel)

Gets detailed info about a given metric, in addition to its value. This may including any statistics or auxilary data that are computed for a given metric

metricLabel: The string label of the given metric (see metrics.MetricSpec)

Returns: A dictionary of metric information, as returned by

opf.metric.Metric.getMetric()

getMetricLabels()

Return the list of labels for the metrics that are being calculated

getMetrics()

Gets the current metric values

Returns: A dictionary where each key is the metric-name, and the values are

it scalar value. Same as the output of update()

update(results)

Compute the new metrics values, given the next inference/ground-truth values

results: An opfutils.ModelResult object that was computed during the last

iteration of the model

Returns: A dictionary where each key is the metric-name, and the values are

it scalar value.

MetricSpec

class nupic.frameworks.opf.metrics.MetricSpec(metric, inferenceElement, field=None, params=None)

This class represents a single Metrics specification in the TaskControl block

classmethod getInferenceTypeFromLabel(label)

Extracts the PredicitonKind (temporal vs. nontemporal) from the given metric label

label: A label (string) for a metric spec generated by getMetricLabel

(above)

Returns: An InferenceType value

getLabel(inferenceType=None)

Helper method that generates a unique label for a MetricSpec / InferenceType pair. The label is formatted as follows:

<predictionKind>:<metric type>:(paramName=value)*:field=<fieldname>

For example:

classification:aae:paramA=10.2:paramB=20:window=100:field=pounds

Metrics Interface

class nupic.frameworks.opf.metrics.MetricsIface(metricSpec)

A Metrics module compares a prediction Y to corresponding ground truth X and returns a single measure representing the “goodness” of the prediction. It is up to the implementation to determine how this comparison is made.

addInstance(groundTruth, prediction, record=None, result=None)

add one instance consisting of ground truth and a prediction.

groundTruth:

The actual measured value at the current timestep

prediction:

The value predicted by the network at the current timestep

groundTruthEncoding:

The binary encoding of the groundTruth value (as a numpy array). Right now this is only used by CLA networks

predictionEncoding:

The binary encoding of the prediction value (as a numpy array). Right now this is only used by CLA networks

result:

An ModelResult class (see opfutils.py)

return:

The average error as computed over the metric’s window size

getMetric()

return:

{value : <current measurement>, “stats” : {<stat> : <value> ...}} metric name is defined by the MetricIface implementation. stats is expected to contain further

information relevant to the given metric, for example the number of timesteps represented in the current measurement. all stats are implementation defined, and “stats” can be None

AggregateMetric

class nupic.frameworks.opf.metrics.AggregateMetric(metricSpec)

Bases: nupic.frameworks.opf.metrics.MetricsIface

Partial implementation of Metrics Interface for metrics that accumulate an error and compute an aggregate score, potentially over some window of previous data. This is a convenience class that can serve as the base class for a wide variety of metrics

accumulate(groundTruth, prediction, accumulatedError, historyBuffer, result)

Updates the accumulated error given the prediction and the ground truth.

groundTruth: Actual value that is observed for the current timestep

prediction: Value predicted by the network for the given timestep

accumulatedError: The total accumulated score from the previous

predictions (possibly over some finite window)

historyBuffer: A buffer of the last <self.window> ground truth values

that have been observed.

If historyBuffer = None, it means that no history is being kept.

result: An ModelResult class (see opfutils.py), used for advanced

metric calculation (e.g., MetricNegativeLogLikelihood)

retval:

The new accumulated error. That is: self.accumulatedError = self.accumulate(groundTruth, predictions, accumulatedError)

historyBuffer should also be updated in this method. self.spec.params[“window”] indicates the maximum size of the window

aggregate(accumulatedError, historyBuffer, steps)

Updates the final aggregated score error given the prediction and the ground truth.

accumulatedError: The total accumulated score from the previous

predictions (possibly over some finite window)

historyBuffer: A buffer of the last <self.window> ground truth values

that have been observed.

If historyBuffer = None, it means that no history is being kept.

steps: The total number of (groundTruth, prediction) pairs that have been passed to the metric. This does not include pairs where the groundTruth = SENTINEL_VALUE_FOR_MISSING_DATA

retval:

The new aggregate (final) error measure.

MetricPassThruPrediction

class nupic.frameworks.opf.metrics.MetricPassThruPrediction(metricSpec)

Bases: nupic.frameworks.opf.metrics.MetricsIface

This is not a metric, but rather a facility for passing the predictions generated by a baseline metric through to the prediction output cache produced by a model.

For example, if you wanted to see the predictions generated for the TwoGram metric, you would specify ‘PassThruPredictions’ as the ‘errorMetric’ parameter.

This metric class simply takes the prediction and outputs that as the aggregateMetric value.

addInstance(groundTruth, prediction, record=None, result=None)

Compute and store metric value

getMetric()

Return the metric value

CustomErrorMetric

class nupic.frameworks.opf.metrics.CustomErrorMetric(metricSpec)

Bases: nupic.frameworks.opf.metrics.MetricsIface

Custom Error Metric class that handles user defined error metrics

class CircularBuffer(length)

implementation of a fixed size constant random access circular buffer

CustomErrorMetric.expValue(pred)

Helper function to return a scalar value representing the expected value of a probability distribution

CustomErrorMetric.mostLikely(pred)

Helper function to return a scalar value representing the most likely outcome given a probability distribution

MetricMovingMode

class nupic.frameworks.opf.metrics.MetricMovingMode(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

computes error metric based on moving mode prediction

MetricTrivial

class nupic.frameworks.opf.metrics.MetricTrivial(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

computes a metric against the ground truth N steps ago. The metric to compute is designated by the ‘errorMetric’ entry in the metric params.

MetricTwoGram

class nupic.frameworks.opf.metrics.MetricTwoGram(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

computes error metric based on one-grams. The groundTruth passed into this metric is the encoded output of the field (an array of 1’s and 0’s).

MetricAccuracy

class nupic.frameworks.opf.metrics.MetricAccuracy(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

computes simple accuracy for an enumerated type. all inputs are treated as discrete members of a set, therefore for example 0.5 is only a correct response if the ground truth is exactly 0.5. Inputs can be strings, integers, or reals

MetricAveError

class nupic.frameworks.opf.metrics.MetricAveError(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

Simply the inverse of the Accuracy metric More consistent with scalar metrics because they all report an error to be minimized

MetricNegAUC

class nupic.frameworks.opf.metrics.MetricNegAUC(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

Computes -1 * AUC (Area Under the Curve) of the ROC (Receiver Operator Characteristics) curve. We compute -1 * AUC because metrics are optimized to be LOWER when running hypersearch.

For this, we assuming that category 1 is the “positive” category and we are generating an ROC curve with the TPR (True Positive Rate) of category 1 on the y-axis and the FPR (False Positive Rate) on the x-axis.

accumulate(groundTruth, prediction, accumulatedError, historyBuffer, result=None)

Accumulate history of groundTruth and “prediction” values.

For this metric, groundTruth is the actual category and “prediction” is a dict containing one top-level item with a key of 0 (meaning this is the 0-step classificaton) and a value which is another dict, which contains the probability for each category as output from the classifier. For example, this is what “prediction” would be if the classifier said that category 0 had a 0.6 probability and category 1 had a 0.4 probability: {0:0.6, 1: 0.4}

MetricMultiStep

class nupic.frameworks.opf.metrics.MetricMultiStep(metricSpec)

Bases: nupic.frameworks.opf.metrics.AggregateMetric

This is an “uber” metric which is used to apply one of the other basic metrics to a specific step in a multi-step prediction.

The specParams are expected to contain:

‘errorMetric’: name of basic metric to apply ‘steps’: compare prediction[‘steps’] to the current

ground truth.

Note that the metrics manager has already performed the time shifting for us - it passes us the prediction element from ‘steps’ steps ago and asks us to compare that to the current ground truth.

When multiple steps of prediction are requested, we average the results of the underlying metric for each step.