# Metric¶

A Metric specifies the fitness function measuring the performance of the simulation. This function gets applied on each simulated trace. A few metrics are already implemented and included in the toolbox, but the user can also provide their own metric.

## Mean Square Error¶

MSEMetric is provided for use with TraceFitter. It calculates the mean squared difference between the data and the simulated trace according to the well known formula:

$MSE ={\frac {1}{n}}\sum _{i=1}^{n}(Y_{i}-{\hat {Y_{i}}})^{2}$

It can be initialized in the following way:

metric = MSEMetric()


Additionally, MSEMetric accepts an optional input argument start time t_start (as a Quantity). The start time allows the user to ignore an initial period that will not be included in the error calculation.

metric = MSEMetric(t_start=5*ms)


Alternatively, the user can specify a weight vector emphasizing/de-emphasizing certain parts of the trace. For example, to ignore the first 5ms and to weigh the error (in the sense of the squared error) between 10 and 15ms twice as high as the rest:

# total trace length = 50ms
weights = np.ones(int(50*ms/dt))
weights[:int(5*ms/dt)] = 0
weights[int(10*ms/dt):int(15*ms/dt)] = 2
metric = MSEMetric(t_weights=weights)


Note that the t_weights argument cannot be combined with t_start.

In OnlineTraceFitter, the mean square error gets calculated in online manner, with no need of specifying a metric object.

## GammaFactor¶

GammaFactor is provided for use with SpikeFitter and measures the coincidence between spike times in the simulated and the target trace. It is calculcated according to:

$\Gamma = \left (\frac{2}{1-2\Delta r_{exp}}\right) \left(\frac{N_{coinc} - 2\delta N_{exp}r_{exp}}{N_{exp} + N_{model}}\right)$

$$N_{coinc}$$ - number of coincidences

$$N_{exp}$$ and $$N_{model}$$- number of spikes in experimental and model spike trains

$$r_{exp}$$ - average firing rate in experimental train

$$2 \Delta N_{exp}r_{exp}$$ - expected number of coincidences with a Poission process

For more details on the gamma factor, see:

The coincidence factor $$\Gamma$$ is 1 if the two spike trains match exactly and lower otherwise. It is 0 if the number of coincidences matches the number expected from two homogeneous Poisson processes of the same rate. To turn the coincidence factor into an error term (that is lower for better matches), two options are offered. With the rate_correction option (used by default), the error term used is $$2\frac{\lvert r_\mathrm{data} - r_\mathrm{model}\rvert}{r_\mathrm{data}} - \Gamma$$, with $$r_\mathrm{data}$$ and $$r_\mathrm{model}$$ being the firing rates in the data/model. This is useful because the coincidence factor $$\Gamma$$ on its own can give high values (low errors) if the model generates many more spikes than were observed in the data; this is penalized by the above term. If rate_correction is set to False, $$1 - \Gamma$$ is used as the error.

Upon initialization the user has to specify the $$\Delta$$ value, defining the maximal tolerance for spikes to be considered coincident:

metric = GammaFactor(delta=2*ms)


Warning

The delta parameter has to be smaller than the smallest inter-spike interval in the spike trains.

## FeatureMetric¶

FeatureMetric is provided for use with TraceFitter. This metric allows the user to optimize the match of certain features between the simulated and the target trace. The features get calculated by Electrophys Feature Extract Library (eFEL) library, for which the documentation is available under following link: https://efel.readthedocs.io

To get a list of all the available eFEL features, you can run the following code:

import efel
efel.api.getFeatureNames()


Note

Currently, only features that are described by a single value are supported (e.g. the time of the first spike can be used, but not the times of all spikes).

To use the FeatureMetric, you have to provide the following input parameters:

• stim_times - a list of times indicating start and end of the stimulus for each of input traces. This information is used by several features, e.g. the voltage_base feature will consider the average membrane potential during the last 10% of time before the stimulus (see the eFel documentation for details).
• feat_list - list of strings with names of features to be used
• combine - function to be used to compare features between output and simulated traces (uses the absolute difference between the values by default).

Example code usage:

stim_times = [(50*ms, 100*ms), (50*ms, 100*ms), (50*ms, 100*ms), (50, 100*ms)]
feat_list = ['voltage_base', 'time_to_first_spike', 'Spikecount']
metric = FeatureMetric(traces_times, feat_list, combine=None)


Note

If times of stimulation are the same for all of the traces, then you can specify a single interval instead: traces_times = [(50*ms, 100*ms)].

## Custom Metric¶

Users are not limited to the metrics provided in the toolbox. If needed, they can provide their own metric based on one of the abstract classes TraceMetric and SpikeMetric.

A new metric will need to specify the following functions:

• get_features()
calculates features / errors for each of the simulations. The representation of the model results and the target data depend on whether traces or spikes are fitted, see below.
• get_errors()
weights features/multiple errors into one final error per each set of parameters and inputs. The features are received as a 2-dimensional ndarray of shape (n_samples, n_traces) The output has to be an array of length n_samples, i.e. one value for each parameter set.
• calc()
performs the error calculation across simulation for all parameters of each round. Already implemented in the abstract class and therefore does not need to be reimplemented necessarily.

### TraceMetric¶

To create a new metric for TraceFitter, you have to inherit from TraceMetric and overwrite the get_features and/or get_errors method. The model traces for the get_features function are provided as a 3-dimensional ndarray of shape (n_samples, n_traces, time steps), where n_samples are the number of different parameter sets that have been evaluated, and n_traces the number of different stimuli that have been evaluated for each parameter set. The output of the function has to take the shape of (n_samples, n_traces). This array is the input to the get_errors method (see above).

class NewTraceMetric(TraceMetric):
def get_features(self, model_traces, data_traces, dt):
...

def get_errors(self, features):
...


### SpikeMetric¶

To create a new metric for SpikeFitter, you have to inherit from SpikeMetric. Inputs of the metric in get_features are a nested list structure for the spikes generated by the model: a list where each element contains the results for a single parameter set. Each of these results is a list for each of the input traces, where the elements of this list are numpy arrays of spike times (without units, i.e. in seconds). For example, if two parameters sets and 3 different input stimuli were tested, this structure could look like this:

[
[array([0.01, 0.5]), array([]), array([])],
[array([0.02]), array([]), array([])]
]


This means that the both parameter sets only generate spikes for the first input stimulus, but the first parameter sets generates two while the second generates only a single one.

The target spikes are represented in the same way as a list of spike times for each input stimulus. The results of the function have to be returned as in TraceMetric, i.e. as a 2-d array of shape (n_samples, n_traces).