btb.session module

class btb.session.BTBSession(tunables, scorer, tuner_class=<class 'btb.tuning.tuners.gaussian_process.GPTuner'>, selector_class=<class 'btb.selection.ucb1.UCB1'>, maximize=True, max_errors=1, verbose=False)[source]

Bases: object

BTBSession class.

A BTBSession represents the process of selecting and tuning several tunables until the best possible configuration for a specific scorer is found.

For this, a loop is run in which for each iteration a combination of a Selector and Tuner is used to decide which tunable to score next and with which hyperparameters.

While running, the BTBSession handles the errors discarding, if configured to do so, the tunables that have reached as many errors as the user specified.

best_proposal

Best configuration found with the name of the tunable and the hyperparameters and crossvalidated score obtained for it.

Type

dict

best_score

Best score obtained for this session so far.

Type

float

proposals

Dictionary containing all the proposals generated by the BTBSession.

Type

dict

iterations

Amount of iterations run.

Type

int

errors

A Counter of the errors that each Tunable had during the session.

Type

Counter

Parameters

  • tunables (dict) – Python dictionary that has as keys the name of the tunable and as value a dictionary with the tunable hyperparameters or an btb.tuning.tunable.Tunable instance.

  • scorer (callable object / function) – A callable object or function with signature scorer(tunable_name, config) wich should return only a single value.

  • tuner_class (btb.tuning.tuner.BaseTuner) – A tuner based on BTB BaseTuner class. This tuner will manage the new proposals. Defaults to btb.tuning.tuners.gaussian_process.GPTuner

  • selector_class (btb.selection.selector.Selector) – A selector based on BTB Selector class. This will determinate which one of the tunables is performing better, and which one to test next. Defaults to btb.selection.selectors.ucb1.UCB1

  • maximize (bool) – If True the scores are interpreted as bigger is better, if False then smaller is better, this should depend on the problem type (maximization or minimization). Defaults to True.

  • max_erors (int) – Amount of errors allowed for a tunable to not generate a score. Once this amount of errors is reached, the tunable will be removed from the list. Defaults to 1.

  • verbose (bool) – If True a progress bar will be displayed for the run process.

best_proposal = None
best_score = None
errors = None
handle_error(tunable_name)[source]

Handle errors when score is None.

If the given tunable_name accumulates more errors than self._max_errors this is removed from the selector’s choices.

Parameters

tunable_name (str) – The name of the tunable to which this configuration belongs.

iterations = None
proposals = None
propose()[source]

Propose a new configuration to score.

Every time propose is called, a new tunable will be selected and a new hyperparameter proposal will be generated for it.

At the begining, the default hyperparameters of each one of the tunables will be returned sequencially in the same order as they were passed to the BTBSession.

After that, once each tunable has been scored at least once, the tunable used to generate the new proposals will be selected optimally each time by the selector.

If a tunable runs out of proposals, it will be discarded from the list and will not be proposed again.

Finally, when all the tunables have ran out of proposals, a StopTuning exception will be raised.

Returns

  • Name of the tunable to try next.

  • Hyperparameters proposal.

Return type

tuple (str, dict)

Raises

StopTuning – If the BTBSession has run out of proposals to generate.

record(tunable_name, config, score)[source]

Record the configuration and the obtained score to the tuner.

If the score is the best one so far, the best_proposal and best_score are updated.

Parameters

  • tunable_name (str) – The name of the tunable to which this configuration belongs.

  • config (dict) – Hyperparameter proposal, as given by the tunable.

  • score (float) – Obtained score with the given configuration.

run(iterations=None)[source]

Run the selection and tuning loop for the given number of iterations.

At each iteration, the BTBSession will generate a new proposal calling self.propose, score it using the self.scorer, and finally record the obtained score back to the tuner calling self.record.

If no iterations are given, run infinitely until interrupted or until all the tuner proposals are exhausted.

Scoring errors will also be captured and recorded.

Returns

Best configuration found with the name of the tunable and the hyperparameters and crossvalidated score obtained for it.

Return type

best_proposal (dict)