Train Config

class neural_pipeline.train_config.train_config.TrainConfig(train_stages: [], loss: torch.nn.modules.module.Module, optimizer: torch.optim.optimizer.Optimizer)[source]

Train process setting storage

Parameters:
  • train_stages – list of stages for train loop
  • loss – loss criterion
  • optimizer – optimizer object
loss() → torch.nn.modules.module.Module[source]

Get loss object

Returns:loss object
optimizer() → torch.optim.optimizer.Optimizer[source]

Get optimizer object

Returns:optimizer object
stages() → [<class 'neural_pipeline.train_config.train_config.AbstractStage'>][source]

Get list of stages

Returns:list of stages
class neural_pipeline.train_config.train_config.TrainStage(data_producer: neural_pipeline.data_producer.data_producer.DataProducer, metrics_processor: neural_pipeline.train_config.train_config.MetricsProcessor = None, name: str = 'train')[source]

Standard training stage

When call run() it’s iterate process_batch() of data processor by data loader with is_tran=True flag.

After stop iteration ValidationStage accumulate losses from DataProcessor.

Parameters:
  • data_producer – DataProducer object
  • metrics_processor – MetricsProcessor
  • name – name of stage. By default ‘train’
disable_hard_negative_mining() → neural_pipeline.train_config.train_config.TrainStage[source]

Enable hard negative mining.

Returns:self object
enable_hard_negative_mining(part: float) → neural_pipeline.train_config.train_config.TrainStage[source]

Enable hard negative mining. Hard negatives was taken by losses values

Parameters:part – part of data that repeat after train stage
Returns:self object
on_epoch_end()[source]

Method, that calls after every epoch

run(data_processor: neural_pipeline.data_processor.data_processor.TrainDataProcessor) → None[source]

Run stage

Parameters:data_processor – TrainDataProcessor object
class neural_pipeline.train_config.train_config.ValidationStage(data_producer: neural_pipeline.data_producer.data_producer.DataProducer, metrics_processor: neural_pipeline.train_config.train_config.MetricsProcessor = None, name: str = 'validation')[source]

Standard validation stage.

When call run() it’s iterate process_batch() of data processor by data loader with is_tran=False flag.

After stop iteration ValidationStage accumulate losses from DataProcessor.

Parameters:
  • data_producer – DataProducer object
  • metrics_processor – MetricsProcessor
  • name – name of stage. By default ‘validation’
class neural_pipeline.train_config.train_config.AbstractMetric(name: str)[source]

Abstract class for metrics. When it works in neural_pipeline, it store metric value for every call of calc()

Parameters:name – name of metric. Name wil be used in monitors, so be careful in use unsupported characters
calc(output: torch.Tensor, target: torch.Tensor) → numpy.ndarray[source]

Calculate metric by output from model and target

Parameters:
  • output – output from model
  • target – ground truth
get_values() → numpy.ndarray[source]

Get array of metric values

Returns:array of values
static max_val() → float[source]

Get maximum value of metric. This used for correct histogram visualisation in some monitors

Returns:maximum value
static min_val() → float[source]

Get minimum value of metric. This used for correct histogram visualisation in some monitors

Returns:minimum value
name() → str[source]

Get name of metric

Returns:metric name
reset() → None[source]

Reset array of metric values

class neural_pipeline.train_config.train_config.MetricsGroup(name: str)[source]

Class for unite metrics or another MetricsGroup’s in one namespace. Note: MetricsGroup may contain only 2 level of MetricsGroup’s. So MetricsGroup().add(MetricsGroup().add(MetricsGroup())) will raises MGException

Parameters:name – group name. Name wil be used in monitors, so be careful in use unsupported characters
exception MGException(msg: str)[source]

Exception for MetricsGroup

add(item: neural_pipeline.train_config.train_config.AbstractMetric) → neural_pipeline.train_config.train_config.MetricsGroup[source]

Add AbstractMetric or MetricsGroup

Parameters:item – object to add
Returns:self object
Return type:MetricsGroup
calc(output: torch.Tensor, target: torch.Tensor) → None[source]

Recursive calculate all metrics in this group and all nested group

Parameters:
  • output – predict value
  • target – target value
groups() → ['MetricsGroup'][source]

Get list of metrics groups

Returns:list of metrics groups
have_groups() → bool[source]

Is this group contains another metrics groups

Returns:True if contains, otherwise - False
metrics() → [<class 'neural_pipeline.train_config.train_config.AbstractMetric'>][source]

Get list of metrics

Returns:list of metrics
name() → str[source]

Get group name

Returns:name
reset() → None[source]

Recursive reset all metrics in this group and all nested group

class neural_pipeline.train_config.train_config.MetricsProcessor[source]

Collection for all AbstractMetric’s and MetricsGroup’s

add_metric(metric: neural_pipeline.train_config.train_config.AbstractMetric) → neural_pipeline.train_config.train_config.AbstractMetric[source]

Add AbstractMetric object

Parameters:metric – metric to add
Returns:metric object
Return type:AbstractMetric
add_metrics_group(group: neural_pipeline.train_config.train_config.MetricsGroup) → neural_pipeline.train_config.train_config.MetricsGroup[source]

Add MetricsGroup object

Parameters:group – metrics group to add
Returns:metrics group object
Return type:MetricsGroup
calc_metrics(output, target) → None[source]

Recursive calculate all metrics

Parameters:
  • output – predict value
  • target – target value
get_metrics() → {}[source]

Get metrics and groups as dict

Returns:dict of metrics and groups with keys [metrics, groups]
reset_metrics() → None[source]

Recursive reset all metrics values

class neural_pipeline.train_config.train_config.AbstractStage(name: str)[source]

Stage of training process. For example there may be 2 stages: train and validation. Every epochs in train loop is iteration by stages.

Parameters:name – name of stage
get_losses() → numpy.ndarray[source]

Get losses from this stage

Returns:array of losses or None if this stage doesn’t need losses
metrics_processor() → neural_pipeline.train_config.train_config.MetricsProcessor[source]

Get metrics processor

Returns::class:’MetricsProcessor` object or None
name() → str[source]

Get name of stage

Returns:name
on_epoch_end() → None[source]

Callback for train epoch end

run(data_processor: neural_pipeline.data_processor.data_processor.TrainDataProcessor) → None[source]

Run stage

class neural_pipeline.train_config.train_config.StandardStage(stage_name: str, is_train: bool, data_producer: neural_pipeline.data_producer.data_producer.DataProducer, metrics_processor: neural_pipeline.train_config.train_config.MetricsProcessor = None)[source]

Standard stage for train process.

When call run() it’s iterate process_batch() of data processor by data loader

After stop iteration ValidationStage accumulate losses from DataProcessor.

Parameters:
get_losses() → numpy.ndarray[source]

Get losses from this stage

Returns:array of losses
metrics_processor() → neural_pipeline.train_config.train_config.MetricsProcessor[source]

Get merics processor of this stage

Returns:MetricsProcessor if specified otherwise None
on_epoch_end() → None[source]

Method, that calls after every epoch

run(data_processor: neural_pipeline.data_processor.data_processor.TrainDataProcessor) → None[source]

Run stage. This iterate by DataProducer and show progress in stdout

Parameters:data_processor – DataProcessor object