Data Processor

class neural_pipeline.data_processor.data_processor.DataProcessor(model: torch.nn.modules.module.Module, device: torch.device = None)

DataProcessor manage: model, data processing, device choosing

Parameters:

• model – model, that will be used for process data
• device – what device pass model and data for processing

load() → None

Load model weights from checkpoint

model() → torch.nn.modules.module.Module

Get current module

predict(data: torch.Tensor) → object

Make predict by data

Parameters:data – data as torch.Tensor or dict with key data Returns:processed output Return type:the model output type

save_state() → None

Save state of optimizer and perform epochs number

class neural_pipeline.data_processor.data_processor.TrainDataProcessor(model: torch.nn.modules.module.Module, train_config: TrainConfig, device: torch.device = None)

TrainDataProcessor is make all of DataProcessor but produce training process.

Parameters:

• model – model, that will be used for process data
• train_config – train config
• device – what device pass model, data and optimizer for processing

exception TDPException(msg)

get_lr() → float

Get learning rate from optimizer

get_state() → {}

Get model and optimizer state dicts

Returns:dict with keys [weights, optimizer]

load() → None

Load state of model, optimizer and TrainDataProcessor from checkpoint

predict(data, is_train=False) → torch.Tensor

Make predict by data. If is_train was True

Parameters:

• data – data in dict
• is_train – is data processor need train on data or just predict

Returns:

processed output

Return type:

model return type

process_batch(batch: {}, is_train: bool, metrics_processor: AbstractMetricsProcessor = None) → numpy.ndarray

Process one batch of data

Parameters:

• batch – dict, contains ‘data’ and ‘target’ keys. The values for key must be instance of torch.Tensor or dict
• is_train – is batch process for train
• metrics_processor – metrics processor for collect metrics after batch is processed

Returns:

array of losses with shape (N, …) where N is batch size

save_state() → None

Save state of optimizer and perform epochs number

update_lr(lr: float) → None

Update learning rate straight to optimizer

Parameters:lr – target learning rate

Model

class neural_pipeline.data_processor.model.Model(base_model: torch.nn.modules.module.Module)

Wrapper for torch.nn.Module. This class provide initialization, call and serialization for it

Parameters:base_model – torch.nn.Module object

exception ModelException(msg)

load_weights(weights_file: str = None) → None

Load weight from checkpoint

model() → torch.nn.modules.module.Module

Get internal torch.nn.Module object

Returns:internal torch.nn.Module object

save_weights(weights_file: str = None) → None

Serialize weights to file

set_checkpoints_manager(manager: neural_pipeline.utils.file_structure_manager.CheckpointsManager) → neural_pipeline.data_processor.model.Model

Set checkpoints manager, that will be used for identify path for weights file reading an writing

Parameters:manager – CheckpointsManager instance Returns:self object

to_device(device: torch.device) → neural_pipeline.data_processor.model.Model

Pass model to specified device