File structure management utils

This module contains all classes, that work with file structure

class neural_pipeline.utils.file_structure_manager.FileStructManager(base_dir: str, is_continue: bool, exists_ok: bool = False)[source]

Class, that provide directories registration in base directory.

All modules, that use file structure under base directory should register their paths in this class by pass module to method register_dir(). If directory also registered registration method will raise exception FSMException

  • base_dir – path to directory with checkpoints
  • is_continue – is FileStructManager used for continue training or predict
  • exists_ok – if True - all checks for existing directories will be disabled
exception FSMException(message: str)[source]
get_path(obj: neural_pipeline.utils.file_structure_manager.FolderRegistrable, create_if_non_exists: bool = False, check: bool = True) → str[source]

Get path of registered object

  • obj – object
  • create_if_non_exists – is need to create object’s directory if it doesn’t exists
  • check – is need to check object’s directory existing

path to directory


FSMException – if directory exists and check == True

in_continue_mode() → bool[source]

Is FileStructManager in continue mode

Returns:True if in continue
register_dir(obj: neural_pipeline.utils.file_structure_manager.FolderRegistrable, check_name_registered: bool = True, check_dir_registered: bool = True) → None[source]

Register directory in file structure

  • obj – object to registration
  • check_name_registered – is need to check if object name also registered
  • check_dir_registered – is need to check if object path also registered

FileStructManager – if path or object name also registered and if path also exists (in depends of optional parameters values)

class neural_pipeline.utils.file_structure_manager.CheckpointsManager(fsm: neural_pipeline.utils.file_structure_manager.FileStructManager, prefix: str = None)[source]

Class that manage checkpoints for DataProcessor.

All states pack to zip file. It contains few files: model weights, optimizer state, data processor state

  • fsm – :class:’FileStructureManager’ instance
  • prefix – prefix of saved and loaded files
exception SMException(message: str)[source]

Exception for CheckpointsManager

clear_files() → None[source]

Clear unpacked files

optimizer_state_file() → str[source]

Get optimizer state file path

pack() → None[source]

Pack all files in zip

trainer_file() → str[source]

Get trainer state file path

unpack() → None[source]

Unpack state files

weights_file() → str[source]

Get model weights file path

class neural_pipeline.utils.file_structure_manager.FolderRegistrable(fsm: neural_pipeline.utils.file_structure_manager.FileStructManager)[source]

Abstract class for implement classes, that use folders

Parameters:fsm – FileStructureManager class instance