File structure management utils

This module contains all classes, that work with file structure

• FileStructManager provide all modules registration
• CheckpointsManager provide checkpoints management

class neural_pipeline.utils.file_structure_manager.FileStructManager(base_dir: str, is_continue: bool, exists_ok: bool = False)

Class, that provide directories registration in base directory.

All modules, that use file structure under base directory should register their paths in this class by pass module to method register_dir(). If directory also registered registration method will raise exception FSMException

Parameters:

• base_dir – path to directory with checkpoints
• is_continue – is FileStructManager used for continue training or predict
• exists_ok – if True - all checks for existing directories will be disabled

exception FSMException(message: str)

get_path(obj: neural_pipeline.utils.file_structure_manager.FolderRegistrable, create_if_non_exists: bool = False, check: bool = True) → str

Get path of registered object

Parameters:

• obj – object
• create_if_non_exists – is need to create object’s directory if it doesn’t exists
• check – is need to check object’s directory existing

Returns:

path to directory

Raises:

FSMException – if directory exists and check == True

in_continue_mode() → bool

Is FileStructManager in continue mode

Returns:True if in continue

register_dir(obj: neural_pipeline.utils.file_structure_manager.FolderRegistrable, check_name_registered: bool = True, check_dir_registered: bool = True) → None

Register directory in file structure

Parameters:

• obj – object to registration
• check_name_registered – is need to check if object name also registered
• check_dir_registered – is need to check if object path also registered

Raises:

FileStructManager – if path or object name also registered and if path also exists (in depends of optional parameters values)

class neural_pipeline.utils.file_structure_manager.CheckpointsManager(fsm: neural_pipeline.utils.file_structure_manager.FileStructManager, prefix: str = None)

Class that manage checkpoints for DataProcessor.

All states pack to zip file. It contains few files: model weights, optimizer state, data processor state

Parameters:

• fsm – :class:’FileStructureManager’ instance
• prefix – prefix of saved and loaded files

exception SMException(message: str)

Exception for CheckpointsManager

clear_files() → None

Clear unpacked files

optimizer_state_file() → str

Get optimizer state file path

Returns:path

pack() → None

Pack all files in zip

trainer_file() → str

Get trainer state file path

Returns:path

unpack() → None

Unpack state files

weights_file() → str

Get model weights file path

Returns:path

class neural_pipeline.utils.file_structure_manager.FolderRegistrable(fsm: neural_pipeline.utils.file_structure_manager.FileStructManager)

Abstract class for implement classes, that use folders

Parameters:fsm – FileStructureManager class instance