cascade.lines#

Container for tracking Datasets. Can pickle data pipelines completely or just used to store metadata. Uses versioning system and meta hashing to tell datasets apart and assign versions.

__init__(root: str, ds_cls: ~typing.Type[~typing.Any] = <class 'cascade.data.dataset.Dataset'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', obj_backend: ~typing.Literal['pickle'] = 'pickle', *args: ~typing.Any, **kwargs: ~typing.Any) → None[source]#

Parameters:

• description – String description of an object

• tags (Iterable[str], optional) – The list of tags to be added

See also

cascade.base.MetaHandler

get_latest_version() → Optional[Version][source]#

Returns latest known version of a dataset or None if empty.

get_meta() → List[Dict[Any, Any]][source]#

Being on disk adds updated_at field to the Meta

Return type:

Meta

get_version(ds: Dataset) → Version[source]#

Given a dataset, returns its version. If the dataset was seen previously, will retrieve the version and if not, will assign appropriate latest version.

Does not record the info about dataset - use save() for that purposes.

load(num: Union[int, str]) → Dataset[source]#

Loads a dataset by using its number or version string

save(ds: Dataset, only_meta: bool = False) → None[source]#

Saves a dataset into a folder corresponding with its version. Version is determined by meta from get_meta method and consists of two parts. Major and minor like 0.1 or 2.12 etc. If the structure of the pipeline changed e.g. some new step added, then major version updates. An when the structure is the same, but meta changed in some way, then minor version is updated.

Abstract Line

abstract __getitem__(num: int) → Any[source]#

Should return the object using numerical index

abstract __len__() → int[source]#

Should return the number of object inside

abstract get_root() → str[source]#

Should return root folder of Line

abstract load(num: int) → None[source]#

abstract load_obj_meta(pathspec: str) → List[Dict[Any, Any]][source]#

Should read and return object meta using its path specification

abstract reload()[source]#

abstract save(obj: Any, only_meta: bool = False) → None[source]#

A manager for a line of models. Used by Repo to access models on disk. A line of models is typically models with the same hyperparameters and architecture, but different epochs or trained using different data.

__init__(root: str, model_cls: ~typing.Type[~typing.Any] = <class 'cascade.models.model.Model'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', *args: ~typing.Any, **kwargs: ~typing.Any) → None[source]#

All models in line should be instances of the same class.

create_model(*args: Any, **kwargs: Any) → Model[source]#

Creates a model using the class given on line’s creation, registers default log callback for it and returns. Passes all args to the object constructor.

Returns:

Created and prepared model

Return type:

Any

get_meta() → List[Dict[Any, Any]][source]#

Being on disk adds updated_at field to the Meta

Return type:

Meta

get_model_names() → List[str][source]#

Get the list of model names, which are folder names relative to line’s root

Returns:

The list of names

Return type:

List[str]

load(num: int) → Model[source]#

Loads a model

Parameters:

num (int) – Model number in line

load_artifact_paths(model: Union[int, str]) → Dict[str, List[str]][source]#

Returns full paths to the files and artifacts of the model

Parameters:

model (Union[int, str]) – Model slug or number

Returns:

Lists of files under the keys “artifacts” and “files”

Return type:

Dict[str, List[str]]

load_model_meta(path_spec: Union[str, int]) → List[Dict[Any, Any]][source]#

Given a model num or a slug, loads its metadata from disk. Alias for load_obj_meta

Parameters:

path_spec (Union[str, int]) – Can be an int number or a str slug

Returns:

Model’s meta

Return type:

Meta

Raises:

FileNotFoundError – When the num or the slug was not found in the line

save(model: Model, only_meta: bool = False) → None[source]#

Saves a model and its metadata into a model’s folder

Model is automatically assigned a number and a slug then it is saved using its own method save.

Folder names are assigned using f’{idx:0>5d}’. For example: 00001 or 00042.

It is Model’s responsibility to save its own state given a folder.

Also saves ModelLine’s meta to the Line’s root.

Parameters:

• model (Model) – Model to be saved

• only_meta (bool, optional) – Flag, that indicates whether to save model’s artifacts. If True saves only metadata