cascade.lines#

class cascade.lines.DataLine(root: str, ds_cls: ~typing.Type[~typing.Any] = <class 'cascade.data.dataset.Dataset'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', obj_backend: ~typing.Literal['pickle'] = 'pickle', *args: ~typing.Any, **kwargs: ~typing.Any)[source]#

__init__(root: str, ds_cls: ~typing.Type[~typing.Any] = <class 'cascade.data.dataset.Dataset'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', obj_backend: ~typing.Literal['pickle'] = 'pickle', *args: ~typing.Any, **kwargs: ~typing.Any) → None[source]#

Parameters:

description – String description of an object
tags (Iterable[str], optional) – The list of tags to be added

See also

cascade.base.MetaHandler

get_latest_version() → Version | None[source]#: Returns latest known version of a dataset or None if empty.

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where first element is this object’s metadata. All other elements represent the other stages of pipeline if present.

Meta can be anything that is worth to document about the object and its properties.

Meta is a list (see Meta type alias) to allow the formation of pipelines.

Return type:

Meta

get_version(ds: Dataset) → Version[source]#

Given a dataset, returns its version. If the dataset was seen previously, will retrieve the version and if not, will assign appropriate latest version.

Does not record the info about dataset - use save() for that purposes.

load(num: int | str) → Dataset[source]#: Loads a dataset by using its number or version string

save(ds: Dataset, only_meta: bool = False) → None[source]#: Saves a dataset into a folder corresponding with its version. Version is determined by meta from get_meta method and consists of two parts. Major and minor like 0.1 or 2.12 etc. If the structure of the pipeline changed e.g. some new step added, then major version updates. An when the structure is the same, but meta changed in some way, then minor version is updated.

class cascade.lines.disk_line.DiskLine(root: str, item_cls: Type[Any], meta_fmt: Literal['.json', '.yml', '.yaml', None], *args: Any, **kwargs: Any)[source]#

__getitem__(num: int) → Any[source]#: Loads the item using load method of a given class

__init__(root: str, item_cls: Type[Any], meta_fmt: Literal['.json', '.yml', '.yaml', None], *args: Any, **kwargs: Any) → None[source]#

Parameters:

description – String description of an object
tags (Iterable[str], optional) – The list of tags to be added

See also

cascade.base.MetaHandler

__len__() → int[source]#

Return type:: A number of items in line

get_item_names() → List[str][source]#

Returns names of folders items live in

Returns:: Only names of folders without whole path
Return type:: List[str]

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where first element is this object’s metadata. All other elements represent the other stages of pipeline if present.

Meta can be anything that is worth to document about the object and its properties.

Meta is a list (see Meta type alias) to allow the formation of pipelines.

Return type:

Meta

load(num: int) → Any[source]#

load_obj_meta(path_spec: int) → List[Dict[Any, Any]][source]#

Loads metadata of a item from disk

Parameters:

path_spec (int) – item number

Returns:

Model metadata

Return type:

MetaFromFile

Raises:

FileNotFoundError – Raises if failed to find the item with spec specified
RuntimeError – If found more than one metadata files in the specified item folder

reload() → None[source]#

class cascade.lines.Line(*args: Any, description: str | None = None, tags: Iterable[str] | None = None, **kwargs: Any)[source]#

abstract get_root() → str[source]#

abstract load(num: int) → None[source]#

abstract load_obj_meta(pathspec: str) → List[Dict[Any, Any]][source]#

abstract reload()[source]#

abstract save(obj: Any, only_meta: bool = False) → None[source]#

class cascade.lines.ModelLine(root: str, model_cls: ~typing.Type[~typing.Any] = <class 'cascade.models.model.Model'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', *args: ~typing.Any, **kwargs: ~typing.Any)[source]#

A manager for a line of models. Used by Repo to access models on disk. A line of models is typically models with the same hyperparameters and architecture, but different epochs or trained using different data.

__init__(root: str, model_cls: ~typing.Type[~typing.Any] = <class 'cascade.models.model.Model'>, meta_fmt: ~typing.Literal['.json', '.yml', '.yaml'] = '.json', *args: ~typing.Any, **kwargs: ~typing.Any) → None[source]#: All models in line should be instances of the same class.

create_model(*args: Any, **kwargs: Any) → Model[source]#

Creates a model using the class given on line’s creation, registers default log callback for it and returns. Passes all args to the object constructor.

Returns:: Created and prepared model
Return type:: Any

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where first element is this object’s metadata. All other elements represent the other stages of pipeline if present.

Meta can be anything that is worth to document about the object and its properties.

Meta is a list (see Meta type alias) to allow the formation of pipelines.

Return type:

Meta

get_model_names() → List[str][source]#

Get the list of model names, which are folder names relative to line’s root

Returns:: The list of names
Return type:: List[str]

load(num: int, only_meta: bool | None = None) → Model[source]#

Loads a model

Parameters:

num (int) – Model number in line
only_meta (bool, optional) – If True doesn’t load model’s artifacts, by default False

load_artifact_paths(model: int | str) → Dict[str, List[str]][source]#

Returns full paths to the files and artifacts of the model

Parameters:: model (Union[int, str]) – Model slug or number
Returns:: Lists of files under the keys “artifacts” and “files”
Return type:: Dict[str, List[str]]

load_model_meta(path_spec: str | int) → List[Dict[Any, Any]][source]#

Given a model num or a slug, loads its metadata from disk. Alias for load_obj_meta

Parameters:: path_spec (Union[str, int]) – Can be an int number or a str slug
Returns:: Model’s meta
Return type:: MetaFromFile
Raises:: FileNotFoundError – When the num or the slug was not found in the line

save(model: Model, only_meta: bool = False) → None[source]#

Saves a model and its metadata into a model’s folder

Model is automatically assigned a number and a slug then it is saved using its own method save.

Folder names are assigned using f’{idx:0>5d}’. For example: 00001 or 00042.

It is Model’s responsibility to save its own state given a folder.

Also saves ModelLine’s meta to the Line’s root.

Parameters:

model (Model) – Model to be saved
only_meta (bool, optional) – Flag, that indicates whether to save model’s artifacts. If True saves only metadata