cascade.data#

Modifier that applies a function to given dataset’s items in each __getitem__ call.

Can be applied to Iterators too.

__init__(dataset: Union[Dataset[T], IteratorDataset[T]], func: Callable[[T], Any], p: Optional[float] = None, seed: Optional[int] = None, *args: Any, **kwargs: Any) → None[source]#

Parameters:

• dataset (Dataset) – A dataset to modify

• func (Callable) – A function to be applied to every item of a dataset - each __getitem__ calls func on an item obtained from a previous dataset

• p (Optional[float], by default None) – The probability [0, 1] with which to apply func

• seed (Optional[int], by default None) – Random seed is used when p is not None

Examples

>>> from cascade import data as cdd
>>> ds = cdd.Wrapper([0, 1, 2, 3, 4])
>>> ds = cdd.ApplyModifier(ds, lambda x: x ** 2)

Now function will only be applied when items are retrieved

>>> assert [item for item in ds] == [0, 1, 4, 9, 16]

get(index: int) → Any[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

The container for the info on the people who were in charge of labeling process, their experience and other properties.

This is a dataclass, so any additional fields will not be recorded if added. If it needs to be extended, please create a new class instead.

__init__(id: Optional[str] = None, position: Optional[str] = None) → None#

id: Optional[str] = None#

position: Optional[str] = None#

Base class of any object that constitutes a step in a data-pipeline

__init__(*args: Any, data_card: Optional[DataCard] = None, **kwargs: Any) → None[source]#

property data_card#

declare_volatiles(*fields)[source]#

Volatiles are metadata fields that should not be used for this dataset versioning

Example

from datetime import datetime
from cascade.data import Dataset
from cascade.lines import DataLine

class ScheduledDataset(Dataset):
    def __init__(self):
        self.time_of_arrival = datetime.now()
        self.declare_volatiles("time_of_arrival")

    def get(self):
        return None

    def __len__(self):
        return 1

    def get_meta(self):
        meta = super().get_meta()
        meta[0]["time_of_arrival"] = self.time_of_arrival
        return meta

ds = ScheduledDataset()
meta = ds.get_meta()

assert "time_of_arrival" in meta[0]
assert meta[0]["cascade_volatiles"] == ["time_of_arrival"]

line = DataLine("volatile_check")
line.save(ds, only_meta=True)

initial_version = line.get_version(ds)
initial_time = meta[0]["time_of_arrival"]

ds = ScheduledDataset()
meta = ds.get_meta()

line.save(ds, only_meta=True)

recreated_version = line.get_version(ds)
recreated_time = meta[0]["time_of_arrival"]

assert initial_version == "0.1"
assert initial_time != recreated_time
assert initial_version == recreated_version

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where last element is this dataset’s metadata. Meta can be anything that is worth to document about the dataset and its data. This is done in form of list to enable cascade-like calls in Modifiers and Samplers.

Return type:

Meta

property volatiles#

Base class for Modifiers, mostly unifies metadata management

__init__(dataset: BaseDataset[T], *args: Any, **kwargs: Any) → None[source]#

Constructs a Modifier. Modifier represents a step in a pipeline - some data transformation

Parameters:

dataset (BaseDataset[T]) – A dataset to modify

from_meta(meta: List[Dict[Any, Any]]) → None[source]#

Calls the same method as base class but does it cascade-like which allows to roll a list of meta on a pipeline

Parameters:

meta (Meta) – Meta of a single object or a pipeline

get_meta() → List[Dict[Any, Any]][source]#

Overrides base method enabling cascade-like calls to previous datasets. The metadata of a pipeline that consist of several modifiers can be easily obtained with get_meta of the last block.

update_meta(meta)[source]#

Updates self.meta_prefix, which then updates dataset’s meta when get_meta() is called

Parameters:

meta (Union[Meta, MetaBlock, Config]) – The object to update with

Raises:

ValueError – If the list passed and it is not of the unit length

Special modifier that calls all previous pipeline in __init__ loading everything in memory.

Examples

>>> from cascade import data as cdd
>>> ds = cdd.Wrapper([0 for _ in range(1000000)])
>>> ds = cdd.ApplyModifier(ds, lambda x: x + 1)
>>> ds = cdd.ApplyModifier(ds, lambda x: x + 1)
>>> ds = cdd.ApplyModifier(ds, lambda x: x + 1)

Cache heavy upstream part

>>> ds = cdd.BruteforceCacher(ds)

__init__(dataset: BaseDataset[T], *args: Any, **kwargs: Any) → None[source]#

Loads every item in dataset in internal list.

get(index: int) → T[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

Unifies two or more datasets element-wise.

Example

>>> from cascade import data as cdd
>>> items = cdd.Wrapper([0, 1, 2, 3, 4])
>>> labels = cdd.Wrapper([1, 0, 0, 1, 1])
>>> ds = cdd.Composer((items, labels))
>>> assert ds[0] == (0, 1)

__init__(datasets: List[Dataset[Any]], *args: Any, **kwargs: Any) → None[source]#

Parameters:

datasets (Iterable[Dataset]) – Datasets of the same length to be unified

from_meta(meta: List[Dict[Any, Any]]) → None[source]#

Updates its own fields as usual and if meta has data key then sequentially updates data of all its datasets

Parameters:

meta (Meta) – Meta of a single object or a pipeline

get(index: int) → Tuple[T][source]#

get_meta() → List[Dict[Any, Any]][source]#

Composer calls get_meta() of all its datasets

Unifies several Datasets under one, calling them sequentially in the provided order.

Examples

>>> from cascade.data import Wrapper, Concatenator
>>> ds_1 = Wrapper([0, 1, 2])
>>> ds_2 = Wrapper([2, 1, 0])
>>> ds = Concatenator((ds_1, ds_2))
>>> assert [item for item in ds] == [0, 1, 2, 2, 1, 0]

__init__(datasets: List[Dataset[T]], *args: Any, **kwargs: Any) → None[source]#

Creates concatenated dataset from the list of datasets provided

Parameters:

datasets (Union[Iterable[Dataset], Mapping[Dataset]]) – A list or tuple of datasets to concatenate

__len__() → int[source]#

Length of Concatenator is a sum of lengths of its datasets

from_meta(meta: List[Dict[Any, Any]]) → None[source]#

Updates its own fields as usual and if meta has data key then sequentially updates data of all its datasets

Parameters:

meta (Meta) – Meta of a single object or a pipeline

get(index: int) → T[source]#

get_meta() → List[Dict[Any, Any]][source]#

Concatenator calls get_meta() of all its datasets

A Sampler that iterates num_samples times through an input Dataset in cyclic manner

Example

>>> from cascade.data import CyclicSampler, Wrapper
>>> ds = Wrapper([1,2,3])
>>> ds = CyclicSampler(ds, 7)
>>> assert [item for item in ds] == [1, 2, 3, 1, 2, 3, 1]

get(index: int) → T[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

The container for the information on dataset. The set of fields here is general and can be extended by providing new keywords into __init__.

Example

>>> from cascade.data import DataCard, Assessor, LabelingInfo
>>> person = Assessor(id=0, position="Assessor")
>>> info = LabelingInfo(who=[person], process_desc="Labeling description")
>>> dc = DataCard(
...     name="Dataset",
...     desc="Example dataset",
...     source="Database",
...     goal="Every dataset should have a goal",
...     labeling_info=info,
...     size=100,
...     metrics={"quality": 100},
...     schema={"label": "value"},
...     custom_field="hello")

__init__(name: Optional[str] = None, desc: Optional[str] = None, source: Optional[str] = None, goal: Optional[str] = None, labeling_info: Optional[LabelingInfo] = None, size: Optional[Union[int, Tuple[int]]] = None, metrics: Optional[Dict[str, Any]] = None, schema: Optional[Dict[Any, Any]] = None, **kwargs: Any) → None[source]#

Parameters:

• name (Optional[str]) – The name of dataset

• desc (Optional[str]) – Short description

• source (Optional[str]) – The source of data. Can be URL or textual description of source

• goal (Optional[str]) – The datasets have a goal - what should be achieved using this data?

• labeling_info (Optional[LabelingInfo]) – The instance of dataclass describing labeling process placed here

• size (Union[int, Tuple[int], None]) – This can usually be done automatically - number of items or shape of the table.

• metrics (Optional[Dict[str, Any]]) – Dictionary with names and values of metrics. Any quality metrics can be included

• schema (Optional[Dict[Any, Any]]) – Schema dictionary describing table datasets, their columns, data types, possible values, etc.

to_dict() → Dict[str, Any][source]#

An abstract class to represent a dataset with __len__ method present. Inheritance of this class should mean the presence of length.

If your dataset does not have length defined you can use IteratorModifier

abstract get(index: Any) → T[source]#

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where last element is this dataset’s metadata. Meta can be anything that is worth to document about the dataset and its data. This is done in form of list to enable cascade-like calls in Modifiers and Samplers.

Return type:

Meta

Filter for Datasets with length. Uses a function to create a mask of items that will be stored once and applied for each access.

Example

Here we select only even numbers from a dataset

>>> from cascade.data import Filter, Wrapper

>>> ds = Wrapper([0, 1, 2, 3])
>>> ds = Filter(ds, lambda x: x % 2 == 0)
>>> list(ds)
[0, 2]

__init__(dataset: Dataset, filter_fn: Callable, *args: Any, **kwargs: Any) → None[source]#

Filter a dataset using a filter function. Does not accumulate items in memory, will store only an index mask.

Parameters:

• dataset (Dataset) – A dataset to filter

• filter_fn (Callable) – A function to be applied to every item of a dataset - should return bool. Will be called on every item on __init__.

Raises:

RuntimeError – If filter_fn raises an exception

get(index: Any)[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

Basic “folder of files” dataset. Accepts root folder in which considers all files. Is abstract - getitem is not defined, since it is specific for each file type.

__init__(root: str, *args: Any, **kwargs: Any) → None[source]#

Parameters:

root (str) – A path to the folder of files

get(index: Any) → T[source]#

get_meta() → List[Dict[Any, Any]][source]#

Returns meta containing root folder

get_names() → List[str][source]#

Returns a list of full paths to the files

An abstract class to represent a dataset as an iterable object

Filter for datasets without length

Does not filter on init, returns only items that pass the filter

Example

Here we select only even numbers from a dataset

>>> from cascade.data import IteratorFilter, IteratorWrapper

>>> ds = IteratorWrapper([0, 1, 2, 3])
>>> ds = IteratorFilter(ds, lambda x: x % 2 == 0)
>>> list(ds)
[0, 2]

__init__(dataset: IteratorDataset, filter_fn: Callable, *args: Any, **kwargs: Any) → None[source]#

Constructs a Modifier. Modifier represents a step in a pipeline - some data transformation

Parameters:

dataset (BaseDataset[T]) – A dataset to modify

The Modifier for Iterator datasets

__init__(dataset: IteratorDataset[T], *args: Any, **kwargs: Any) → None[source]#

Constructs a Modifier. Modifier represents a step in a pipeline - some data transformation

Parameters:

dataset (BaseDataset[T]) – A dataset to modify

get_meta() → List[Dict[Any, Any]][source]#

Overrides base method enabling cascade-like calls to previous datasets. The metadata of a pipeline that consist of several modifiers can be easily obtained with get_meta of the last block.

Wraps IteratorDataset around any Iterable. Does not have map-like interface.

__init__(data: Iterable[T], *args: Any, **kwargs: Any) → None[source]#

get_meta() → List[Dict[Any, Any]][source]#

Returns:

meta – A list where last element is this dataset’s metadata. Meta can be anything that is worth to document about the dataset and its data. This is done in form of list to enable cascade-like calls in Modifiers and Samplers.

Return type:

Meta

The container for the information on labeling process, people involved, description of the process, documentation links.

This is a dataclass, so any additional fields will not be recorded if added. If it needs to be extended, please create a new class instead.

__init__(who: Optional[List[Assessor]] = None, process_desc: Optional[str] = None, docs: Optional[str] = None) → None#

docs: Optional[str] = None#

process_desc: Optional[str] = None#

who: Optional[List[Assessor]] = None#

Basic pipeline building block in Cascade. Every block which is not a data source should be a successor of Sampler or Modifier.

This structure enables having a data pipeline which consists of uniform blocks each of them has a reference to the previous one in its _dataset field

Basically Modifier defines an arbitrary transformation on every dataset’s item that is applied in a lazy manner on each __getitem__ call.

Applies no transformation if __getitem__ is not overridden

Does not change the length of a dataset. See Sampler for this functionality

Example

from cascade.data import Modifier, Wrapper

class PowModifier(Modifier):
    def __init__(self, ds, p, *args, **kwargs):
        super().__init__(ds, *args, **kwargs)

        self.p = p

    def get(self, index):
        number = self._dataset[index]
        return number ** self.p

ds = Wrapper([0, 1, 2, 3, 4])
ds = PowModifier(ds, 2)

assert list(ds) == [0, 1, 4, 9, 16]

get(index: Any) → T[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

get_meta() → List[Dict[Any, Any]][source]#

Overrides base method enabling cascade-like calls to previous datasets. The metadata of a pipeline that consist of several modifiers can be easily obtained with get_meta of the last block.

Shuffles a dataset

__init__(dataset: Dataset[T], num_samples: Optional[int] = None, *args: Any, **kwargs: Any) → None[source]#

Parameters:

• dataset (Dataset[T]) – Input dataset to sample from

• num_samples (int, optional) – If less or equal than len(dataset) samples without repetitions (shuffles indices) If more than len(dataset) generates random integers as indices If None, then just shuffles the dataset

get(index: int) → T[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

Implements Python range as a Dataset

Example

>>> from cascade.data import RangeSampler, Wrapper
>>> ds = Wrapper([1, 2, 3, 4, 5])
>>> # Define start, stop and step exactly as in range()
>>> sampler = RangeSampler(ds, 1, 5, 2)
>>> for item in sampler:
...     print(item)
...
2
4
>>> ds = Wrapper([1, 2, 3, 4, 5])
>>> sampler = RangeSampler(ds, 3)
>>> for item in sampler:
...     print(item)
...
1
2
3

__init__(dataset: Dataset[T], start: Optional[int] = None, stop: Optional[int] = None, step: int = 1, *args: Any, **kwargs: Any) → None[source]#

Parameters:

• dataset (SizedDataset) – A dataset to sample from

• start (int) – Start index in range - included

• stop (int) – Stop index in range - excluded

• step (int, default is 1) – Step of range

Raises:

• ValueError – when no start or stop present or:

• when parameters given produce empty dataset –

get(index: int) → T[source]#

Returns previous Dataset’s item without changes if not overridden

Parameters:

index (Any) – Item’s index

Returns:

Item

Return type:

T

Defines certain sampling over a Dataset.

Its distinctive feature is that it changes the number of items in dataset.

Can be used to build a batch sampler, random sampler, etc.

__init__(dataset: Dataset[T], num_samples: int, *args: Any, **kwargs: Any) → None[source]#

Constructs a Sampler.

Parameters:

• dataset (Dataset) – A dataset to sample from

• num_samples (int) – The number of samples to use as a new length

Data validation modifier

When self._dataset is called and has self.in_schema defined, wraps self._dataset into validator, which is another Modifier that checks the output of __getitem__ of the dataset that was wrapped.

In the end it will look like this:

If in_schema is not None:

dataset = SchemaModifier(ValidationWrapper(dataset))

If in_schema is None:

dataset = SchemaModifier(dataset)

How to use it: 1. Define pydantic schema of input

from typing import List, Tuple
import pydantic

class AnnotImage(pydantic.BaseModel):
    image: List[List[List[float]]]
    segments: List[List[int]]
    bboxes: List[Tuple[int, int, int, int]]

2. Use schema as in_schema

from cascade.data import SchemaModifier

class ImageModifier(SchemaModifier):
    in_schema = AnnotImage

3. Create a regular Modifier by subclassing ImageModifier.

class IDoNothing(ImageModifier):
    def get(self, idx):
        item = self._dataset[idx]
        return item

4. That’s all. Schema check will be held automatically every time self._dataset[idx] is accessed. If it is not like AnnotImage, cascade.data.ValidationError will be raised.

__init__(dataset: Dataset, *args: Any, **kwargs: Any) → None[source]#

Constructs a Modifier. Modifier represents a step in a pipeline - some data transformation

Parameters:

dataset (BaseDataset[T]) – A dataset to modify

get_meta() → List[Dict[Any, Any]][source]#

Since SchemaModifier will add ValidationWrapper dynamically to _dataset it will be used in meta. This call will clean meta and remove wrappers

Return type:

Meta

in_schema: Optional[Any] = None#

Simple batch builder - given a sequence and a size of batch breaks it in the subsequences

>>> from cascade.data import SimpleDataloader
>>> dl = SimpleDataloader([0, 1, 2], 2)
>>> [item for item in dl]
[[0, 1], [2]]

__init__(data: Sequence[T], batch_size: int = 1) → None[source]#

Wraps Dataset around any list-like object

__init__(obj: Sequence[T], *args: Any, **kwargs: Any) → None[source]#

get(index: Any) → T[source]#

Return an item from wrapped container

Parameters:

index (Any) –

Returns:

Item from a container

Return type:

T

Example

>>> from cascade.data import Wrapper
>>> ds = Wrapper([1, 2, 3])
>>> print(ds.get(0))
1

get_meta() → List[Dict[Any, Any]][source]#

Wrapper’s meta adds obj_type field to the default meta of a Modifier

Return type:

Meta

Example

>>> from cascade.data import Wrapper
>>> ds = Wrapper([1, 2, 3])
>>> print(ds.get_meta())
[{'name': 'cascade.data.dataset.Wrapper', ..., 'len': 3, 'obj_type': "<class 'list'>"}]