Tutorial#

Installation#

Install the latest version using pip

pip install cascade-ml

Cascade has a set of optional dependencies, which can be installed with the following commands

pip install cascade-ml[opencv]   # Use opencv as image backend
pip install cascade-ml[pil]      # Use Pillow as image backend
pip install cascade-ml[pydantic] # Use data validation modifiers
pip install cascade-ml[sklearn]  # Scikit-learn integration
pip install cascade-ml[torch]    # PyTorch integration
pip install cascade-ml[view]     # Cascade viewers based on dash
pip install cascade-ml[all]      # Installs everything

If you have completed the tutorial you can see the How-to guides section for more specific and complex workflows.

1. Pipelines basics#

In this step you will learn basic pipeline building blocks of Cascade. This is the first Cascade tutorial from a series. All of the tutorials are meant to form a single project and can be thought of as a series of lessons.

Some lessons come with links for further reading or advanced how-to guides on related topics.

Cascade pipelines allow building data processing routines from steps called Datasets and Modifiers.

Datasets are the sources of data. In the first step let’s make a Dataset for digits from sklearn.

from cascade.data import Dataset


class DigitsDataset(Dataset):
    def __init__(self) -> None:
        self.x, self.y = load_digits(return_X_y=True)
        super().__init__()

    def get(self, index):
        item = {
            "x": self.x[index],
            "y": self.y[index],
        }
        return item

    def __len__(self):
        return len(self.x)

Minimal setup for a Dataset is a get and __len__ methods. Now we can do basic access.

ds = DigitsDataset()
print(ds[0])

Cascade Datasets are not only a system of data organization. They allow using rich set of defaults for data manipulation.

import numpy as np

from cascade.data import ApplyModifier

NOISE_RANGE = (-1, 1)


def add_noise(item):
    item["x"] = item["x"] + np.random.randint(*NOISE_RANGE)
    return item


ds_noise = ApplyModifier(ds, add_noise)

print(ds_noise[0])

Modifiers take datasets and transform their values. In previous example we added noise to digits by using ApplyModifier and created a new noisy dataset.

We can augment our data by concatenating those two datasets.

from cascade.data import Concatenator

ds = Concatenator([ds, ds_noise])
print(len(ds))

2. Metadata#

Cascade helps to generate a lot of useful metadata. This is why we need wrappers - they allow capturing info about underlying objects automatically.

To see what it looks like, you can call get_meta method on a Cascade object. In the next step we will try calling it on a pipeline that was made on the Pipelines step.

from pprint import pprint
pprint(ds.get_meta())

[{'comments': [],
'data': [[{'comments': [],
            'description': None,
            'len': 1797,
            'links': [],
            'name': '__main__.DigitsDataset',
            'tags': [],
            'type': 'dataset'}],
        [{'comments': [],
            'description': None,
            'len': 1797,
            'links': [],
            'name': 'cascade.data.apply_modifier.ApplyModifier',
            'tags': [],
            'type': 'dataset'},
            {'comments': [],
            'description': None,
            'len': 1797,
            'links': [],
            'name': '__main__.DigitsDataset',
            'tags': [],
            'type': 'dataset'}]],
'description': None,
'len': 3594,
'links': [],
'name': 'cascade.data.concatenator.Concatenator',
'num_concatenated': 2,
'tags': [],
'type': 'dataset'}]

You can see all the stages of the pipeline in this metadata. It is a list of dicts with JSON-serializable fields, each block in this list represents a pipeline step.

Datasets, Models and some other objects have metadata. It is a very flexible tool, that can be easily customized to record valuable info about training process. We will see the power of it in the following tutorials.

3. Experiments basics#

Cascade provides a rich set of ML-experiment tracking tools. You can easily track history of model changes, save and restore models in a structured manner along with metadata.

In this step we create a wrapper around logistic regression model. Minimal setup for the model is not strictly defined as in Dataset case. We define fit and predict. BasicModel will handle everything else for us - like saving and loading for example.

from cascade.models import BasicModel


class LR(BasicModel):
    def __init__(self, penalty):
        self.model = LogisticRegression(penalty=penalty)
        super().__init__()

    def fit(self, dataset):
        x, y = [], []
        for item in dataset:
            x.append(item["x"])
            y.append(item["y"])

        self.model.fit(x, y)

    def predict(self, x):
        return self.model.predict(x)

We can create and fit the model now using the dataset from the Pipelines step.

model = LR("l2")
model.fit(ds)

To track important hyperparameters and how they influence metrics, Cascade Models feature special field called params. This is an empty dict that you can fill with any (serializable) data. Cascade custom JSON serializer can also serialize some non-default things like numpy arrays.

Here we fill our param externally, but could also do it above inside the class.

model.params["penalty"] = "l2"

Model lines are basic structured storage units in Cascade. They represent a lineage of a model. Usually they represent a single training run, but can be used arbitrarily.

In this step we create a line and save our new model.

from cascade.lines import ModelLine

line = ModelLine("line", model_cls=LR)
line.save(model)

Lines handle storage of models and their metadata and can retrieve saved models by index or a name.

In the next step we load the model and infer it on a dataset.

The line knows little about models it manages - we provided a class of our model to be able to restore it correctly when loading.

model = line.load(0)
x = [item["x"] for item in ds]
preds = model.predict(x)

print(preds[0], ds[0]["y"])

Lines also enhance model’s meta by recording useful environment information. Let’s see what was saved automatically about this experiment. We load model meta with a default line method.

from pprint import pprint
pprint(line.load_model_meta(0))

[{'comments': [],
'created_at': '2024-07-14T21:08:58.466812+00:00',
'cwd': '/home/ilia/local/cascade_proj/cascade/cascade/docs/source/tutorials',
'description': None,
'git_commit': '62de43afb7dbf51afe2d08dd0825366661c76055',
'git_uncommitted_changes': ['M '
                            'cascade/docs/source/tutorials/experiment_basics.rst',
                            'M cascade/docs/source/tutorials/tutorials.py',
                            'M cascade/docs/source/tutorials/tutorials.rst\n'
                            '?? cascade/docs/source/tutorials/line/'],
'host': 'my-pc-name',
'links': [],
'metrics': [],
'name': '__main__.LR',
'params': {'penalty': 'l2'},
'path': '/home/ilia/local/cascade_proj/cascade_repo/cascade/docs/source/tutorials/line/00000',
'python_version': '3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]',
'saved_at': '2024-07-14T21:09:01.453262+00:00',
'slug': 'imperial_magenta_cheetah',
'tags': [],
'type': 'model',
'user': 'ilia'}]

4. Custom Meta and Versioning#

Metadata is a very flexible tool. It contains lots of useful info by default, and can be customized.

In previous steps of the tutorial we created a dataset with an important parameter, that was not recorded in our meta. If it changes in code, we wouldn’t see the effect in our logs. Now we can fix that issue.

from cascade.lines import DataLine

ds.update_meta(
    {
        "long_description": "This is digits pipeline. It was augmented with some uniform noise",
        "noise_range": NOISE_RANGE,
    }
)

DataLines are the same thing as ModelLine but for data pipelines. You can use them to track only metadata of your pipelines or even save and load whole pipelines to reproduce an experiment.

dataline = DataLine("dataline")
dataline.save(ds)

Unlike models, data pipelines are not numbered, but versioned. Versions are derived from metadata of a pipeline and consist of two parts - major and minor.

Let’s see how it works.

version = dataline.get_version(ds)
print(version) # 0.1

The starting version is 0.1 and then, when metadata changes, parts of the version are bumped automatically. When saving the version of a dataset that already exists, line will notice that and overwrite older record with a new object.

ds.update_meta({"detail_i_almost_forgot": "Changes in meta bump minor version"})
version = dataline.get_version(ds)
print(version) # 0.2

dataline.save(ds)

In previous example minor version was bumped by changing the part of the pipeline’s meta.

In the next one we add a new pipeline stage, which is what will bump a major part of the version and we will see 1.0.

changed_ds = ApplyModifier(ds, add_noise)
dataline.save(changed_ds)
version = dataline.get_version(changed_ds)
print(version) # 1.0

If we plug in an old dataset it will still get us the same version. As long as meta is the same. Using version string we can load saved pipeline object from disk.

version = dataline.get_version(ds)
print(version) # 0.2

loaded_ds = dataline.load("0.2")
version = dataline.get_version(loaded_ds)
print(version) # 0.2

5. Metrics and Evaluation#

Metrics are first-class citizens in Cascade. For every ML-project they should be a central aspect.

Metric API is very flexible - you have freedom to define metrics in different ways. First case is the regular way metrics are usually defined in projects - as functions.

Metric function can be passed in the default evaluate method of BasicModel. Evaluation of the model will return nothing, but instead fill its metrics field with a list of metrics.

from sklearn.metrics import f1_score


def f1(gt, pred):
    return f1_score(gt, pred, average="macro")


x = [item["x"] for item in loaded_ds]
y = [item["y"] for item in loaded_ds]

model.evaluate(x, y, [f1])

pprint(model.metrics)

[Metric(name=f1, value=1.0, created_at="2024-07-29 19:41:09.344039+00:00")]

Let’s try defining a metric in another, more flexible way. We need to implement a descendant of cascade.metrics.Metric class. The one required method is compute that should return value and also set self.value.

After that evaluate can be called with a list of Metric objects.

from cascade.metrics import Metric


class Accuracy(Metric):
    def __init__(self):
        super().__init__(name="acc")

    def compute(self, gt, pred):
        self.value = sum([g == p for g, p in zip(gt, pred)]) / len(gt)
        return self.value


model.evaluate(x, y, [Accuracy()])

pprint(model.metrics)

[Metric(name=f1, value=1.0, created_at=2024-07-29 19:47:33.435828+00:00),
 Accuracy(name=acc, value=1.0, created_at=2024-07-29 19:47:33.437724+00:00)]

Metrics are saved and written in metadata automatically after calling evaluate.

line.save(model)
pprint(line.load_model_meta(1))

[{'comments': [],
'created_at': '2024-07-28T14:47:30.451860+00:00',
'description': None,
'host': 'my-pc-name',
'links': [],
'metrics': [{'created_at': '2024-07-28T14:47:32.860739+00:00',
            'name': 'f1',
            'value': 1.0},
            {'created_at': '2024-07-28T14:47:32.862089+00:00',
            'name': 'acc',
            'value': 1.0}],
'name': '__main__.LR',
'params': {'penalty': 'l2'},
'path': '/home/ilia/local/cascade_proj/line/00001',
'python_version': '3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]',
'saved_at': '2024-07-28T14:47:32.902304+00:00',
'slug': 'pompous_lori_from_lemuria',
'tags': [],
'type': 'model',
'user': 'ilia'}]

6. Meta defaults#

Cascade objects feature methods for managing some useful meta default fields.

For example descriptions - they can be useful if you want to convey basic information about the model not only in code, but in saved metadata of this model.

model.describe("This is simple linear model")

Tags can be used to identify certain models, or filter them.

model.tag(["tutorial", "dummy"])

Links allow connecting a model to any relevant external media. You can link a file using its URI, or a Cascade object like training data or some other related model.

model.link(ds)
model.link(name="training_file", uri=__file__)

There are also methods that allow removing certain fields. This part is mostly self-explanatory.

model.remove_tag("dummy")

model.remove_link("1")

Here we removed tag using its name and the first link using its ID. It is the dataset link, just for example.

pprint(model.get_meta())
line.save(model)

7. CLI#

Cascade features simple command line interface to manage storage of your models, metadata and experiments. You can comment on models, edit tags, descriptions and manage artifacts from the command line.

This tutorial is connected with Meta Defaults step since CLI allows editing meta defaults without writing special scripts for it.

Go to the directory of previously created line and execute the following.

cascade status

This is basic utility now just serves as a check that everything is okay with you installation and directory. Cascade will look for meta.json file in the folder you are running a command and if found, output short description of what is in this folder.

If everything is ok, previous command should output the following. If not, do not hesitate filling a GitHub issue.

This is model_line of len 25

To print the contents of objects metadata you can visit a folder of an object and run.

cascade cat

This will give you a nice prettyprint of meta.json that will look something like this.

[{'cascade_version': '0.14.0-alpha',
'comments': [],
'created_at': '2024-07-28 14:47:31.825546+00:00',
'description': None,
'item_cls': "<class '__main__.LR'>",
'len': 25,
'links': [],
'name': "<class 'cascade.lines.model_line.ModelLine'>(3) items of <class "
        "'cascade.models.basic_model.BasicModel'>",
'root': '/home/ilia/local/cascade_proj/line',
'tags': [],
'type': 'model_line',
'updated_at': '2024-07-31 20:03:03.111970+00:00'}]

For different objects commands are similar. For example to list tags of the current objects you run.

cascade tag ls

Since no tags in this line yet, it wil show an empty list.

[]

Let’s add two tags with one command and check.

cascade tag add one two
cascade tag ls

['one', 'two']

Now we remove one tag and check again.

cascade tag rm one
cascade tag ls

['two']

Comments are whole separate thing to consider in Cascade. They proved to be very useful for logging your progress in a project. You can log your take on experiment results or your future plans inside a ModelLine, using comments as notes. Or you can add sequential comments to a model so that they will be recorded in its metadata.

Comments differ from descriptions in this sense because they store username, host and date when comment was written. This allows to have a log of notes with time that you can use to track your progress and if used on a shared machine as a collaboration tool.

cascade comment add
cascade comment ls

After writing a command you will see a prompt for typing your comment in. When done, press enter and check your comment by doing similar ls command.

1, ilia, my-pc-name   hello mlops
a few seconds before

8. Data Validation#

Data quality in ML projects is as important as the quality of the model. This is why Cascade focuses on integrated and effortless data validation.

When the project grows, it becomes hard to control what is going on with different Modifiers. Some may accept certain formats of data and it is hard to explicitly define those requirements within Modifier API.

This is where SchemaModifiers come in. They are special kind of Modifiers that allow defining input schema for when we do __getitem__.

Schema is defined using pydantic - an established tool for data validation and also an optional dependency, you’ll need to install it if you haven’t yet.

The problem with our initial setup is that we operated with raw dicts, making our schema implicit. If we were to reuse our datasets later, it would be hard for us or other engineers to quickly grasp the return value layout and it will also be easy to introduce errors in datasets that will be hard to debug.

Let’s define a simple schema for our dataset from the beginning of the tutorial.

from pydantic import BaseModel


class LabeledImage(BaseModel):
    x: np.ndarray
    y: int

    model_config = {"arbitrary_types_allowed": True}

Previous part is how we define schema in pydantic. You can use complex schemas and Fields to place requirements on the input of your Modifiers.

This is the entry point of data in our pipeline, so this part is important. However, we also can ensure data integrity inside of the pipeline.

Each time self._dataset[idx] is called, it will automatically check the returned value against our model.

from cascade.data import SchemaModifier


class ValidatingModifier(SchemaModifier):
    in_schema = LabeledImage

    def get(self, idx):
        item = self._dataset[idx]
        # Here you can do anything
        return item

Here we build a pipeline and augment our data using padding.

ds = ValidatingModifier(ds)

Let’s see the output.

print(ds[0])

Nothing special - validators are made to be effortless. They allow avoiding writing manual checks in every instance of a dataset. We just define a schema inside of the whole class of datasets and they automatically check values that they accept. And the return values stay the same.

Next example will show an actual case of input validation.

We will purposefully define some erroneous data to place before our padding transform. In this case we mess up the type of a label. This seems to be very real practical situation that would easily pass in our previous setup at would take some time to debug.

class EvilDataset(Dataset):
    def get(self, idx):
        return dict(x=np.zeros(18*18), y="hehe")

    def __len__(self):
        return 67

The following code will raise ValidationError, which we will catch and display the latest message.

from cascade.data import ValidationError

evil = EvilDataset()
evil = ValidatingModifier(evil)

try:
    evil[0]
except ValidationError as e:
    print(e)

9. Artifacts and Files#

Cascade wrappers provide unified interface for different ML solutions however in deployment scenarios they may obstruct underlying models.

To solve this problem artifacts were created. They are special methods that when implemented save only the artifact of the model and not the wrapper.

In the next block we implement those methods - they accept a folder (usually from ModelLine) and should save/load their artifact using it.

import os
import pickle

from sklearn.neural_network import MLPClassifier


class NeuralNet(BasicModel):
    def __init__(self):
        self._model = MLPClassifier()
        super().__init__()

    def save_artifact(self, path: str) -> None:
        with open(os.path.join(path, "artifact.pkl"), "wb") as f:
            pickle.dump(self._model, f)

    def load_artifact(self, path: str) -> None:
        with open(os.path.join(path, "artifact.pkl"), "rb") as f:
            self._model = pickle.load(f)

We create and save this model in line.

nn = NeuralNet()
line.save(nn)

To verify what is saved, let’s peek into the folder of the model. Special artifacts folder was created by the line where the model saved its artifact.

last_model_dir = os.path.join(line.get_root(), line.get_model_names()[-1])
print(os.listdir(last_model_dir))
print(os.listdir(os.path.join(last_model_dir, "artifacts")))

['model.pkl', 'meta.json', 'artifacts', 'SLUG']
['artifact.pkl']

Lots of cases may require linking files to the saved model. It can be sample predictions, figures and plots, logs or anything you want to keep around each model you save.

To allow this, Cascade features special method. Call add_file with a path to the required file and it will be copied into files folder inside a folder of the model in line.

Here we create dummy file with fake predictions and save it.

import json

dummy_predictions = [0, 1, 2, 3]

with open("dummy_predictions.json", "w") as f:
    json.dump(dummy_predictions, f)

We link the file by putting its path in add_file method. ModelLine will copy it on save.

nn.add_file("dummy_predictions.json")
line.save(nn)

Like previously we verify the files.

last_model_dir = os.path.join(line.get_root(), line.get_model_names()[-1])
print(os.listdir(last_model_dir))
print(os.listdir(os.path.join(last_model_dir, "files")))

['model.pkl', 'meta.json', 'files', 'artifacts', 'SLUG']
['dummy_predictions.json']

10. Scikit-learn Integration#

Many of the things we implemented in this tutorial can be reused in similar projects. This is one of the main principles on which Cascade was built. This is why most of the things we done using sklearn library is already implemented in Cascade utils module.

In this tutorial we will overview scikit-learn library integration in Cascade. It features default model class that can wrap pipelines of sklearn transformers and also special metric wrapper for sklearn.metrics module.

Now we do not need to implement our own model wrapper or care about different methods. Everything is already implemented in SkModel class. Notice how we pass blocks as a list of transforms. The explicit use of keyword parameter here is required.

from cascade.utils.sklearn import SkModel

model = SkModel(blocks=[LogisticRegression()])

The interface of this model’s fit function accepts lists of elements.

ds = DigitsDataset()

x = [item["x"] for item in ds]
y = [item["y"] for item in ds]

model.fit(x, y)

SkMetric class provides a wrapper around metrics module. You can pass a valid name from this module and it will be imported by Cascade for you. Cascade also features some aliases for metrics. acc will import sklearn.metrics.accuracy_score.

If metrics require any keyword arguments, you can pass them at the creation time.

from cascade.utils.sklearn import SkMetric

model.evaluate(
    x,
    y,
    [
        SkMetric("f1_score", average="macro"),
        SkMetric("acc"),
    ],
)

Let’s save the model and see how everything is handled automatically.

pprint(model.metrics)

line.save(model)

[SkMetric(name=f1_score, value=1.0, created_at=2024-08-14 19:37:46.556587+00:00),
SkMetric(name=acc, value=1.0, created_at=2024-08-14 19:37:46.556701+00:00)]

Notice how an artifact and a model are saved using the default implementation of save and save_artifact.

last_model_dir = os.path.join(line.get_root(), line.get_model_names()[-1])
print(os.listdir(last_model_dir))

['model.pkl', 'meta.json', 'artifacts', 'SLUG']

What’s Next#

Congratulations for completing the tutorial!

You can see other more specific tutorials or you can proceed to the How-to guides section.