kedro PartitionedDataSet lazy saving to spare memory? - kedro

Question

I am working with PartionedDataSet in kedro. One of the data set is of type pillow.ImageDataSet:
raw_images:
type: PartitionedDataSet
<<: *data_path_on_disk
dataset:
type: pillow.ImageDataSet
filename_suffix: ".png"
I want to process this set of image (for example cropping them) and save them in a new PartionedDataSet of type pillow (same as before except for the path).
node(func=crop_image, inputs="raw_images",outputs="cropped_images")
where crop_image is defined as follow:
def crop_image(images: dict):
return {image_path: image().crop([10,10,20,20]) for image_path, image in images.items()}
How will the dictionary be build? Will it be built and be completely stored in memory (which will overload soon for a big dataset) or will it write the image progressively to the disk as they are computed?

Answer 1

In the form you've presented it:
def crop_image(images: dict):
return {image_path: image().crop([10,10,20,20]) for image_path, image in images.items()}
it has nothing to do with Kedro. Your crop_image will be executed by Python eagerly, making the execution allocate the dictionary as well as the images, before you return it.
If you want to have lazy saving, follow the Partitioned dataset lazy saving guide: https://kedro.readthedocs.io/en/stable/data/kedro_io.html#partitioned-dataset-lazy-saving
You still need to allocate the dictionary with keys (so if you have millions of them, the dictionary will get large), but the save method can be deffered and executed lazily, like this (remember about closures):
def crop_single_image_lazy(img):
def crop():
return img().crop([10,10,20,20])
return crop
def crop_image(images: dict):
"""
Returns:
Dictionary of the partitions to create to a function that creates them.
"""
return {
image_path: crop_single_image_lazy(image) for image_path, image in images.items()
}

Answer 2

If you want to enable lazy-saving, you can use a dict of callable to do that.
See this part of the documentations.
https://kedro.readthedocs.io/en/stable/data/kedro_io.html#partitioned-dataset