How to read/write/sync data on cloud with Kedro - kedro

Question

In short: how can I save a file both locally AND on cloud, similarly how to set to read from local.
Longer description: There are two scenario, 1) building model 2) serving model through API. In building the model, a series of analysis is done to generate feature and model. The result will be written locally. At the end everything will be uploaded to S3. For serving the data, first all required files which are generated from the first step, will be downloaded.
I am curious how I can leverage Kedro here. Perhaps I can define two entries for each file conf/base/catalog.yml one corresponds to the local version and the second for the S3. But perhaps not the most efficient way when I am dealing with 20 files.
Alternatively, I can upload the files using my own script to S3 and exclude the synchronization from Kedro ! in otherwords, Kedro is blind from the fact that there are copies exist on the cloud. Perhaps this approach is not the most Kedro-friendly way.

Answer 1

Not quite the same, but my answer here could potentially be useful.
I would suggest that the simplest approach in your case is indeed defining two catalog entries and having Kedro save to both of them (and load from the local one for additional speed up) which gives you the ultimate flexibility, though I do admit isn't the prettiest.
In terms of avoiding all your node functions needing to return two values, I'd suggest applying a decorator to certain nodes that you tag with a certain tag, e.g tags=["s3_replica"] taking inspiration from the below script (stolen from a colleague of mine):
class S3DataReplicationHook:
"""
Hook to replicate the output of any node tagged with `s3_replica` to S3.
E.g. if a node is defined as:
node(
func=myfunction,
inputs=['ds1', 'ds2'],
outputs=['ds3', 'ds4'],
tags=['tag1', 's3_replica']
)
Then the hook will expect to see `ds3.s3` and `ds4.s3` in the catalog.
"""
#hook_impl
def before_node_run(
self,
node: Node,
catalog: DataCatalog,
inputs: Dict[str, Any],
is_async: bool,
run_id: str,
) -> None:
if "s3_replica" in node.tags:
node.func = _duplicate_outputs(node.func)
node.outputs = _add_local_s3_outputs(node.outputs)
def _duplicate_outputs(func: Callable) -> Callable:
def wrapped(*args, **kwargs):
outputs = func(*args, **kwargs)
return (outputs,) + (outputs,)
return wrapped
def _add_local_s3_outputs(outputs: List[str]) -> List[str]:
return outputs + [f'{o}.s3' for o in outputs]
The above is a hook, so you'd place it in your hooks.py file (or wherever you want) in your project and then import it into your settings.py file and put:
from .hooks import ProjectHooks, S3DataReplicationHook
hooks = (ProjectHooks(), S3DataReplicatonHook())
in your settings.py.
You can be slightly cleverer with your output naming convention so that it only replicates certain outputs (for example, maybe you agree that all catalog entries that end with .local also have to have a corresponding .s3 entry and you mutate the outputs of your node in that hook accordingly, rather than do it for every output.
If you wanted to be even cleverer, you could inject the corresponding S3 entry into the catalog using a after_catalog_created hook rather than manually writing the S3-versioon of the dataset in your catalog, again, as per a naming convention you choose. Though I'd argue that writing the S3 entries is more readable in the long-run.

Answer 2

There are 2 ways I can think of. A simpler approach is to use --env conf for both cloud and local. https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#additional-configuration-environments
conf
├── base
│ └──
├── cloud
│ └── catalog.yml
└── my_local_env
└── catalog.yml
And you can call kedro run --env=cloud or kedro run --env=my_local depending on which env you want to use.
Another more advanced way is to use TemplatedConfigLoader https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html
conf
├── base
│ └── catalog.yml
├── cloud
│ └── globals.yml (contains `base_path:s3-prefix-path`)
└── my_local
└── globals.yml (contains `base_path:my_local_path`)
In catalog.yml, you can refer to base_path like this
my_dataset:
filepath: s3:${base_path}/my_dataset
And you can call kedro run --env=cloud or kedro run --env=my_local depending on which env you want to use.