Shared Storage

When you have a workflow with more than one piece you may want to transfer data between them.

Transfering data between pieces can be done in two ways:

1. Using XCOM: This is recommended only for small data. By default all values returned by a piece, defined in the OuputModel, are returned to xcom and can be read by downstream pieces.

2. Using Shared Storage: Some Pieces produce data that can not be direclty transfered to downstream Pieces in other way than using disk. A common example is a piece that produces a large file that needs to be used by another piece. In this case, you can use the available Shared Storage options.

What exactly is Shared Storage?

Domino's Shared Storage is a mounted volume that can be used by all pieces in a workflow, where each piece can have different access level to this volume. With Shared Storage, you can work with files from different pieces as if they were all on the same file system, even if they are running on different machines.

Currently, Domino support the following Shared Storage options:

Running in compose

• None: No shared storage is available. This means that pieces can not "see" each other files.
• Local: Shared storage is available and the volume is mounted on the same machine that is running the platform.

The diagram below shows how the Shared Storage works when running in compose.

Running in Kubernetes

• None:No hared storage is available. This means that pieces can not "see" each other files.
• Local:Shared storage is available and the volume is mounted on an AWS S3 bucket folder. This mean each piece result is synced to the S3 bucket as it was a local folder. This option is only available when running in Kubernetes.

The diagram below shows how the Shared Storage works when running in Kubernetes.

Workflow Shared Storage

To set what type of shared storage the workflow will use go to workflow Settings and choose your desired option in Storage Source dropdown.

Piece Shared Storage

To set the access level to the shared storage a Piece will have simply double click on the desired Piece, go to Advanced Options and select the access level on the Storage Access Mode dropdown.