Skip to main content

Domino-py

danger

Domino package can not be tested in docker compose with hot reload.

Kubernetes​

When running the application in development mode using the key DOMINO_DEPLOY_MODE=local-k8s-dev in your config.toml file, the application will use the Kubernetes API to create some volumes mappings, allowing some development features:

  1. Hot reloading for local domino package.
  2. Use of local helm charts when creating the cluster.
  3. Use of local frontend image in cluster.
  4. Use of local rest image in cluster.
  5. Hot reloading for local pieces repositories code.

1. Activating hot reloading for local domino package​

If you want to run your cluster with hot reloading for your local domino package you should follow the next steps:

  1. Go to your config.toml file in the dev section
  2. Add the key DOMINO_LOCAL_DOMINO_PACKAGE with the value to the path of your src/domino folder of your local domino package. Example: DOMINO_LOCAL_DOMINO_PACKAGE=path/to/your/domino/src/domino

2. Using local helm charts when creating the cluster​

If you need to test run your cluster with local helm charts you should follow the same steps as the previous section. The applicaton will get the helm folder from the path path/to/your/domino/helm/domino based on the path you defined in the DOMINO_LOCAL_DOMINO_PACKAGE key. So, edit your helm charts in the path/to/your/domino/helm/domino folder and the application will use them when creating the cluster.

3. Using local frontend image in cluster​

If you want to run your cluster with a local frontend image you should follow the next steps:

  1. Build your local frontend image. You can use the following command from the root of the project: DOCKER_BUILDKIT=1 docker build -f ./frontend/Dockerfile.dev -t domino-frontend ./frontend
  2. Add the key DOMINO_FRONTEND_IMAGE with the value domino-frontend (or other tag you choose) to your config.toml file in the dev section. Important: This will not activate hot reloading for your local frontend code. If you need to update your frontend code in the cluster you should build the image again.

4. Using local rest image in cluster​

As the frontend image, if you want to run your cluster with a local rest image you should follow the next steps:

  1. Build your local rest image. You can use the following command from the root of the project: DOCKER_BUILDKIT=1 docker build -f ./rest/Dockerfile -t domino-rest ./rest
  2. Add the key DOMINO_REST_IMAGE with the value domino-rest (or other tag you choose) to your config.toml file in the dev section.

This will load your local rest image in the kind cluster. Important: This will not activate hot reloading for your local rest code. If you need to update your rest code in the cluster you should build the image again.

5. Using local airflow base image​

If you need to update the base airflow image used in worker and scheduler containers you can do it by following the next steps:

  1. Build your local airflow base image. You can use the following command from the root of the project: DOCKER_BUILDKIT=1 docker build -f Dockerfile-airflow-domino.dev -t domino-airflow .
  2. Addthe key DOMINO_AIRFLOW_IMAGE with the value domino-airflow (or other tag you choose) to your config.toml file in the dev section.

This will load your local airflow base image in the kind cluster. Important: This is useful if you want to update airflow image dependencies. To update only the domino package code in the airflow containers you don't need to build the image again, you can use the hot reloading feature for local domino package describe in the section 1.

5. Activating hot reloading for local pieces repositories code​

You can activate hot reloading for the local code of your pieces repositories. To do so, you should follow the next steps:

  1. Go to your config.toml file in the dev section
  2. Add the key <piece_repository_name> with the value to the path of your <piece_repository_name> folder of your local pieces repository. Example: default_domino_pieces=path/to/your/default_domino_pieces The <piece_repository_name> must be the same name in the piece repository config.toml file in REPOSITORY_NAME key.

The application will create a volume mapping between the path you specified and the path in the cluster where the pieces repository is mounted. This will allow you to update your pieces repository code and see the changes in the cluster without the need to build the image again.

Docker Compose​

You can run your application in development mode using docker compose file. To do so, you should run using the docker-compose-dev.yaml file, as follows:

docker-compose -f docker-compose-dev.yaml up

This will activate some development features, such as:

  1. Hot reloading for local domino package in worker and scheduler containers.
  2. Hot reloading for rest code.
  3. Hot reloading for frontend code.
danger

Important: This will not activate hot reloading for your pieces repositories code and will not activate hot reloading for local domino package in the pieces containers. If you are running a piece and you need to have the domino package updated in the piece container you should update the piece image wit the domino development package you want to use.

note

Workaround: As mentioned before, currently we don't have a direct way to allow hot reloading for domino package in pieces containers, however, you can use the following workaround:

  1. Go to src/domino/custom_operators/docker_operator.py file.
  2. find the following code:
dev_pieces = False
if dev_pieces:
    piece_repo_name = repository_url.split("/")[-1]
    local_repos_path = f"/path/to/your/pieces/location/{piece_repo_name}"
    mounts = [
        Mount(source='/path/to/your/domino/location/src/domino', target='/usr/local/lib/python3.10/site-packages/domino/', type='bind', read_only=True),
        Mount(source=local_repos_path, target='/home/domino/pieces_repository/', type='bind', read_only=True),
    ]

  1. Change the dev_pieces parameter to True and populate the source in each mount path with a folder pointing to: domino source code and a folder containing all source codes for the pieces used by your workflows .
note

The workflow will attempt to use your mounted pieces folder, so all pieces must be present in the mounted folder. If not, the workflow will fail in execution.