then. Is that use case normal or am I thinking about it wrong? Just to double check that I'm not attempting anything totally unidiomatic here.
dbt_task = DbtShellTask( ... ) with Flow("dbt-flow") as flow: dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True) dbt_commands = dbt_command_string.split(',') for command in dbt_commands: dbt_task( command=dbt_command, ... )
that forces it to run serially?
It's always the same task for both pipelines - I've split out a common task to load data to BigQuery into another file and this is the task that fails. It will run fine for a few days and then randomly decide that it can't find this task, even though the exact same task is called two other times during each pipeline, foo-1 and foo-2. I'm wondering if I'm the first person to have their install randomly lose tasks..
Unexpected error while running flow: KeyError('Task slug foo-3 not found in the current Flow; this is usually caused by changing the Flow without reregistering it with the Prefect API.')
has been released and here are a few notable changes: ⁉️ Formatted GraphQL errors are now raised by the Client 👟 Refactored a few Client API calls for performance 📚 Various enhancements and fixes to tasks in the Task Library A big thank you to our contributors who helped out with this release! Full changelog https://github.com/PrefectHQ/prefect/releases/tag/0.13.18
FROM python:3.8 RUN python -m pip install sqlalchemy prefect
from prefect import Flow, task from prefect.environments.storage import Docker import sqlalchemy @task def some_task(): print(sqlalchemy.__version__) with Flow("Some Flow") as flow: some_task() if __name__ == "__main__": flow.storage = Docker( registry_url="<http://docker.io|docker.io>", image_name="storage_image", image_tag="latest", base_image="test-base", local_image=True, ) flow.register(project_name="Development")
Please could you guide me with the best approach? Many thanks!
#!/usr/bin/env bash set -x docker build . -t test-base #: Approach 1 python main.py #: Approach 2 docker run -v /var/run/docker.sock:/var/run/docker.sock -v ~/.docker:/root/.docker -e PREFECT__CLOUD__AUTH_TOKEN="$PREFECT__CLOUD__AUTH_TOKEN" -v $(pwd)/main.py:/main.py test-base bash -c 'python /main.py'