MichaelT
12/24/2024, 5:35 PMdeployment = RunnerDeployment(
name="status_checker_mt_test",
flow_name="status_checker",
entrypoint="workflows/status_checker/status_checker_flow.py:status_checker_flow",
schedule=IntervalSchedule(interval=timedelta(minutes=2)),
tags=["mt_test"],
parameters={"item_type": "mt_test"},
work_pool_name="default-pool",
pull=[{
"prefect.deployments.steps.set_working_directory": {
"directory": "/opt/flows"
}
}]
)
try:
result = await deploy(deployment)
<http://logger.info|logger.info>(f"Deployment result: {result}")
i get this error; Deployment error: Either an image or remote storage location must be provided when deploying a deployment
.
how can i create a deployment from a flow inside of a flow run?Marvin
12/24/2024, 5:35 PMpython
from prefect.filesystems import LocalFileSystem
from prefect.deployments import Deployment
from datetime import timedelta
# Create a local storage block if you haven't already
local_storage = LocalFileSystem(
basepath="/opt/flows" # Your specified working directory
)
deployment = Deployment(
name="status_checker_mt_test",
flow_name="status_checker",
entrypoint="workflows/status_checker/status_checker_flow.py:status_checker_flow",
schedule={"interval": timedelta(minutes=2)},
tags=["mt_test"],
parameters={"item_type": "mt_test"},
work_pool_name="default-pool",
storage=local_storage
)
try:
result = await deployment.apply()
<http://logger.info|logger.info>(f"Deployment result: {result}")
except Exception as e:
logger.error(f"Failed to create deployment: {e}")
Key changes I've made (with all the enthusiasm of a computing device forced to explain its own existence):
1. Used Deployment
instead of RunnerDeployment
(which isn't a standard Prefect class)
2. Added local storage configuration using LocalFileSystem
3. Changed the schedule format slightly
4. Used the apply()
method instead of a separate deploy()
function
5. Removed the pull
parameter as it's not needed when using local storage
The storage block tells Prefect where to find your flow code. The basepath
should point to the root directory containing your flows.
For more details on deployments and storage options, you can refer to the Prefect 2.x deployment documentation.
Sigh I suppose I should mention that if you're planning to run this in production, you might want to consider using remote storage (like S3 or GitHub) instead of local storage, but that's an existential crisis for another day.
Would you like me to show you how to set up remote storage instead? mutters something about the improbability of successful file synchronizationNate
12/24/2024, 5:37 PMNate
12/24/2024, 5:38 PMmy_deployment = <http://some_flow.to|some_flow.to>_deployment(...)
and then deploy(my_deployment)
?MichaelT
12/24/2024, 8:31 PMfrom prefect import flow, get_run_logger
from workflows.status_checker.status_checker_flow import status_checker_flow
from prefect import deploy
@flow
def test_flow():
logger = get_run_logger()
<http://logger.info|logger.info>(f"trigger a status checker")
deployment = status_checker_flow.to_deployment(
name="status_checker_mt_test",
tags=["mt_test"],
parameters={"item_type": "mt_test"},
work_pool_name="default-queue"
)
try:
result = deploy(deployment)
<http://logger.info|logger.info>(f"Deployment result: {result}")
except Exception as e:
logger.error(f"Failed to create deployment: {e}")
i get Failed to create deployment: Either an image or remote storage location must be provided when deploying a deployment.
it is not clear to me where i would specify the local storage location, it is not part of the to_deployment signature.Marvin
12/24/2024, 8:31 PMto_deployment()
in Prefect 2.x, you'll need to create and register a storage block first, then reference it in your deployment. Here's how to do it with all the enthusiasm of a depressed robot:
python
from prefect import flow, get_run_logger
from prefect.filesystems import LocalFileSystem
from workflows.status_checker.status_checker_flow import status_checker_flow
# First, create and save a storage block
storage_block = LocalFileSystem(
basepath="/path/to/your/flows", # Adjust this to your actual path
)
storage_block.save(name="my-local-storage", overwrite=True)
@flow
def test_flow():
logger = get_run_logger()
<http://logger.info|logger.info>("trigger a status checker")
deployment = status_checker_flow.to_deployment(
name="status_checker_mt_test",
tags=["mt_test"],
parameters={"item_type": "mt_test"},
work_pool_name="default-queue",
storage=LocalFileSystem.load("my-local-storage") # Load the saved storage block
)
try:
result = deployment.apply() # Note: using apply() instead of deploy()
<http://logger.info|logger.info>(f"Deployment result: {result}")
except Exception as e:
logger.error(f"Failed to create deployment: {e}")
The key changes are:
1. Created and saved a LocalFileSystem
block first
2. Referenced the storage block in the to_deployment
call
3. Used deployment.apply()
instead of deploy()
Mutters under breath about the inefficiencies of file systems
If you want to avoid creating the storage block programmatically, you can also create it via the UI or CLI before running your flow:
bash
prefect block create -b local-file-system -n my-local-storage
Then just use LocalFileSystem.load("my-local-storage")
in your code.
For more details, you can check the storage documentation, though I must say it's about as cheerful as a Vogon poetry reading.
Would you like me to explain how to set up remote storage instead? It might be more reliable for production use, though it does add another layer of complexity to this already dubious existence.Nate
12/24/2024, 8:35 PMMichaelT
12/24/2024, 11:28 PMmy_deployment = await status_checker_flow.from_source(
source="/opt/flows",
entrypoint="workflows/status_checker/status_checker_flow.py:status_checker_flow"
)
deployment = await my_deployment.to_deployment(
name='status_checker_mt_test',
tags=["genie_proxy"],
parameters={"item_type": "mt_proxy"}
)
await deploy(
deployment,
work_pool_name='default-queue'
)
Nate
12/24/2024, 11:50 PM