Chris Gunderson
01/10/2023, 2:11 PMChristopher Boyd
01/10/2023, 2:38 PMJames Sopkin
01/10/2023, 3:25 PMlialzm
01/10/2023, 3:45 PMDownloading flow code from storage at None
Flow could not be retrieved from deployment.
my code
ali= RemoteFileSystem.load("ali")
deployment = Deployment.build_from_flow(
flow=download,
name="test",
storage=ali,
schedule=(CronSchedule(cron="0 16 * * *"))
)
deployment.apply()
James Phoenix
01/10/2023, 3:46 PMJames Phoenix
01/10/2023, 3:47 PMprefect kubernetes manifest orion | kubectl apply -f -
0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.James Phoenix
01/10/2023, 3:50 PMIdan
01/10/2023, 3:50 PMos.seteuid(priv_user_id)
(and then later resets to original_user_id
).
This is run in a container, so the original_user_id
is root
, which entails PREFECT_HOME
is /root/.prefect/
.
Our tasks are long-running, so we also cache them. Then, every now and then, a task succeeds, but fails with:
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/root/.prefect/storage'
During handling of the above exception, another exception occurred:
PermissionError: [Errno 13] Permission denied: '/root/.prefect/storage'
Any smart ideas how to facilitate both needs?James Phoenix
01/10/2023, 3:50 PMPaco Ibañez
01/10/2023, 5:54 PMparameter_openapi_schema
Jean-Michel Provencher
01/10/2023, 6:38 PMRyan Peden
01/10/2023, 7:09 PMjack
01/10/2023, 8:47 PMapolisskaya
01/10/2023, 10:10 PMFinished in state Failed('Flow run encountered an exception. MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.\n')
at the end. Manually retrying the flow succeeds, but I do have retries=3
in the flow
decorator and the flow doesn't appear to be retriedIlya Galperin
01/11/2023, 12:32 AMBrokenPipeError: [Errno 32] Broken pipe
crashes in one of our flows on 2.7.7, running on DaskTaskRunner. This flow runs ~1000 tasks, occasionally one of them will enter a Crashed state with this error and cause our flow to enter a Failed state. Retries on these crashed tasks don’t seem to work (I’m guessing Crashed state tasks are excluded from retry logic). Full traceback in the thread.Aaron Goebel
01/11/2023, 1:40 AMcreate_flow_run_from_deployment
/ run_deployment
APIs to compose my deployments on http request. E.g. I get a payload defining a set of flows to chain together, that triggers a parent container flow, and that container flow does the orchestration of tying things together.
The way I'm thinking of running it, because there are dependencies, is to have the run_deployment
code wrapped in a task like this :
@task
async def run_deployment(depl_id: str, parameters: dict):
async with prefect.context(**prefect.context.run_params):
async with prefect.Client() as client:
run = await client.create_flow_run_from_deployment(deployment_id=depl_id, parameters=parameters)
run_state = await run.get_state()
return run_state.result
and then in an orchestration flow I think I want to do something like this where I create these tasks for each deployment.
@flow
async def container_flow(flow_graph: dict):
results = {}
ordered_flows = order(flow_graph)
# create tasks
tasks = {name: run_deployment.map(deployment_id=flow_params['deployment_id'], parameters=flow_params['inputs'])
for name, flow_params in flow_graph.items()}
# set dependencies
for flow_name, flow_params in flow_graph.items():
for dependency in flow_params.get('dependencies', []):
tasks[flow_name].set_upstream(tasks[dependency], flow=True)
# run tasks concurrently
flow_results = await prefect.engine.run(tasks, return_tasks=True)
# store results
for flow_name, flow_result in flow_results.items():
results[flow_name] = flow_result.result
return results
Aaron Goebel
01/11/2023, 2:20 AMrun_deployment
orchestrator pattern to chain together deployments dynamically. Some of these deployments depend on others. I've attempted to finagle a way around this by wrapping run_deployment
in a task as such:
@task
async def run_deployment(depl_id: str, parameters: dict):
async with prefect.context(**prefect.context.run_params):
async with prefect.Client() as client:
run = await client.create_flow_run_from_deployment(deployment_id=depl_id, parameters=parameters)
run_state = await run.get_state()
return run_state.result
I then create tasks as such
tasks = {name: run_deployment.map(deployment_id=flow_params['deployment_id'], parameters=flow_params['inputs'])
for name, flow_params in graph.items()}
and set their dependencies:
# set dependencies
for flow_name, flow_params in graph.items():
for dependency in flow_params.get('dependencies', []):
tasks[flow_name].set_upstream(tasks[dependency], flow=True)
the goal here is to dogfood the taskrunner to chain these together and kick off the flow with something like
await task_runner.submit(tasks)
Two issues I see with this though:
1. Some of the parameters
to downstream create_deployment
runs are derived from upstream deployment runs. I wonder if anyone knows an elegant way of doing that?
2. await task_runner.submit(tasks)
would actually work as anticipated?wonsun
01/11/2023, 6:43 AMwonsun
01/11/2023, 7:22 AMRuntimeWarning: coroutine 'Block.load' was never awaited
secret_block = Secret.load('3f')
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
• So I make a new secret block and that's also don't working with below error message..
C:\Users\user\AppData\Local\Temp\ipykernel_12716\3323813457.py:3: RuntimeWarning: coroutine 'Block.load' was never awaited
secret_block = Secret.load("new-net")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In [27], line 6
3 secret_block = Secret.load("new-net")
5 # Access the stored secret
----> 6 secret_block.get()
AttributeError: 'coroutine' object has no attribute 'get'
Is there are happen to trouble by new network?
Thanks for your reply.Stéphan Taljaard
01/11/2023, 9:18 AMPB
01/11/2023, 9:35 AMVadym Dytyniak
01/11/2023, 10:01 AMState data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
Hoàng Lâm
01/11/2023, 10:02 AMD:\Athena\etl>prefect agent start "work_queue_for_test"
Agents now support multiple work queues. Instead of passing a single argument, provide work queue names with the `-q` or`--work-queue` flag: `prefect agent start -q work_queue_for_test`
Starting v2.7.4 agent with ephemeral API...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): work_queue_for_test...
16:18:20.487 | INFO | prefect.agent - Submitting flow run 'fd08f02a-18ca-4538-bdf1-728dc7e88e22'
16:18:20.604 | INFO | prefect.infrastructure.process - Opening process 'beta736-bajor'...
'C:\Users\Will' is not recognized as an internal or external command,
operable program or batch file.
16:18:20.622 | ERROR | prefect.infrastructure.process - Process 'beta736-bajor' exited with status code: 1
16:18:20.651 | INFO | prefect.agent - Completed submission of flow run 'fd08f02a-18ca-4538-bdf1-728dc7e88e22'
16:18:20.668 | INFO | prefect.agent - Reported flow run 'fd08f02a-18ca-4538-bdf1-728dc7e88e22' as crashed: Flow run infrastructure exited with non-zero status code 1.
Satyasheel
01/11/2023, 10:31 AMmax
01/11/2023, 10:46 AMTim-Oliver
01/11/2023, 12:39 PMKirill Egorov
01/11/2023, 1:42 PMJason Noxon
01/11/2023, 2:26 PMJohn-Craig Borman
01/11/2023, 2:30 PMprefect.deployments.run_deployment()
function:
If I deploy a flow without specifying a deployment name, how can I use run_deployment()
to trigger a flow run? Would it just be run_deployment(name='flow_name')
?
related docs: https://docs.prefect.io/concepts/deployments/#create-a-flow-run-in-a-python-scriptDavid Elliott
01/11/2023, 2:43 PMDavid Elliott
01/11/2023, 2:43 PMChristopher Boyd
01/11/2023, 2:47 PMDavid Elliott
01/11/2023, 2:50 PMChristopher Boyd
01/11/2023, 3:01 PMDavid Elliott
01/11/2023, 4:05 PMChristopher Boyd
01/11/2023, 4:07 PM