Stéphan Taljaard
02/01/2023, 2:27 PMLeon Kozlowski
02/01/2023, 4:23 PMprefect cloud login -w <WORKSPACE_HERE> -k <API_KEY_HERE>
3. Registering a block via prefect block register --file src/block.py
4. Building a deployment via prefect deployment build src/flow.py:<NAME>
(with many other options)
5. Applying a deployment via prefect deployment apply <NAME>-deployment.yaml
This pipeline was working as expected in prefect==2.4.0
however since upgrading to prefect==2.7.5
we can only get to step 2 of this pipeline. After a successful login we see a warning log:
/usr/local/lib/python3.10/site-packages/prefect/context.py:496: UserWarning: Failed to create the Prefect home directory at /.prefect
with SettingsContext(profile=profile, settings=new_settings) as ctx:
This is being run inside of the docker image prefecthq/prefect:2.7.5-python3.10
Not sure of a path forward, I am also unsure why a .prefect
would be needed for a cloud deployment inside of an ephemeral deployment pipelineFuETL
02/01/2023, 4:44 PMJavier Ochoa
02/01/2023, 6:41 PMJohn-Craig Borman
02/01/2023, 6:44 PMMissingResult
exceptions on the initial flow run:
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 624, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/.../insert_data.py", line 246, in insert_data
if is_typing_flow(vendor_config, wait_for=[copy_file_result]):
File "/usr/local/lib/python3.10/dist-packages/prefect/tasks.py", line 367, in __call__
return enter_task_run_engine(
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 824, in enter_task_run_engine
return run_async_from_worker_thread(begin_run)
File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/asyncutils.py", line 152, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.10/dist-packages/anyio/from_thread.py", line 49, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
return f.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 965, in get_task_call_return_value
return await future._result()
File "/usr/local/lib/python3.10/dist-packages/prefect/futures.py", line 237, in _result
return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
File "/usr/local/lib/python3.10/dist-packages/prefect/states.py", line 89, in _get_state_result
raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
Jon
02/01/2023, 7:11 PMTomás Emilio Silva Ebensperger
02/01/2023, 8:11 PMsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: flow
Gaurav
02/01/2023, 8:28 PMNikhil Jain
02/01/2023, 8:34 PMconcurrency limits
work.
1. Say I set a limit of 10 on a particular work-queue (or a flow tag), and I get 15 concurrent requests for the flow run, what happens? Will the agent keep picking up runs until there are 10 runs in running
state? will the 5 remaining runs remain in scheduled
state? or will they go in pending
state? Is there any doc that explains the state transitions for when concurrency limits are set?
2. Currently we don’t have any concurrency limit but I noticed that when we get a big burst of flow runs triggered, some runs go into a pending state and remain pending forever. Wondering if we’d run into similar issues with concurrency limits
.
3. Is there a difference between concurrency limits on work-queue
vs tags
in terms of underlying implementation?Sam Cook
02/01/2023, 9:16 PMexpected_start_time
from within a task? If I retrieve the context using get_run_context
and the expected_start_time
field from within a task, I get a TaskRunContext and the expected start time for the specific task which is different from the flow's expected_start_time
by a fair margin, depending on how long it took to actually start the current task.Luke
02/01/2023, 9:17 PMAlix Cook
02/02/2023, 12:00 AMPending
state. We're running with Process
infrastructure and github storage. We're running the agent in the container, and the agent is pid 1, and I know that the agent process that picked up the flow is still alive (we have the process ping for all the flow runs its submitted).
because we run many things with strict concurrency, this issue is blocking our flows from running unless we kill the agent or cancel the flow. any thoughts?David Wang
02/02/2023, 3:00 AMKisho
02/02/2023, 4:28 AMjaehoon
02/02/2023, 5:39 AMlatif
02/02/2023, 8:59 AMEthienne Marcelin
02/02/2023, 9:44 AM@flow
def my_flow():
a = subflow1(...)
b = subflow2(...)
c = subflow3(a)
Would it be possible to create a graph from this and raise an error because subflow2 is a dangling node ?Malavika S Menon
02/02/2023, 10:21 AM15:48:07.612 | INFO | prefect.infrastructure.process - Process 'omicron-pentarus' exited cleanly.
15:48:22.670 | INFO | prefect.agent - Submitting flow run 'e1e13afc-b6ae-43c9-bfc2-a6608b7948c5'
15:48:22.710 | INFO | prefect.infrastructure.process - Opening process 'vagabond-coucal'...
15:48:22.715 | INFO | prefect.agent - Completed submission of flow run 'e1e13afc-b6ae-43c9-bfc2-a6608b7948c5'
Agent logs are stuck here for a long timeTon Steijvers
02/02/2023, 1:53 PMtimeout-seconds
defined that is already long past. Still the flow run stays in "Running" state. The k8s pod that was running the flow is already terminated. We saw this behaviour occasionally in Prefect 1 and were hopeful that it would improve in Prefect 2. Is there anything we should do to get this more reliable?Guy Altman
02/02/2023, 2:56 PMLin Hui
02/02/2023, 3:22 PMkiran
02/02/2023, 4:21 PMSorry, something went wrong creating block
. I logged out/back in and tried again and got the same thingThet Naing
02/02/2023, 4:25 PMmain
branch but not with any other branches. Anyone else have this issue before? I am on Prefect v2.7.10Ha Pham
02/02/2023, 5:02 PMNick Coy
02/02/2023, 5:19 PMJean-Michel Provencher
02/02/2023, 5:22 PMLeon Kozlowski
02/02/2023, 7:16 PM2.7.10
I've managed to get a flow deployed, but it seems that the agent can't submit flow runs due to a permissions error
(Posting tracebacks in thread)Marc Lipoff
02/02/2023, 9:58 PMIlya Galperin
02/02/2023, 11:27 PMaaron
02/03/2023, 1:13 AM