Alberto Fernández Zazo
10/03/2022, 10:00 AMStéphanie Cérulli
10/03/2022, 10:05 AMKrzysztof Ciesielski
10/03/2022, 10:10 AMMalavika S Menon
10/03/2022, 10:31 AMprefect config set PREFECT_API_URL=<https://example.com/api>
. However when I try to run the server it defaults back to http://127.0.0.1:4200/apiHedgar
10/03/2022, 12:48 PMsecret-block = Secret.load(“api-key”)
I want to use this block in the body of my flow how can I go about this? Should I just pass secret-block
? Do I need to use the get method anywhere?Igor Kotua
10/03/2022, 1:48 PMStefan Rasmussen
10/03/2022, 2:43 PM@task
decorator after that task has been defined? Say I use a task from a collection that and want to use a custom cache_key_fn
.
It looks like i could do it by
from copy import copy
from some_collection import some_task
some_task_cache = copy(some_task)
some_task_cache.cache_key_fn = my_cache_fn
but that seems a bit hacky.
The actual problem is the other way around - I want to provide a collection for users, but they would want to modify the task decorator arguments.Hedgar
10/03/2022, 3:08 PMSean Malone
10/03/2022, 3:13 PMFailed to delete flow run
message in the UI. Is this a known issue? Or any thought as to why this occurs?Farooque Shaikh
10/03/2022, 3:27 PMGhassan Hallaq
10/03/2022, 4:53 PMJeff Quinn
10/03/2022, 5:13 PMChristian Juhl
10/03/2022, 6:54 PMDarin Douglass
10/03/2022, 7:05 PMNace Plesko
10/03/2022, 7:47 PMShellTasks
. Setting timeout like it says in the documentation doesn't have any effect, the task doesn't get terminated after 10 seconds in this case. Is that a bug in Prefect or is it an issue on my part and I'm somehow misconfiguring ShellTask
?
I really appreciate any help on this and thank you in advance!Florian Giroud
10/03/2022, 8:15 PMAustin Anderson
10/03/2022, 8:38 PMname: prefect-service
channels:
- defaults
dependencies:
- python=3.9.13
- pip=22.1.2
- pip:
- prefect==2.4.5
I am hitting an error after running prefect deployment apply log_flow-deployment.yaml
:
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000226A897BEE0>
... redacted for readability ...
RuntimeError: Event loop is closed
Any idea what is happening here? Please let me know if I can provide any additional info.Lukáš Polák
10/03/2022, 8:59 PMquery { api { core_version } }We ran into a problem with version 1.4.0 when prefect API reports version "1.3.0+13.g4142953a5". Is that intended?
Alfred Martinez
10/03/2022, 9:08 PMMilenko Beslic
10/03/2022, 9:59 PMtrigger_dbt_cloud_job_run_and_wait_for_completion
fails with AttributeError: 'NoneType' object has no attribute 'call'
.
Here is the log:
21:33:15.658 | INFO | Flow run 'charcoal-harrier' - Created subflow run 'hot-civet' for flow 'emails-gold'
21:33:15.958 | INFO | Flow run 'hot-civet' - Created subflow run 'cocky-pony' for flow 'Trigger dbt Cloud job run and wait for completion'
21:33:16.069 | ERROR | Flow run 'cocky-pony' - Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 589, in orchestrate_flow_run
result = await flow_call()
File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cloud/jobs.py", line 287, in trigger_dbt_cloud_job_run_and_wait_for_completion
run_id_future = get_run_id.submit(triggered_run_data_future)
File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 492, in submit
return enter_task_run_engine(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 741, in enter_task_run_engine
return flow_run_context.sync_portal.call(begin_run)
AttributeError: 'NoneType' object has no attribute 'call'
21:33:16.133 | INFO | Flow run 'cocky-pony' - Created task run 'Trigger dbt Cloud job run-e4229069-0' for task 'Trigger dbt Cloud job run'
21:33:16.134 | INFO | Flow run 'cocky-pony' - Submitted task run 'Trigger dbt Cloud job run-e4229069-0' for execution.
21:33:16.264 | INFO | Task run 'Trigger dbt Cloud job run-e4229069-0' - Triggering run for job with ID 135917
21:33:16.558 | INFO | Task run 'Trigger dbt Cloud job run-e4229069-0' - Run successfully triggered for job with ID 135917. You can view the status of this run at <https://cloud.getdbt.com/#/accounts/xxxxx/projects/156718/runs/86440872/>
21:33:16.683 | INFO | Task run 'Trigger dbt Cloud job run-e4229069-0' - Finished in state Completed()
21:33:16.812 | ERROR | Flow run 'cocky-pony' - Finished in state Failed('Flow run encountered an exception.')
21:33:16.812 | ERROR | Flow run 'hot-civet' - Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 596, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/opt/prefect/flows/emails_deploy.py", line 35, in emails_gold
run = trigger_dbt_cloud_job_run_and_wait_for_completion(
File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 390, in __call__
return enter_flow_run_engine_from_flow_call(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 163, in enter_flow_run_engine_from_flow_call
return run_async_from_worker_thread(begin_run)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 522, in create_and_begin_subflow_run
return terminal_state.result()
File "/usr/local/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 145, in result
raise data
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 589, in orchestrate_flow_run
result = await flow_call()
File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cloud/jobs.py", line 287, in trigger_dbt_cloud_job_run_and_wait_for_completion
run_id_future = get_run_id.submit(triggered_run_data_future)
File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 492, in submit
return enter_task_run_engine(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 741, in enter_task_run_engine
return flow_run_context.sync_portal.call(begin_run)
AttributeError: 'NoneType' object has no attribute 'call'
21:33:16.892 | ERROR | Flow run 'hot-civet' - Finished in state Failed('Flow run encountered an exception.')
...
merlin
10/03/2022, 11:58 PMp102-13.py
If I leave .result()
in the code, it fails with error:
letter = get_letter().result()
AttributeError: 'str' object has no attribute 'result'
But remove the .result()
bit and I get success:
poetry run python src/quickstart/p102-13.py
16:48:31.385 | INFO | prefect.engine - Created flow run 'jasper-uakari' for flow 'Result Flow'
16:48:31.454 | INFO | Flow run 'jasper-uakari' - Created task run 'Get a letter-c414aff6-0' for task 'Get a letter'
16:48:31.454 | INFO | Flow run 'jasper-uakari' - Executing 'Get a letter-c414aff6-0' immediately...
16:48:31.468 | INFO | Task run 'Get a letter-c414aff6-0' - A
16:48:31.478 | INFO | Task run 'Get a letter-c414aff6-0' - Finished in state Completed()
16:48:31.478 | INFO | Flow run 'jasper-uakari' - You got: A
16:48:31.488 | INFO | Flow run 'jasper-uakari' - Finished in state Completed('All states completed.')
The poetry environment is:
[tool.poetry.dependencies]
python = "^3.10"
prefect = "^2.4.5"
greenlet = "^1.1.3"
wonsun
10/04/2022, 4:33 AMIgor Kotua
10/04/2022, 7:32 AMThomas Fredriksen
10/04/2022, 9:02 AMDeepanshu Aggarwal
10/04/2022, 9:54 AMBogdan Serban
10/04/2022, 10:05 AM/
, for instance /prefect/
. I am having an issue with loading the UI when I access <http://localhost/prefect/>
as the UI cannot load the corresponding assets, because it ignores the location prefix when doing the requests, for instance: <http://localhost/assets/index.1a4d60fb.js>
is available at <http://localhost/prefect/assets/index.1a4d60fb.js>
. Is there a way to specify a prefix or some sort of redirect URL for the UI to use when doing requests?Connor Vaid
10/04/2022, 11:10 AM{
"errors": [
{
"path": [
"flow_run",
0,
"id"
],
"message": "Cannot return null for non-nullable field flow_run.id.",
"extensions": {
"code": "INTERNAL_SERVER_ERROR"
}
}
],
"data": null
}
When I add the following “where” filter it seems to work (this would suggest I have some flows with null ids?), however when running a query to find flows with null ids nothing is returned.
{
"where": {
"flow": {
"id": {
"_is_null": false
}
}
}
}
Please let me know if you have any ideas about why this may happen, thanks!Hamza Naanani
10/04/2022, 2:21 PMJustin Trautmann
10/04/2022, 3:15 PMCrash detected! Request to <https://api.prefect.cloud/api/accounts/[...]/workspaces/[...]/task_runs/> failed.
...
RuntimeError: The connection pool was closed while 65 HTTP requests/responses were still in-flight.
this is most likely not related to local network issues as it is reproducible across different networks. when using a local orion server instead, the flow succeeds without any issues and a couple of days ago, the flow ran successfully on prefect cloud.
I am submitting ~100 parallel tasks using the RayTaskRunner with a local cluster.
Any help is much appreciated.
python 3.8.10
prefect 2.4.5
prefect-ray 0.2.0.post2Sean Turner
10/04/2022, 3:30 PMkubernetesJob
on EKS
orion, agent, and prefect cli are all 2.4.5
When I submit a simple task I get the following error in the agent logs:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
Seems that for some reason the service account called agent
isn't being observed? system:anonymous
is being used instead. The role
and roleBinding
both match what is in the helm chart so permissions should be fine.
$ kgp agent-5fbdcf4bbb-zrrg7 -o yaml | grep serviceAccount
serviceAccount: agent
serviceAccountName: agent
I was not having these problems before upgrading from 2.4.0 -> 2.4.5 😞. I created a new kubernetesjob block for 2.4.5 to silence an error