Mike Marinaccio
10/21/2020, 9:42 PMprefect.context.parameters
. Am I correct to say that I can always rely on the context params being set before any other task is run in an async / Dask environment?Jimmy Le
10/21/2020, 10:20 PMTypeError: cannot pickle '_thread.lock' object
.
Has anyone run into a similar problem? Any suggestions would be appreciated!Zach
10/21/2020, 11:35 PMZach
10/21/2020, 11:36 PMPrefectResult()
result handler doesn't work since a class instance is not JSON serializableZach
10/21/2020, 11:37 PMbral
10/22/2020, 3:29 AMAlberto de Santos
10/22/2020, 2:14 PMnohup
, however, in order to do that, I have to execute everything with the flow.run_agent()
option and then comment that line of code to execute the flows.
The alternative is through CLI, however, I couldn’t make it work (it doesn’t file the libraries I need to execute, despite of using the -p
param)
Which is your experience/view here?Dennis Schneidermann
10/22/2020, 2:31 PMJames Phoenix
10/22/2020, 3:55 PMJames Phoenix
10/22/2020, 3:55 PMJames Phoenix
10/22/2020, 3:56 PMJames Phoenix
10/22/2020, 3:56 PMJames Phoenix
10/22/2020, 3:56 PMNewskooler
10/22/2020, 5:29 PMdef test_smth():
with Flow("test") as flow:
outcome = my_funct()
state = flow.run()
assert state.result[outcome].is_failed()
ale
10/22/2020, 5:59 PMBilly McMonagle
10/22/2020, 6:02 PMFargateAgent
. I am running this agent locally, and it seems to work intermittently. However, I am repeatedly getting this error:
[2020-10-22 17:59:48,212] INFO - agent | Starting FargateAgent with labels ['XXX']
[2020-10-22 17:59:48,212] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-10-22 17:59:48,212] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-10-22 17:59:48,269] INFO - agent | Waiting for flow runs...
[2020-10-22 17:59:48,402] ERROR - agent | [{'path': ['get_runs_in_queue'], 'message': "'NoneType' object has no attribute 'flow_group_id'", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Mitchell Bregman
10/22/2020, 7:54 PM# $ROOT/src/flow.py
from prefect import Flow
flow = Flow(name="test flow")
Now from the command line in dir $ROOT
, I run:
prefect register flow --file src/flow.py --name "test flow"
I get the error
KeyError: "'__name__' not in globals"
What am I doing wrong? I’d like to use the CLI as it will be more simplistic from a CI/CD standpointfabian wolfmann
10/22/2020, 9:40 PMAlberto de Santos
10/22/2020, 10:06 PMJesper van Dijke
10/22/2020, 11:50 PMauth0-spa-js must run on a secure origin
Which of course is completely ok because not running on localhost but rather on an ubuntu image and connect to <http://192.168.88.133:8080>
in chrome address bar <chrome://flags/#unsafely-treat-insecure-origin-as-secure>
added this domain.
No luck, next error : Refused to frame '<https://login.prefect.io/>' because an ancestor violates the following Content Security Policy directive: "frame-ancestors 'none'".
Anyone solved or run it on a different machine?
Alternatively I end up running SSH and tunnel to it...
Putty, ssh tunnel forward 8080 and 4200, and all is good.
Maybe a nice addendum to the documentation.Scott Asher
10/23/2020, 2:54 AMprefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
Scott Asher
10/23/2020, 2:54 AMScott Asher
10/23/2020, 3:13 AMflow.register()
Alberto de Santos
10/23/2020, 9:29 AMAlberto de Santos
10/23/2020, 9:30 AMmap
over a DataFrame
, do you have any idea? I see very clear how to make it over a list
. To my mind, it seems straightforward to convert the DataFrame
into a list
. So, I would like to know your opinion.Zach
10/23/2020, 2:55 PMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': "can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 111, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1177, in set_task_run_state
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 226, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': "can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
I ran this flow 89 times with different input, and one time it failed with this message.Zach
10/23/2020, 3:46 PMMitchell Bregman
10/23/2020, 6:09 PMModuleNotFoundError: No module named src
during the flow healthcheck, traceback here. Am I doing something wrong in terms of __init__
packaging? This is a followup to thread yesterday.Joseph Haaga
10/23/2020, 7:39 PMflow
only defined inside that contexthandler/`with` statement?
from prefect import task, Flow
@task
def say_hello():
print("Hello, world!")
with Flow("Run Me") as flow:
h = say_hello()
flow.run() # prints "Hello, world!"
I’m getting the following
NameError: name 'flow' is not defined
Alexander
10/23/2020, 7:54 PMUsing executor type DaskExecutor
(i use local one).
import time
from prefect import task, Flow
from prefect.engine.executors import DaskExecutor
from prefect.environments import LocalEnvironment
@task
def wait_for(seconds):
time.sleep(seconds)
with Flow('_Concurrency_') as flow:
head = wait_for(5)
leaf1 = wait_for(30)
leaf2 = wait_for(15)
tail = wait_for(3)
head.set_dependencies(downstream_tasks=[leaf1, leaf2])
leaf1.set_downstream(tail)
leaf2.set_downstream(tail)
flow.environment = LocalEnvironment(executor=DaskExecutor())
flow.executor = DaskExecutor()
I expect leaf1 and leaf2 to run in parallel but they are not. In gantt chart i see they running sequentially. If i run them locally, they run in parallel.
BTW whats the difference between environment executor and flow executor?
Flows are run by docker agent.