Alvaro Durán Tovar
05/18/2022, 1:22 PMBen Collier
05/18/2022, 1:56 PMJake
05/18/2022, 2:37 PMflow.run_config = KubernetesRun(job_template_path="<s3://bucket/path/to/spec.yaml>")
But I’m not sure how to write that template / how this template is used (are there any docs on this?). Thanks!iñigo
05/18/2022, 2:42 PMJessica Smith
05/18/2022, 2:42 PMTim Enders
05/18/2022, 2:53 PMGSCResult
?Alexandru Anghel
05/18/2022, 4:34 PMRuntimeError: Tasks cannot be called from within tasks. Did you mean to call this task in a flow?
I'm not sure what i'm missing here as i run the tasks inside the flow. Could you please assist me?
Thanks! Awesome project, by the way!Xavier Witdouck
05/18/2022, 4:35 PMXavier Witdouck
05/18/2022, 4:35 PMraceback (most recent call last):
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/Users/witdxav/d3x/projects/d3x-quanthub-sdk/src/d3x/quanthub/prefect/prefect_tasks.py", line 9, in docker_run
barra_license = Secret("BARRA_LICENSE").get()
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/client/secrets.py", line 170, in get
raise ValueError(
ValueError: Local Secret "BARRA_LICENSE" was not found.
Xavier Witdouck
05/18/2022, 4:35 PMXavier Witdouck
05/18/2022, 4:36 PMXavier Witdouck
05/18/2022, 4:36 PMJacqueline Riley Garrahan
05/18/2022, 5:51 PMMike Vanbuskirk
05/18/2022, 5:55 PMMike Vanbuskirk
05/18/2022, 5:55 PMJohn O'Farrell
05/18/2022, 6:14 PMprefect.Client
api to run the task and it threw an error Failed to load and execute flow run: TypeError("'ABCMeta' object is not subscriptable")
. Now any subsequent attempt to run the flow using the prefect ui throws the same error, even though the code hasn't changed.Jessica Smith
05/18/2022, 6:25 PMAttributeError: 'Edge' object has no attribute 'upstream_task'
I'm getting this when dask tries to use pickle to load the first of my mapped tasks. The first two tasks of the flow are not mapped, and they don't have issues. Any thoughts?Darin Douglass
05/18/2022, 6:56 PMcreate_flow_run
not base the new flow’s name off of the current flow without passing run_name
?Jessica Smith
05/18/2022, 7:30 PMWilliam Jamir
05/18/2022, 9:04 PMwith Flow(...) as flow:
x = Parameter(...)
...
Which is registered with:
prefect register -p flows/my_file.py --project "MyProject"
But now I dont want to have this flow as a global variable, because my tests need to do some settings before and I dont want to trigger any of the code within this context manager.
I can easily move this to a function, and make it return the flow instance, but I dont know how to deal with the registration by command line.
I mean, its possible to still use the register by command line, or do I need to make it now programmatically?
Basically, I’m looking for a solution like this:
def main():
with Flow(...) as flow:
x = Parameter(...)
...
return flow
Registering
prefect register -p flows/my_file.py:main --project "MyProject"
Danny Vilela
05/18/2022, 9:30 PMenable_read
and enable_write
and just check those within the read
and write
methods, but maybe there’s something easier?Shaoyi Zhang
05/18/2022, 10:02 PMKen Nguyen
05/18/2022, 10:31 PMTypeError: Task is not iterable. If your task returns multiple results, pass `nout` to the task decorator/constructor, or provide a `Tuple` return-type annotation to your task.
Pasha Kravtsov
05/18/2022, 10:43 PMpaimoe
05/19/2022, 1:24 AMprefect backend server
in ENV variables? since it's complaining about no API keyEddie Atkinson
05/19/2022, 3:43 AMcluster
which determines whether a cluster is used which is accessed from a callback to DaskCluster
. However, it’s not actually used in the flow which is causing flow run failures locally because it’s an ‘unexpected parameter’. I am 90% sure I’ve seen a way before where I can say to Prefect to relax and that it’s fine (technical terminology), but can’t remember exactly howRaviraja Ganta
05/19/2022, 5:45 AMTypeError: cannot pickle 'google.protobuf.pyext._message.MessageDescriptor' object
Guillaume Latour
05/19/2022, 8:20 AMdistributed.protocol.pickle - INFO - Failed to serialize <Success: "Task run succeeded.">. Exception: cannot pickle 'lxml.etree.XMLSchema' object
which leads to
distributed.worker - ERROR - failed during get data with <ip> -> <ip>
which at some point close the connexion
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://<ip> remote=tcp://<ip>>: Stream is closed
this is managed by prefect by some retries (depending on configuration) and finally the agent raises an error and the flow is marked as failling
distributed.scheduler.KilledWorker: ('concatenate_df_based_on_time-b91c06dc30f54c5084e9f5fe8b6b32a5', <WorkerState 'tcp://<ip>', status: closed, memory: 0, processing: 1>)
Do you have an idea on how to prevent this kind of error?Jan Domanski
05/19/2022, 9:11 AMTom Manterfield
05/19/2022, 10:50 AM