Daniel Saxton
05/17/2022, 10:10 PMDavid Beck
05/17/2022, 10:20 PMdavzucky
05/18/2022, 1:30 AMJan Bršťák
05/18/2022, 8:24 AMprefecthq/prefect:1.2.1-python3.9
image, and we are using Prefect Cloud. All seems to be fine with the Agent, we can see it in the UI, and also in Cloudwatch logs, whenever we trigger a flow run, it says Deploying flow run…
and Completed deployment of flow run
. But no Flows are getting started, it gets stuck on Submitted every time (it worked before, but we changed image for agent from prefecthq/prefect:0.14.13-python3.8
and also roles in AWS which have the same permissions as original had). There are no errors, and nothing in logs. I’m not really sure what to do? Thanks for any help. Run config looks like this: RUN_CONFIG = ECSRun(
_labels_=["prod"],
_task_role_arn_="arn:aws:iam::XXX:role/prefectTaskRole",
_execution_role_arn_="arn:aws:iam::XXX:role/prefectTaskExecutionRole",
_task_definition_arn_="prefect-task:4",
_run_task_kwargs_=dict(
_cluster_="XXX",
),
)
Florian Guily
05/18/2022, 10:16 AMDaniel Sääf
05/18/2022, 12:32 PM>>> result = leonardo_dicaprio("leo")
However, when i try to create the deployment i get a:
ImportError: cannot import name 'hello_world' from 'test' (/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/test/__init__.py)
So basically, my question is: how do i import functions from local python-files in a flow?
(Let me know if this isn’t the right place for this type of questions!)yair friedman
05/18/2022, 12:44 PMKrzysztof Ciesielski
05/18/2022, 1:10 PMmax_retries
+ retry_delay
task parameters. I wanted to wrap some of my tasks with retries, and I thought it should be enough to add these parameters to the @task
annotation. However, when a task raises an exception, it shows in the Prefect Cloud UI that it's state is "Retrying Task (after attempt 1 of 61)", but after a short while the entire flow gets cancelled.
Example:
@task(log_stdout=True, max_retries=60, retry_delay=timedelta(seconds=10))
def perform_something(param1: Optional[str],
param2: any,
year_month_str: str,
environment_label: str) -> datetime:
Alvaro Durán Tovar
05/18/2022, 1:22 PMBen Collier
05/18/2022, 1:56 PMJake
05/18/2022, 2:37 PMflow.run_config = KubernetesRun(job_template_path="<s3://bucket/path/to/spec.yaml>")
But I’m not sure how to write that template / how this template is used (are there any docs on this?). Thanks!iñigo
05/18/2022, 2:42 PMJessica Smith
05/18/2022, 2:42 PMTim Enders
05/18/2022, 2:53 PMGSCResult
?Alexandru Anghel
05/18/2022, 4:34 PMRuntimeError: Tasks cannot be called from within tasks. Did you mean to call this task in a flow?
I'm not sure what i'm missing here as i run the tasks inside the flow. Could you please assist me?
Thanks! Awesome project, by the way!Xavier Witdouck
05/18/2022, 4:35 PMXavier Witdouck
05/18/2022, 4:35 PMraceback (most recent call last):
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/Users/witdxav/d3x/projects/d3x-quanthub-sdk/src/d3x/quanthub/prefect/prefect_tasks.py", line 9, in docker_run
barra_license = Secret("BARRA_LICENSE").get()
File "/Users/witdxav/opt/anaconda3/envs/v0.9.216/lib/python3.10/site-packages/prefect/client/secrets.py", line 170, in get
raise ValueError(
ValueError: Local Secret "BARRA_LICENSE" was not found.
Xavier Witdouck
05/18/2022, 4:35 PMXavier Witdouck
05/18/2022, 4:36 PMXavier Witdouck
05/18/2022, 4:36 PMJacqueline Riley Garrahan
05/18/2022, 5:51 PMMike Vanbuskirk
05/18/2022, 5:55 PMMike Vanbuskirk
05/18/2022, 5:55 PMJohn O'Farrell
05/18/2022, 6:14 PMprefect.Client
api to run the task and it threw an error Failed to load and execute flow run: TypeError("'ABCMeta' object is not subscriptable")
. Now any subsequent attempt to run the flow using the prefect ui throws the same error, even though the code hasn't changed.Jessica Smith
05/18/2022, 6:25 PMAttributeError: 'Edge' object has no attribute 'upstream_task'
I'm getting this when dask tries to use pickle to load the first of my mapped tasks. The first two tasks of the flow are not mapped, and they don't have issues. Any thoughts?Darin Douglass
05/18/2022, 6:56 PMcreate_flow_run
not base the new flow’s name off of the current flow without passing run_name
?Jessica Smith
05/18/2022, 7:30 PMWilliam Jamir
05/18/2022, 9:04 PMwith Flow(...) as flow:
x = Parameter(...)
...
Which is registered with:
prefect register -p flows/my_file.py --project "MyProject"
But now I dont want to have this flow as a global variable, because my tests need to do some settings before and I dont want to trigger any of the code within this context manager.
I can easily move this to a function, and make it return the flow instance, but I dont know how to deal with the registration by command line.
I mean, its possible to still use the register by command line, or do I need to make it now programmatically?
Basically, I’m looking for a solution like this:
def main():
with Flow(...) as flow:
x = Parameter(...)
...
return flow
Registering
prefect register -p flows/my_file.py:main --project "MyProject"
Danny Vilela
05/18/2022, 9:30 PMenable_read
and enable_write
and just check those within the read
and write
methods, but maybe there’s something easier?Shaoyi Zhang
05/18/2022, 10:02 PMShaoyi Zhang
05/18/2022, 10:02 PMKevin Kho
05/18/2022, 10:44 PMShaoyi Zhang
05/18/2022, 10:48 PMKevin Kho
05/18/2022, 10:54 PMShaoyi Zhang
05/18/2022, 10:54 PMKevin Kho
05/18/2022, 10:55 PM