Hi, I want to set up a Prefect agent in AWS ECS. I've set my AWS cred in

~/.aws/config

and did

prefect auth login

using my API Token. Then I'm running the

prefect agent ecs start

command, expecting an Agent to be created in Fargate and linked to my Cloud, but I'm getting the following error:

ERROR - agent | Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
Traceback (most recent call last):
...
ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`

What am I doing wrong?

Hi, does anyone know whether I can integrate Prefect on-prem, as part of my product, without paying for a subscription?

or is it mandatory to pay per use?

And another unrelated question. Can anyone please explain the typical production integration with Prefect? Concretely speaking, I have a web MERN (MongoDB, Express.js, React, Node.js) stack, and I would like to attach/integrate my JavaScript Node.js code of REST APIs (e.g. POST /employees) with an existing Prefect DAG.

Not sure about the API and the integration protocols

Hi I am starting to randomly see this on flow runs;

12:09:39 ERROR prefect-server.ZombieKiller.TaskRun No heartbeat detected from the remote task; marking the run as failed.
12:22:53 INFO prefect-server.Lazarus.FlowRun Rescheduled by a Lazarus process. This is attempt 1.

I am running on GKE autopilot - any ideas how to debug cause of such events and mitigate against their occurence?

Hi, I'm trying to understand what happens if the agent running a flow is interrupted (more specifically if there is a way to gracefully stop it). From a simple experiment of a 60 seconds long flow, that I interrupt the agent running it, the runs remain in the UI even after re-launching the agent:

Hi There, I'm having a problem with task logging using cloud and dask cluster, no logs are being streamed to the cloud except errors and "Starting task run ..." / "Finished task run for task with final state: ....." messages and my logging statements is not showing. keep in mind that I'm using mapped tasks

Hi:) I am running a flow on seclude and would like to set the flow run name as a template something like: run_my_flow_{time.now} Is there is a way to do that? I want to avoid the auto generated name Thanks:)

Hi, how can you delete a storage configuration with the cli utility in prefect 2.0?

If a want to run a large backfill on k8s using the framework defined here https://discourse.prefect.io/t/i-want-to-use-parameter-values-for-backfilling-but-prefe[…]ter-defaults-provided-at-registration-how-to-solve-that/49 - is there a way to batch things up so I am only ever running say 10 jobs at a time?

Hey, 1. What is the best practice for running Tasks (not flows) over Kubernetes ? I think about using DaskExecuter and deploying Dask over Kubernetes cluster. Just want to make sure if that is the correct way 🙂 2. What is the best practice for managing CI/CD of flows? I wish to deploy and run branch specific flows and delete them upon merge Thanks!

Hey, I have a Secret AWS_CREDENTIALS in Prefect Cloud (1.0) following this format:

{
  "ACCESS_KEY": "abcdef",
  "SECRET_ACCESS_KEY": "ghijklmn"
}

If I'm using pandas to read a file in S3:

df = pd.read_csv(f"s3://{s3_bucket_name}/{filename}")

Should I need to pass the credentials as a param to the function read_csv? Or are they read automatically from the Cloud? Currently I'm getting this error: "botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden" Thanks!

Hi, When I run a task locally using

flow.run()

, it works just fine. However, I’m running a local backend server and when I try to register the task using

flow.register()

, I get this error:

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
  File "/Users/shriramholla/.pyenvs/python3.6/lib/python3.6/site-packages/absl/flags/_flagvalues.py", line 677, in __getstate__
    raise TypeError("can't pickle FlagValues")
TypeError: can't pickle FlagValues

Note: Nothing is being passed into or being returned by any task

Hi! I would like to ask about a case with multiple return values from a Task. When I specify the flow edge manually by using

flow.add_edge(_upstream_task_, _downstream_task_, _*key*_)

is there any way to specify which of the multiple return values from the upstream task should pass to the downstream task?

Hello! I’m trying to figure out why my local runs differ from prefect cloud. I expect that my task will handle exception, and it perfectly works local, but in cloud my flow fails with strange error message “‘traceback’ object is not subscriptable”. How can I fix this?

Hey guys, what is the best way to copy a flow? Just use .copy() or deepcopy?

This message was deleted.

Hi #prefect-community, does anyone have any experience running prefect agent in Windows Server?

How do I link two ShellTasks so that they run sequentially?

Prefect 2.0 extra loggers does not seem to work when used with custom libraries

Hi everyone! I'm facing issues with prefect logger, as I'm using ThreadPoolExecutor, I want to ensure that future is completed successfully or at least know the reason of failure. The flow and task with parallel execution looks like this. The deal is that when running locally, I'm able to see the logs, but couldn't see them on Prefect UI, though I'm able to explore other logs outside this task. I'm using

prefect.context.get("logger")

as logger. Could anyone guess how to fix this? Thanks

Hi everyone. This morning I encountered problem when running prefect workflow. the log say : RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS. This never happened before, I try to update kubectl at the worker cluster and not helping. I'm using dask on kubernetes for executor. any idea?

Hi i have multiple agents running on Prefect. Incase any agent goes down (turns yellow/red) can we get slack alert to get notified?

Hi Everyone. I have deployed a prefect server with a Kubernetes agent using the Helm chart in a EKS cluster. Now I'm kind of confused how to register the flow? Where is the Python code supposed to be in Kubernetes?

Hi, how can I stop a running flow-run either with the web ui or the cli?

hi community! is there a way to look-up a flow's id using the flow name/project name via the client? this suggests you can via the cli

Hi , What’s best practice to get the value of a prefect Parameter inside a flow?

Hi :-) StartFlowRun() with map() not creating unique runs. I have a wrapper flow, that creates sub-flow runs:

flow_run = StartFlowRun(project_name=PREFECT_PROJECT, flow_name=SUB_FLOW)

, that I execute with

flow_run.map(parameters=parameters, run_name=run_name)

. (Here

parameters

and

run_name

are generated by other

map()

tasks, and of same length.) When I run the wrapper task locally, targetting the sub-task in Prefect cloud, I get:

└── 12:48:22 | DEBUG   | Flow Run 7951f47c-1ddf-4e33-b913-a2698a89e4c9 created.
└── 12:48:22 | DEBUG   | Creating link artifact for Flow Run 7951f47c-1ddf-4e33-b913-a2698a89e4c9.
└── 12:48:23 | INFO    | Flow Run: <https://cloud.prefect.io/.../flow-run/7951f47c-1ddf-4e33-b913-a2698a89e4c9>
...
└── 12:48:23 | DEBUG   | Flow Run 7951f47c-1ddf-4e33-b913-a2698a89e4c9 created.
└── 12:48:23 | DEBUG   | Creating link artifact for Flow Run 7951f47c-1ddf-4e33-b913-a2698a89e4c9.
└── 12:48:24 | INFO    | Flow Run: <https://cloud.prefect.io/.../flow-run/7951f47c-1ddf-4e33-b913-a2698a89e4c9>
...

Note the same GUIDs!!! When we visit the cloud UI, only the first mapped task is created! If the wrapper task is also running in cloud, it works as expected. I'm not sure how to create a minimal reproducible example for this, since

StartFlowRun

can't trigger a local flow (I don't use prefect server, only prefect cloud).