While registering the flow it is not storing the configs file which is present in a different directory. But some modules needs them. How to solve this?

main-project
  | 
  src
     |
     | __init__.py
     | module1.py
     | module2.py
  |
  configs
     | 
     | config.yaml
  |
  flows
     |
     | sample_flow.py

Hi all, we are using

prefect server

for local development and

prefect cloud

for production which is quite neat in terms of speed. When doing this we set the --external-postgres flag in order to use our development database, which works great! However we are using Hasura for our other backend service which creates some git-pain as Prefect overwrites our local Hasura metadata files. Have anyone run into the same problem ?

Hi again, theoretical question. If most use Docker/K8s to run dataflow on (each dataflow has own image, or each flow executes common image), why do we need Agent at all?

Hey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinning up a new pod, running and deleting it. I think for some reason something got changed on the AWS account and the spinning up of pods failed. However, it kept trying to retry and spin up a new pod, and the old pod didn’t get cleaned up. Now whenever I manually try to delete the pod (kubectl delete pods) it just recreates instantly. So now i have like 6 pods that are stuck in Pending status that I can’t delete. Any thoughts on how to clean everything up?

How would you describe Prefect in a compact way that highlights its features? It's not a workflow manager in the sense that it is not Camunda, it is not Temporal, it is not AWS Step. It's not Conductor by Netflix

While

prefect_test_harness

provides an environment for flows to be tested, how might I go about testing a task in Prefect 2.0? It seems like calling

.run()

on tasks has not made it over from 1.0 yet.

I am struggling a little with Prefect programming patterns. Take this example from this doc: https://docs.prefect.io/core/advanced_tutorials/task-looping.html

with Flow("mapped-fibonacci") as mapped_flow:
    ms = Parameter("ms")
    fib_nums = compute_large_fibonacci.map(ms)

flow_state = mapped_flow.run(ms=[10, 100, 1000, 1500])

nice but what about non array parameters? Let's say I have two parameters, one that's an array, and the other which is a constant.

with Flow("mapped-fibonacci") as mapped_flow:
    ms = Parameter("ms")
    ms2 = Parameter("ms2")
    fib_nums = compute_large_fibonacci.map(ms, ms2)

flow_state = mapped_flow.run(ms=[10, 100, 1000, 1500])

Hi guys ! I’m trying to setup a

Prefect Agent

and a

prefect server

on my local computer, everything residing inside of

containers

. My

Prefect Server

`container(s)`is based on the

docker-compose

file retrieved from

prefect server config > server-docker-compose.yaml

. When spinning up a container for an Agent and running

prefect agent local start --api <http://host.docker.internal:4200>

, it says that I must be authenticated to communicate with the containerized server living in my local machine. My first question is: Is

authentication

mandatory in this kind of situation , for local run ? When I authenticate by creating a new tenant using

prefect server create-tenant -n default

outside the Agent container, I’m able to spin up the

Agent Container

and run everything. The problem is that I wanted to run the

create-tenant

inside the container too, but it’s not possible as I’m getting connection refused between containers.

Hi, what’s the best way to switch certain flow variables depending on execution environment? e.g. a database schema of dev vs prod? (I am using set_run_config method at the moment to switch execution between local and ECS)

How so I terminate an unknown local agent?

Hi! Regarding templating, I copied and pasted the code here into a new flow and it still seems to name the task “compute”, rather than “hello” Am I doing something wrong?

Hi - I just happened to inspect my agent pod logs on k8s - I’m seeing a ton of errors with the following traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'

I found a thread discussing this - but I wanted to know if there was another way to alleviate this other than upgrading to prefect 1.0+ - If that is the only solution, are there any gotchas/things I should look out for when upgrading? And will flows as old as 0.14.X still run as expected?

We are running into an issue with an ECS agent where the logs from individual flow tasks aren't appearing in Prefect Cloud. Any advice for how to troubleshoot this issue? We're at a total loss.

Hi all - I have some odd behavior happening for a flow deployed to prefect cloud This might be hard to explain, but I will put most of the details in a thread

I just started at a company using perfect, but it's a version from before service accounts. I want to bring it up to match the docs and guides, but I am failing to understand the need for more than one service account with agent runs. What is the benefit to having individual service accounts for each agent instead of one service account with one key per agent?

Hi all, can you tell me which version of hasura prefect cloud is running on? 2.1.1? I’m asking because I’d like to put hasura graphql API behind my own hasura instance as remote schema and experiencing schema missmatches on 2.6.1

Hi, the page about v2 mentions “Event-Based Workflows” and “Trigger flows from external events…“. I read the API Reference but I couldn’t find how to do it. My use case is precisely this, but I don’t know if Prefect supports it. I need external systems to send events to persisted workflows and for these to continue after that. Can someone please point to the API Reference?

Airflow, for instance, have Sensors. Sensors wait for something external to happen.

Something I am struggling to get my head around is security in regards to

FargateCluster

. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass

"fargate_private_ip": True

to

cluster_kwargs

my cluster fails to start with the error:

Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s

That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic through

I have a question with the

IntervalClock

scheduler. If i were to pass

interval=timedelta(hours=1)

to schedule a run for every hour, would this be scheduled to occur at the beginning of every hour? Or would it be relative to when I register the Flow? eg if I register the flow at 1:13pm then the

IntervalClock

would trigger at 13 minutes past the hour every hour? Or would it just kick off at the first minute of every hour?

I'm curious how people tend to document their Prefect flows. Do people primarily fall back on Python module-level documentation (and build a static website using something like Sphinx), or is there a way to display documentation of a flow in the Prefect UI?

Investigating S3 storage a bit deeper. If we register Flow with S3 and 'local_script_path' set, Prefect copies dataflow to S3 named with specific timestamp; if we register new version, then new file is created. In this way we achieve dataflow versioning. If we use only 'key' option of S3 storage, then we can keep our own names of files, but have to copy dataflow manually. Registering new version of dataflow still increases version in UI, but in fact you have the same dataflow file on S3. So versioning doesn't actually work, right?

Hi, I’ve switched to the git storage class from gitlab and I’m getting an error when using the attached setup. The error is: {‘repo’: [‘Field may not be null.’]} According to the docs, repo is optional when git_clone_url_secret_name is set

Hi, Is there any specific usage when working with dict() returned by task? below part is failing with message that 'get_connections' task class has no method get().

token = EnvVarSecret("TOKEN")
connections = get_connections(token=token)  # this is a task that returns dict(name, connection_string)
start_task_result = start_task(connections.get('some_key'))  # i'd expect to get value from dict, not attribute error

Is anyone else having issues logging onto prefect cloud? I keep getting “unable to log in”

Hi all 👋 😛refect-duck: Working on having our Flows automatically re-registered when there are changes with `1.1.0`'s

--watch

flag. However, as we’re spinning it all up with docker-compose, and we have multiple Prefect Projects, so I want to run a few of these

prefect register

commands in the container’s entrypoint shell script (code example in 🧵) Is there an easy way to: • either run the

prefect register

command in background, so it lets the rest run • or, run

prefect register

with specifying the project in the Flow script, so we’d need to run the command only once for all projects

Hi, I’m getting time to time 404 not found nginx/1.21.0 error when I try to access prefect server UI. My setup is in EKS with helm. It comes and goes but I noticed that it happens usually after helm update. Any idea what’s causing this?

Does anyone have a good explanation of the permissions the Prefect Slack app needs? Our security-conscious admins were not happy that the app needed the "identify" permission, other apps that posts to slack is OK with only the "incoming-webhook" permission. Any help would be appreciated, as I need to document the decision to approve the Prefect app.

Is it possible to set the final state of the flow to the state of one of the intermediate tasks?