Hi, what’s the best way to switch certain flow variables depending on execution environment? e.g. a database schema of dev vs prod? (I am using set_run_config method at the moment to switch execution between local and ECS)

How so I terminate an unknown local agent?

Hi! Regarding templating, I copied and pasted the code here into a new flow and it still seems to name the task “compute”, rather than “hello” Am I doing something wrong?

Hi - I just happened to inspect my agent pod logs on k8s - I’m seeing a ton of errors with the following traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'

I found a thread discussing this - but I wanted to know if there was another way to alleviate this other than upgrading to prefect 1.0+ - If that is the only solution, are there any gotchas/things I should look out for when upgrading? And will flows as old as 0.14.X still run as expected?

We are running into an issue with an ECS agent where the logs from individual flow tasks aren't appearing in Prefect Cloud. Any advice for how to troubleshoot this issue? We're at a total loss.

Hi all - I have some odd behavior happening for a flow deployed to prefect cloud This might be hard to explain, but I will put most of the details in a thread

I just started at a company using perfect, but it's a version from before service accounts. I want to bring it up to match the docs and guides, but I am failing to understand the need for more than one service account with agent runs. What is the benefit to having individual service accounts for each agent instead of one service account with one key per agent?

Hi all, can you tell me which version of hasura prefect cloud is running on? 2.1.1? I’m asking because I’d like to put hasura graphql API behind my own hasura instance as remote schema and experiencing schema missmatches on 2.6.1

Hi, the page about v2 mentions “Event-Based Workflows” and “Trigger flows from external events…“. I read the API Reference but I couldn’t find how to do it. My use case is precisely this, but I don’t know if Prefect supports it. I need external systems to send events to persisted workflows and for these to continue after that. Can someone please point to the API Reference?

Airflow, for instance, have Sensors. Sensors wait for something external to happen.

Something I am struggling to get my head around is security in regards to

FargateCluster

. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass

"fargate_private_ip": True

to

cluster_kwargs

my cluster fails to start with the error:

Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s

That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic through

I have a question with the

IntervalClock

scheduler. If i were to pass

interval=timedelta(hours=1)

to schedule a run for every hour, would this be scheduled to occur at the beginning of every hour? Or would it be relative to when I register the Flow? eg if I register the flow at 1:13pm then the

IntervalClock

would trigger at 13 minutes past the hour every hour? Or would it just kick off at the first minute of every hour?

I'm curious how people tend to document their Prefect flows. Do people primarily fall back on Python module-level documentation (and build a static website using something like Sphinx), or is there a way to display documentation of a flow in the Prefect UI?

Investigating S3 storage a bit deeper. If we register Flow with S3 and 'local_script_path' set, Prefect copies dataflow to S3 named with specific timestamp; if we register new version, then new file is created. In this way we achieve dataflow versioning. If we use only 'key' option of S3 storage, then we can keep our own names of files, but have to copy dataflow manually. Registering new version of dataflow still increases version in UI, but in fact you have the same dataflow file on S3. So versioning doesn't actually work, right?

Hi, I’ve switched to the git storage class from gitlab and I’m getting an error when using the attached setup. The error is: {‘repo’: [‘Field may not be null.’]} According to the docs, repo is optional when git_clone_url_secret_name is set

Hi, Is there any specific usage when working with dict() returned by task? below part is failing with message that 'get_connections' task class has no method get().

token = EnvVarSecret("TOKEN")
connections = get_connections(token=token)  # this is a task that returns dict(name, connection_string)
start_task_result = start_task(connections.get('some_key'))  # i'd expect to get value from dict, not attribute error

Is anyone else having issues logging onto prefect cloud? I keep getting “unable to log in”

Hi all 👋 prefect duck Working on having our Flows automatically re-registered when there are changes with `1.1.0`'s

--watch

flag. However, as we’re spinning it all up with docker-compose, and we have multiple Prefect Projects, so I want to run a few of these

prefect register

commands in the container’s entrypoint shell script (code example in 🧵) Is there an easy way to: • either run the

prefect register

command in background, so it lets the rest run • or, run

prefect register

with specifying the project in the Flow script, so we’d need to run the command only once for all projects

Hi, I’m getting time to time 404 not found nginx/1.21.0 error when I try to access prefect server UI. My setup is in EKS with helm. It comes and goes but I noticed that it happens usually after helm update. Any idea what’s causing this?

Does anyone have a good explanation of the permissions the Prefect Slack app needs? Our security-conscious admins were not happy that the app needed the "identify" permission, other apps that posts to slack is OK with only the "incoming-webhook" permission. Any help would be appreciated, as I need to document the decision to approve the Prefect app.

Is it possible to set the final state of the flow to the state of one of the intermediate tasks?

👋 hi! have a question about a particular case where a flow gets stuck running forever in prefect 2.0, example in thread

If I have a complex schedule with an hourly interval clock and a cron clock - will there be 2 runs 1 of the clocks take precedence for an overlap, or will there be 2 scheduled runs?

Hey all, has anybody successfully started a dask cluster + prefect on top of an existing spark cluster (databricks)? I have a databricks cluster but want to use distributed prefect to schedule

whats the correct pattern for spinning our own cloud runner? we simply want to add some initialization to

CloudFlowRunner

so that we can initialize sentry and datadog as early as possible, as well as flush our statsd buffer to track metrics. it seems like the place i'd expect it to read from the env var is hardcoded tho https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/agent.py#L562-L586

Hi, does Prefect Cloud support multi-factor authentication? I’m looking through the features table at https://cloud.prefect.io/plans, and through my admin account profile page, and I don’t see a MFA option listed.

I am getting this error while executing a flow via Cloud Prefect 2.0 "RuntimeError: This portal is not running". Any clue what is going wrong. The same flow was working fine before I use Cloud Prefect 2.0. Actually it finishes all my flows and tasks successfully and at the end it crashes and it happens to all my existing flows. line 1512, in propose_state raise prefect.exceptions.Abort(response.details.reason) prefect.exceptions.Abort: This run has already terminated.

is there a way to find which tasks are holding the task tag concurrency locks? we’ve a case where our lock is taken up by a task when no flows are running

Fun with dataframes and map. So I have a map function that loads a dataframe in one task and then uses that dataframe in the next task. When I look at the return type on the first task, it is pandas.core.frame.DataFrame. When I look at the type of the dataframe value in the second task, it is class 'list'. I have the type hint set to pandas.DataFrame in the second task. Does anyone have any idea why the DataFrame is becoming a list?

Our prefect agent has been running without issue on ECS for the past ~11 months (since June 28 2021), but about 1 hour ago we noticed we were unable to submit new flows. Upon further investigation it appears our agent crashed and has been trying to restart but keeps crashing. I’m not sure if this is what initially caused it to crash but the error message we are seeing in cloudwatch logs now is :

ValueError: You have not set an API key for authentication.

Is anyone familiar with this error? Was something updated in prefect that could have caused this crash?