The task is successful but I get this

terminal_state_handler

issue

Been strugging to use coiled + prefect to bring up gpu workers with pytorch + cuda. Has anyone successfully done this? I bought a pro coiled account, and put gpu_workers=1, installed cuda_toolkit =10.2 as part of my conda package and used the base container recommended in the docs

The docs at https://docs.prefect.io/orchestration/concepts/kv_store.html#ui say that KV store is limited to 10 keys on the free tier. What's the limit on the standard tier?

For some reason after migration from prefect server to prefect cloud all our tasks which raise signals.FAIL are marked as TimedOut after execution.

22 July 2021,11:22:45 MSK	prefect.sometask	ERROR	Our job is failed
22 July 2021,11:22:45 MSK	prefect.CloudTaskRunner	DEBUG	Task 'facebook_insights': Execution process closed, collecting result...
22 July 2021,11:22:45 MSK	prefect.CloudTaskRunner	DEBUG	Task 'facebook_insights': Handling state change from Running to TimedOut

The "Execution process closed, collecting result..." log is coming from

prefect.utilities.executors.run_with_multiprocess_timeout

. But i dont see any other logs from this functions (according to source code they exist there). There is indeed a timeout set for a task, like a 1 hour, but according to logs it finished in 20 minutes. How can i debug this? We are running latest prefect version with ECS agent

Is prefect cloud down right now? Can’t seem to get to our tenant

Hello! I am confused: my flow failed in a positive case. I have three tasks Parent, ChildSuccess, ChildFailed. Childs depend on Parent. ChildFailed with trigger rule any_failed. When Parent- Success and ChildSuccess - success my flow Failed - I don't understand why? I thought that ChaildFailed should be with state skipped but it failed.

Hi - I had a quick question regarding parameters. Is it possible for a parameter to be dynamic? I have a function within my flow that returns the current date minus 7 days, that is then used as a parameter called process_date. This parameter is set within my flow as such:

with *Flow*("dummy_flow") as flow:

    

process_date = *Parameter*(name="process_date", default=*date_minus_seven*())

The issue with this is the process_date parameter is set at flow registration, but isn’t updated each time the flow runs. For example, if I registered the flow today, the process_date parameter would be set as 2021-07-15, but when the flow runs tomorrow, the process_date would still be set to 2021-07-15.

Hi all, Is there a way to completely nuke a Prefect account? Our account is in a bad state - Nothing will run. Previous suggestions have not worked. There are a bunch of tasks stuck in the

Queued

state and I can't change them at all (the

set_task_run_states

mutation claims

"SUCCESS"

for each of the task runs, but the same task run IDs are still returned when I look for

Queued

task runs). If I could just delete everything, my deployment scripts would be able to recreate the flows, projects, etc. and everything would finally work.

Hi! Silly question: I have a somewhat flaky task that fails sometimes, so I wrapped the function in

@task(max_retries=10, retry_delay=timedelta(minutes=5))

. But the failure is “silent”, in that it’s wrapped by a lower-level library. I know how to check whether it’s failed, so how should I tell Prefect that the task failed (and retry?) — is it enough to

raise prefect.engine.signals.FAIL

? Or does it need to be an explicit

raise ValueError

or something?

Hey folks 🙂 We have two agents, one used for testing and one for production. The testing agent is labelled "test", the production agent is labelled "production". We have a flow that we want to run on both environments, so we set both "test" and "production" labels on it. However, from Prefect docs I see that the agent labels must be a superset of the flow labels, otherwise the flow won't be picked up to run... What options do we have to run the same flow on two different agents? We are on Prefect 0.13.13, but it seems that this requirement has not changed in newer versions... Any suggestions are much appreciated!

Hi. Is there a way to mount a volume without using job template in

KubernetesRun

? I don’t want to change the existing job template but just override volume.

Hello, when trying to register or run flows on a certain machine, I always get an

UnicodeEncodeError

and the script quits (although after passing the command to Cloud). I've updated all packages, running on Win10 with Python 3.9. It doesn't happen on my local machine. Any ideas what may cause this?

Does anyone have any experience / guidance as to the wisdom of running Prefect flows as part of a GitHub Actions CI/CD pipeline? My org's machine learning team would like to start setting up some GitHub workflows that might run some potentially complicated DAGs – some of the nodes /tasks would need to able query our data warehouse for some sample data and run a machine learning model against it. They want these workflows to run on opening a PR, and to trigger a PR check failure in the event that some step of the DAG fails. This workflow sounds to me like it might be suited to using prefect's GraphQL API for launching and monitoring the status of such a flow, but I'm not sure if that would be trying to use Prefect for something it might not really be meant for. Bringing Prefect into the picture sounds like it could be more overhead than necessary, but I think it might make our lives easiest in the end. Any thoughts on this would be much appreciated!

👋 Happy Friday! Does the Prefect Cloud GraphQL API have any rate limits? I am writing a Ruby client for the API and want to make sure its well behaved and knows what to do if it runs into a rate limit.

Hello! Does anyone know the conda syntax for installing the optional dependencies? I’d like to install from conda using an

environment.yml

file the equivalent of this:

pip install "prefect[extra_1, extra_2]"

Does Prefect support multiple targets for a single task? Trying to output multiple attributes from an object as csv rather than pickling the object itself and want to require all of them to be present to skip the task

Hi everybody! I need to persisting information in a Task's run method, but according to prefect documentation I should not do this. Is there a way to keep a list in memory and update its content every time the task is run?

Has anyone ever deployed a k8s agent with a helm chart as opposed to cli command?

Hi, I'm confused about using agent locally. I started a prefect server with "prefect server start" I created a project with "prefect create project "myproject"" Then, I ran a local agent with "prefect agent local start -l mylabel" Then I run a dummy workflow with flow.register(project_name="myproject") flow.run_agent() When I go to http://localhost:8080, I can not see my flow. It looks like it is create on Cloud...

Secrets on VM Prefect server on the cloud the secrets are secure in Vault when using a VM Prefect server, or even when using the cloud, but do not want company service credentials to be stored in Prefect cloud. what are possible solution for secrets management? using ~/.prefect/config.toml might not be ideal, since the secrets are not encrypted this way

Some cool stuff in 15.2! The

--no-schedule

argument would make it a lot easier to set up separate CI/CD pipelines for main vs feature branches (would rather not have the schedule run for prototypes).

Hi how does one pass in a list parameter to the prefect cli? I tried

--param overrides='["exp=test_exp", "dataset=test_MS", "pairs_file=test_MS_chr22.tsv"]'

and the variants but it complained about the format

Hi, question about log schema when using GraphQL API. Each log entry used to have "info" field with additional information like function name and filepath, but in 0.15 it seems to be always null. Can this be re-enabled somehow in configuration?

Hi, I'd like to discuss parametrizing flows. At my job we have a single 'flow type' let's say (the same sequence of tasks defined in Python) for a specific feature. This flow type has different parameters per customer, leading to a list of Flows in Prefect that have the same Python functions but different parameter values (and different schedules usually). We want to add a new Flow automatically when a new set of customer parameters is added to our backend, but without having access to the

Flow

object in Python. So I was thinking I'd query an existing flow (possibly a special 'template' one), modify it and insert it again, but it feels kind of hacky. Any thoughts?

Hello all, I am currently trialling Prefect with respect to a current deployment that uses Airflow for a data pipeline job from MySQL (on RDS) to BigQuery. I am assuming that there are no direct replacements for specific operators such as MySQLToGCS? If I code up such an operator, is there an obvious place for me to land the code back into the community? Thanks for any responses, appreciated! 😉

wanted to bump this https://github.com/PrefectHQ/prefect/discussions/4672, as i noticed there is another discussion on the same topic that didn’t get much traction. Really isn’t much code and would certainly be a nice to have for many.

hey can i enforce a complete flow retry when task fails instead of only retrying from the task that failed?

Hi! Question about caching - here all tasks have finished, but only one has "duration". Does it mean that all the task that finished, are in one of the successful state, but don't have "duration" were read from cache?

Hi! No rush on this, but let’s say we have a function task that’s been retried twice already, and it succeeds on the third retry. Is there any way for us to “access” that

nth_retry

data? Does prefect internally maintain that state that we can access? Or in

prefect.context

? Thank you!