Hey folks, is there a way to add metadata tags/labels to Flows/Flow Groups? Adding a Label to a Flow Group has the added feature of helping decide which Agent to run on, but I'm looking for something that's purely informational. The goal is to be able to query GraphQL for flows with specific tags and do something with that list of IDs. For example, if we want to pause all jobs for Client X, but not Client Y or Z (and there's nothing in the flow naming convention to indicate), we want to be able to just run a query for all jobs with tag "Client-X" and then run a mutation to pause/delay those flows. Or maybe we want to track slow flows -- so we query for flows running 50% slower than their past monthly average runtime and add a "slow" tag so we can make some changes. Does this exist? Is there a place in a flow for arbitrary information? Maybe a "tags" parameter that just never gets used?

My prefect agent was unreachable for several hours and a bunch of jobs got queued up behind it. When the agent restarted, it ran a bunch of duplicate jobs in a row. They’re idempotent so I think they’re okay, but I’m worried I’ll introduce a bug that depends on being the only job of its kind running at the same time, and in this kind of situation something bad could happen. Can I specify that a

Schedule

try and run, and if it fails expire the job after 60 seconds of trying?

Hey prefect people, a

resource manager

related question: I am trying to setup a resource manager to do some larger-than-memory computations within a task. I wanted to start simple, but even the example gives the following error for me:

multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

So, one more question for the day, may not have an easy solution for Prefect atm from what i've read... What is the best way to deal with needing a persistent database connection between tasks, because those tasks are doing work within a transaction? I have some Talend jobs which currently deal with that just fine, but I simply cannot see a similar way to do it in Prefect.

The flow run names (e.g.

daffy-wolf

) already look very docker-name like It seems like it'd be pretty slick if the

DockerAgent

then also used the same name for launching containerized flows. Not strictly needed, and maybe only relevant for our setup where our log aggregator shows the container name that the logs came from

Hi all, is there a good way of deleting flows. Trying to delete a flow gives a popup error:

We could not delete your flow. Please try again. If this problem continues, contact <mailto:help@prefect.io|help@prefect.io>

Hi all, Im experiencing the dread “white dashboard screen” once again (and have upgraded to v0.14.17). In the chrome extension browser I see:

Uncaught (in promise) AuthSdkError: PKCE requires a modern browser with encryption support running in a secure context.
The current page is not being served with HTTPS protocol. PKCE requires secure HTTPS protocol.

Maybe this is causing issues for the dashboard (even though I can see a flows runs without issue if I navigate to the url). Currently the IP address of the server is only accessible within a VPN so we have been using http, and it used to be fine. Is there a way of saying “Dont care about this?” or a way of attaching a certificate to a given service. Very similar to this old bug here: https://github.com/PrefectHQ/ui/issues/588 Did some env vars not propagate correctly again? Confused as to why this used to work, but a redeploy from the prefect images no longer does so

Hi all, does anyone know of any condition that should cause a flow to become totally stuck? This diagram seems to indicate that all the tasks upstream from this one are done, so I'd expect that this one (or another one with no pending upstream tasks) should start running. Currently there is only one flow run, and all tasks are either completed or pending. The flow run has been idle for 8 hours. Running Prefect 0.14.2.

Has anyone seen this message?

TypeError: ‘ContextualVersionConflict’ object is not iterable

I have a flow that I am trying to call

run()

on to run it locally.

Resurrecting https://prefect-community.slack.com/archives/CL09KU1K7/p1616759310466600 When I have the following in my requirements.in file:

boto3
dask-cloudprovider[aws] == 2021.3.1
prefect[aws] == 0.14.17

Trying to run pip-compile on it fails with a complaint about botocore versioning:

botocore<1.20.50,>=1.20.49 (from aiobotocore==1.3.0->dask-cloudprovider[aws]==2021.3.1->-r requirements/production.in (line 14))
  botocore<1.21.0,>=1.20.67 (from boto3==1.17.67->-r requirements/production.in (line 9))
  botocore<2.0a.0,>=1.12.36 (from s3transfer==0.4.2->boto3==1.17.67->-r requirements/production.in (line 9))

This boils down to: botocore ==1.20.49,>=1.20.67 which cannot be satisfied. What is a safe version of boto3 that works with prefect and dask cloud provider? Removing the version numbers above from the requirements.in does not help, same complaints. Which is to be expected since pip is going to try the newest it can without guidance.

is there a way to give docker tasks a name?

Agent Issue. I'm just trying prefect out - using the backend server and running on an EC2 instance m5.large with 8080 and 4200 ports allowed external.  I have the server running, the UI attached, the graphql connected, and I started an agent on the same machine. For me I installed prefect using a conda activation - so to start the agent - i did the conda activation then prefect agent start command. HOWEVER, when I add a flow and run it - it fails on

unexpected error: NameError("name 'prefect' is not defined

This appears to me that it is not running in the condra activation and yes if thats true then the pip install will not be there. Please advise how I can set the python environment of my local agent.

What's the canonical way to provide ENV vars to DaskExecutor Schedulers/Workers? I'm assuming it's going to be in either

cluster_kwargs

or

client_kwargs

on https://docs.prefect.io/api/latest/executors.html#daskexecutor ?

Would there be interest in a

FilterTask

variant that had an extra parameter along the lines of:

log_filtered (Callable, optional): a function to use for logging any result that did not pass `filter_func`.

Use would be something like:

valid_configurations = FilterTask(filter_func=some_validator, log_filtered=log_invalid_configuration)

where

def log_invalid_configuration(configuration):
    # log the reasons this configuration is invalid. Or some other special logging for invalid configurations.
    # Like, notifying a team to fix the configuration.

Adding this to

FilterTask

would be straightforward. Turn the list comprehension info a for loop appending to

filtered

and call

log_filtered

whenever a result does not match the predicate. I am currently making 2 calls to FilterTask, once to collect valid and once to collect invalid. Which is not pretty.

Hello P, I have a hard time understanding the depth-first-execution of mapped tasks with Dask executor. From my experiments it seems like the execution is bfe, but maybe I'm missing something trivial like task trigger or something? More info with code example in 🧵

Hi all, I have some newbie Prefect deployment questions if anyone can help: 1. What is considered best practice for automating flow registration, docker builds, etc. when deploying new code? Is it preferable to simply lean on CI/CD tools for this, or do people use things like the docker BuildImage task as part of a multi-flow setup? I want to use something like GitHub for storage but locally-built docker images for the run config, but am not sure how to close the gap on automating builds from local Dockerfiles. 2. Is the only way to define non-python docker image dependencies for a flow simply to write a Dockerfile?

Hi everyone, I'm interested in using prefect to continuously replicate a bigquery table into a postgres database. Is this possible to do within prefect?

Is it possible to retry mapped tasks? Whenever I try it I get nulls being passed as the mapped input parameter. We are using prefect cloud with prefect results.

Hi everyone, I am trying to push a markdown file as an artifact which contains some js in order to render a chart, but the chart disappeared in prefect UI. Actually is it possible to render scripted markdown in prefect? here are the markdown file I am trying to pushed and the flow definition. Thanks

Hi everyone, I am trying to run prefect agent in Detached mode using supervisor as per prefect documentation, even though i am able to spawn up the prefect agent, the additional params passed is not read. I have attached the CLI command that i use to run the agent and it works as expected but at the same time when i run it using supervisor, it doesn't attach the labels or get the API for the agent to connect to. If there is any other way to run prefect agent in detached mode or if there is an error in the way i am using is could you please let me know. Thanks in advance

Hey, I’m wondering if there’s a supported way for the prefect agent to run jobs using a specific kubernetes nodeselector? For example, a flow labeled

high-memory

runs on a

high-memory

agent which spawns jobs via a

high-memory

nodeselector.

Anyone ever have issues using the docker side car recipe? I can’t get the docker side car to authenticate to docker hub. We are using the same registry credentials we use in job containers, and that works not problem. Thinking the docker daemon inside the sidecar needs to login also somehow? Not sure though.

Hey all. We’re trying to move to github storage for our flows, but don’t use Prefect secrets. Is there a way we could set the Github access token to make this work?

Hey all, I am trying to get the parameter values of a flow run inside my state handler, is there a way to fetch the parameter values ?

Hello, I am trying to run the tutorial for prefect cloud

In [1]: import prefect
   ...: from prefect import task, Flow
   ...:
   ...: @task
   ...: def say_hello():
   ...:     logger = prefect.context.get("logger")
   ...:     <http://logger.info|logger.info>("Hello, Cloud!")
   ...:
   ...: with Flow("hello-flow") as flow:
   ...:     say_hello()
   ...:
   ...: # Register the flow under the "tutorial" project
   ...: flow.register(project_name="tutorial")

but get

ClientError: [{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}]

Hi everyone, I’m having issues using jupyter tasks with prefect core. When I try to run a jupyter task it returns

ImportError: Using `prefect.tasks.jupyter` requires Prefect to be installed with the "jupyter" extra.

and when I try installing it by running

pip3 install prefect[jupyter]

it returns

no matches found: prefect[jupyter]

My current workaround is to use Prefect API and to add that as dependency during flow registration

python_dependencies = ["prefect[jupyter]"]
flow.storage = Docker(...
        python_dependencies=python_dependencies,
        ...)

Is there a way to pip install the jupyter things needed to run a jupyter task using Prefect Core only?

Hi everyone - I'm currently evaluating using Prefect/Dask for our ETL engine, running through our various use-cases - reading through the open-source docs I'm wondering if there is a simple way to trigger execution of a task from the outside, say, from an AWS lambda function written in Javascript...any ideas?

I am running into some issues deploying Docker Containers that I am not quite sure how to debug. I am using bitbucket as my flow storage and Docker as my run environment. I try to run the flow through my Docker Agent and it successfully pulls down the docker image and then does nothing else for a long time until it seems to fail silently and a Lazarus process restarts it. Any ideas on where to start looking to figure out the issue? The logs aren't any help and I can't tell why it is failing.

hello everyone, how do i flatten/join a task of tasks? let’s say i already have tasks a,b,c defined but i would like to group them as a task, perhaps because i want a conditional flow: for one branch i want to execute a,b,c in order, otherwise i want to execute d,e,f how do i go about this? without redefining a task which does what a,b,c/d,e,f already do?

I frequently can't load cloud.prefect.io. I get blinking dots forever, Doing a hard reload doesn't help. Has anyone else run into this?