ie, I set my think to run every 3 minutes, but then 1 run winds up taking longer than 3 minutes. Do I have to worry about this at all or will Prefect just handle it?

Hello all, I'm currently working as Data Engineer and I'm intrested in Prefect as the main workflow management system in my team. Prefect's pros are numerous but user management is only available in the Cloud version. In my case, some dataflows must run inhouse and without user management is really difficult to include Prefect in our stack. I'm wondering if Prefect Team have considered adding Team Settings in the UI of Prefect Core. Thanks in advance

Hey guys, has anyone come across this when executing multiple concurrent Fargate Tasks [trigger 10 isntances of the same flow]

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.

Appears to be amazon complaining about the job family name, but I don’t know if there’s a way around it in the agent config.

Hi all, Is there any best-practice for deploying flows through CI/CD? I have seen the https://docs.prefect.io/orchestration/recipes/multi_flow_storage.html but how would one go about it with maybe 20+ flows? Should you simply create a main script that imports all the flows and registrer as part of the CI/CD pipeline? I get that one of the ideas with Prefect is that it is easy to create prototype flows quickly but it would be nice to have a flow suite version controlled and "one truth"

Hello 🙂 I have for all of my tasks

checkpoint=False

but still data is stored in the results folder. Do you know of any other place I need to check/set something?

Hi all, Documentation says that safe value from task run result is exposed in Cloud UI. Is this true for core as well?

Hello, I am testing prefect setup using KIND. Basically there is no difference between kind and kubernetes. I am facing to connect to graphql api. Have anybody had similar problem?

Hey everyone! The Prefect Server survey is still open 👉 https://forms.gle/WzjMYx3JtpLngghq8

Your responses help make Prefect even better. Let us know what you think!

Hi folks. In my flow file, I have some local lib imports like this:

from lib.package1 import fn1
from lib.package2 import fn2

These correspond to files alongside my flow like:

lib/package1.py

and

lib/package2.py

. When calling

flow.register()

it looks like my Docker Storage healthcheck fails because it cannot find the lib module...

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 135, in <module>
    flows = cloudpickle_deserialization_check(flow_file_path)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
ModuleNotFoundError: No module named 'lib'

Any tips on how to make that work?

hi! I'm just reading about prefect and have a basic question. Can you fan out kubernetes pods in a dynamic dag with prefect? E.g. given an

env_var

with 10 parameters, fan out to 10 pods when an initial task has finished. Basically I mainly want to use prefect as a plain orchestrator of pods like airflow pod operator. I see in the docs it documents a similar use case with Dask, but couldn't see an example which did similar with kubernetes pods.

Hey guys, having a problem using the

prefecthq/prefect:master

in flows - it has a feature I need [DFE execution on mapped tasks], but I can’t seem to figure out how to actually use master. I’ve updated both my Prefect Server and the image I base all our flows off [

FROM prefecthq/prefect:master

] but even with all that I still see the following when I build the flow storage:

Step 3/9 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.11.5#egg=prefect[kubernetes]>

And in the UI it still reports 0.11.5 as Prefect Core Version, and the expected DFE mapping behaviour is not there, i.e. mapped tasks are executed one by one. .Any ideas on how to get around this?

Is there any architecture diagram for prefect core. It can be really helpful when self hosting core server and tell how various components in system interact with each other. Currently I am facing issues where individual components stops working/responding due to various reasons (Majority of them being triggered by system resource constraints). The architecture diagram will help understand if it is possible to selectively scale individual components.

Hi everyone, I've just joined following a promoted tweet. I code primarily in Python. Looking forward to tinkering with the product and I'll take a look at the code. You can catch my dev story here.

Hey everyone! The Prefect Server survey is still open 👉 https://forms.gle/WzjMYx3JtpLngghq8 Your responses help make Prefect even better. Let us know what you think!

Hey team, Prefect core version 

0.12.0

 has been released! Here are a few notable highlights since `0.11.0`: 🗺️   Depth-first execution for mapping 🎈   Lightweight Dask serialization for lower memory footprint 🌡️   API health check tile on server UI 💎   Configure volumes in server for persistent orchestration state 📑   Task signature tab completion 🔎   New Great Expectations task library task 🛠️   Lots of enhancements and fixes Read the full changelog here: https://github.com/PrefectHQ/prefect/releases/tag/0.12.0 Special thanks to all of the contributors who have assisted in discussions, issues, and PRs that have contributed to this release!

Hello folks. Trying to get a Prefect Cloud Flow running in GKE. Just got this error in StackDriver Logs after registering the flow, and manually triggering it from Cloud UI:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 986, in get_task_run_state
    result = self.result.write(value, filename="output", **prefect.context)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 73, in write
    self.gcs_bucket.blob(new.location).upload_from_string(binary_data)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 35, in gcs_bucket
    from prefect.utilities.gcp import get_storage_client
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/gcp.py", line 6, in <module>
    from google.cloud import bigquery, storage
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)

It seems the process cannot find a prefect core bigquery module inside of google.cloud. I thought perhaps it had something to do with setting

PYTHONPATH

to my own application & lib directories in my Flow's Dockerfile:

ENV PYTHONPATH="/app:/app/lib"

And now Prefect core cannot find it's own packages? But experimenting with the alternative "extension" of

PYTHONPATH

instead of overwrite:

ENV PYTHONPATH="/app:/app/lib:${PYTHONPATH}"

simply yields a

PYTHONPATH

with a colon on the end (

/app:/app/lib:

), so I opted not to extend, since there is no default. My suspicions about

PYTHONPATH

could very well be a red herring... Any ideas?

Hi, we're using and evaluating prefect and are loving it. Our question is: How to handle continuous deployments with prefect? We're using gitlab pipelines to deploy code and register prefect flows any time we push in the master branch. Can we tell prefect which python environment and which version of our custom code to use for each flow? So that we don't break running flows during a deployment? Thanks in advance!

Hey, guys! Not sure if understand it. I'm sure that prefect can work with loops. But I don't understand how prefect can help me organize my current pipeline. I work with time series forecasting and for some problems I need to iteratively predict future timestamps. Like on the pic. Can I make prefect be able to have that loop and have my feature generator functions and model as separate tasks?

Hello, due to security I need to run all prefect containers as non-root. Are there any not superuser images versions?

Is there a way to access a flow’s start time (or any other distinct-per-flow timestamp) from within a flow?

Or any other kind of one-per-flow identifier?

Hello! 👋 I'm evaluating prefect server for a new project and I'm curious - what is the suggested way to deploy

prefect server start

where its not running in the foreground? I was expecting to find a

-d

option (similar to docker-compose). I'm sure this has been asked before - just looking to be pointed in the right direction! Thanks! 🙇

Just curious. After updating to the most recent version of prefect

0.12.0

, all tasks on

dask

/

bokeh

are labeled as "run_task" instead of the function name. Did the

0.12.0

update make it so I need to explicitly attach a name to each task?

bokeh==2.0.2

dask==2.18.1

distributed==2.18.0

Hello! I'm trying to get Prefect going on an on-prem server and I'm having some trouble getting the API to connect (it never connects, just keeps spinning) I assume I'm doing something wrong in my config.toml?

hey all - is there a way to use a python environment generated by conda with prefect? I’m using the docker agent, and it performs health checks by checking pip

Hi there! Our task concurrency limits feature was recently enabled on prefect cloud. I am just doing some testing and getting a UI error as I attempt to create a new tag. Is there something I am missing here for getting this to work?

Hello from Chicago! I'm hoping someone can help me. I'm super excited about the new support for non-containerized storage combined with

KubernetesAgent

(https://github.com/PrefectHQ/prefect/pull/2796). I'm struggling with something and hoping someone can help. I have the following setup: 1. Flow code uses

KubernetesJobEnvironment

+

S3

storage, using

flow.register()

to register flows with Prefect Cloud 2. Running a

KubernetesAgent

I can see that the agent is successfully communicating with Prefect Cloud. When I run a flow from the Prefect Cloud UI, I can see it in the agent's logs and I see a Kubernetes job created. That job is now failing with this error

[2020-06-18 194454] INFO - prefect.S3 | Downloading hello-flow-w-s3-k8s-env/2020-06-18t19-18-00-707752-00-00 from prefect-d94f436a-25b1-1699546c3

... big stacktrace ...

botocore.exceptions.NoCredentialsError: Unable to locate credentials

My flow code is just the hello world example that prints to the

prefect

logger, so that error must come from the

prefect

code that is trying to pull the flow from storage in S3. In the manifest for the

KubernetesAgent

, I've set up the environment based on the directions in https://docs.prefect.io/core/concepts/secrets.html#default-secrets

env:
- name: PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS
  value: '{"ACCESS_KEY": "REDACTED", "SECRET_ACCESS_KEY": "REDACTED"}'

I expected that setting this on the agent would mean that every job it creates has access to those credentials to download the flow. What am I doing wrong? Happy to provide more context in thread.

Does the prefect server persist it's data on shutdown by default? This is running prefect with a server as the backend