https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    matta

    06/15/2020, 11:06 PM
    How is scheduling repeated tasks handled if they wind up kinda overlapping?
  • m

    matta

    06/15/2020, 11:07 PM
    Like I'm looking to use Prefect to do some Airtable automation. So I'd want it to run, say, every few minutes. But the Airtable API rate-limits to 5 record per second - for a big enough Flow, I might bump into this.
  • m

    matta

    06/15/2020, 11:08 PM
    ie, I set my think to run every 3 minutes, but then 1 run winds up taking longer than 3 minutes. Do I have to worry about this at all or will Prefect just handle it?
    j
    5 replies · 2 participants
  • p

    psimakis

    06/16/2020, 9:45 AM
    Hello all, I'm currently working as Data Engineer and I'm intrested in Prefect as the main workflow management system in my team. Prefect's pros are numerous but user management is only available in the Cloud version. In my case, some dataflows must run inhouse and without user management is really difficult to include Prefect in our stack. I'm wondering if Prefect Team have considered adding Team Settings in the UI of Prefect Core. Thanks in advance
    z
    j
    3 replies · 3 participants
  • d

    Darragh

    06/16/2020, 1:20 PM
    Hey guys, has anyone come across this when executing multiple concurrent Fargate Tasks [trigger 10 isntances of the same flow]
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
    Appears to be amazon complaining about the job family name, but I don’t know if there’s a way around it in the agent config.
    z
    b
    +3
    104 replies · 6 participants
  • t

    Thomas Hoeck

    06/16/2020, 2:43 PM
    Hi all, Is there any best-practice for deploying flows through CI/CD? I have seen the https://docs.prefect.io/orchestration/recipes/multi_flow_storage.html but how would one go about it with maybe 20+ flows? Should you simply create a main script that imports all the flows and registrer as part of the CI/CD pipeline? I get that one of the ideas with Prefect is that it is easy to create prototype flows quickly but it would be nice to have a flow suite version controlled and "one truth"
    j
    z
    +4
    10 replies · 7 participants
  • m

    Matthias

    06/16/2020, 3:07 PM
    Hello 🙂 I have for all of my tasks
    checkpoint=False
    but still data is stored in the results folder. Do you know of any other place I need to check/set something?
    a
    z
    3 replies · 3 participants
  • s

    Sandeep Aggarwal

    06/16/2020, 3:59 PM
    Hi all, Documentation says that safe value from task run result is exposed in Cloud UI. Is this true for core as well?
    z
    2 replies · 2 participants
  • r

    Rafal

    06/16/2020, 5:32 PM
    Hello, I am testing prefect setup using KIND. Basically there is no difference between kind and kubernetes. I am facing to connect to graphql api. Have anybody had similar problem?
    n
    37 replies · 2 participants
  • d

    Dylan

    06/16/2020, 6:29 PM
    Hey everyone! The Prefect Server survey is still open 👉 https://forms.gle/WzjMYx3JtpLngghq8
    🚀 2
    :marvin: 3
  • d

    Dylan

    06/16/2020, 6:30 PM
    Your responses help make Prefect even better. Let us know what you think!
  • j

    jars

    06/17/2020, 3:23 AM
    Hi folks. In my flow file, I have some local lib imports like this:
    from lib.package1 import fn1
    from lib.package2 import fn2
    These correspond to files alongside my flow like:
    lib/package1.py
    and
    lib/package2.py
    . When calling
    flow.register()
    it looks like my Docker Storage healthcheck fails because it cannot find the lib module...
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 135, in <module>
        flows = cloudpickle_deserialization_check(flow_file_path)
      File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
        flows.append(cloudpickle.load(f))
    ModuleNotFoundError: No module named 'lib'
    Any tips on how to make that work?
    c
    a
    +3
    17 replies · 6 participants
  • e

    Eamon Keane

    06/17/2020, 10:10 AM
    hi! I'm just reading about prefect and have a basic question. Can you fan out kubernetes pods in a dynamic dag with prefect? E.g. given an
    env_var
    with 10 parameters, fan out to 10 pods when an initial task has finished. Basically I mainly want to use prefect as a plain orchestrator of pods like airflow pod operator. I see in the docs it documents a similar use case with Dask, but couldn't see an example which did similar with kubernetes pods.
    j
    2 replies · 2 participants
  • d

    Darragh

    06/17/2020, 11:46 AM
    Hey guys, having a problem using the
    prefecthq/prefect:master
    in flows - it has a feature I need [DFE execution on mapped tasks], but I can’t seem to figure out how to actually use master. I’ve updated both my Prefect Server and the image I base all our flows off [
    FROM prefecthq/prefect:master
    ] but even with all that I still see the following when I build the flow storage:
    Step 3/9 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@0.11.5#egg=prefect[kubernetes]>
    And in the UI it still reports 0.11.5 as Prefect Core Version, and the expected DFE mapping behaviour is not there, i.e. mapped tasks are executed one by one. .Any ideas on how to get around this?
    j
    10 replies · 2 participants
  • s

    Sandeep Aggarwal

    06/17/2020, 12:28 PM
    Is there any architecture diagram for prefect core. It can be really helpful when self hosting core server and tell how various components in system interact with each other. Currently I am facing issues where individual components stops working/responding due to various reasons (Majority of them being triggered by system resource constraints). The architecture diagram will help understand if it is possible to selectively scale individual components.
    j
    a
    +2
    11 replies · 5 participants
  • s

    Stanley Ndagi

    06/17/2020, 12:49 PM
    Hi everyone, I've just joined following a promoted tweet. I code primarily in Python. Looking forward to tinkering with the product and I'll take a look at the code. You can catch my dev story here.
    👋 6
    j
    2 replies · 2 participants
  • d

    Dylan

    06/17/2020, 1:43 PM
    Hey everyone! The Prefect Server survey is still open 👉 https://forms.gle/WzjMYx3JtpLngghq8 Your responses help make Prefect even better. Let us know what you think!
    👍 1
    :marvin: 1
  • j

    josh

    06/17/2020, 5:54 PM
    Hey team, Prefect core version 
    0.12.0
     has been released! Here are a few notable highlights since `0.11.0`: 🗺️   Depth-first execution for mapping 🎈   Lightweight Dask serialization for lower memory footprint 🌡️   API health check tile on server UI 💎   Configure volumes in server for persistent orchestration state 📑   Task signature tab completion 🔎   New Great Expectations task library task 🛠️   Lots of enhancements and fixes Read the full changelog here: https://github.com/PrefectHQ/prefect/releases/tag/0.12.0 Special thanks to all of the contributors who have assisted in discussions, issues, and PRs that have contributed to this release!
    🚀 14
    💯 11
    🎉 12
    :marvin: 11
    🤘 9
    j
    2 replies · 2 participants
  • j

    jars

    06/18/2020, 3:02 AM
    Hello folks. Trying to get a Prefect Cloud Flow running in GKE. Just got this error in StackDriver Logs after registering the flow, and manually triggering it from Cloud UI:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 986, in get_task_run_state
        result = self.result.write(value, filename="output", **prefect.context)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 73, in write
        self.gcs_bucket.blob(new.location).upload_from_string(binary_data)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 35, in gcs_bucket
        from prefect.utilities.gcp import get_storage_client
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/gcp.py", line 6, in <module>
        from google.cloud import bigquery, storage
    ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)
    It seems the process cannot find a prefect core bigquery module inside of google.cloud. I thought perhaps it had something to do with setting
    PYTHONPATH
    to my own application & lib directories in my Flow's Dockerfile:
    ENV PYTHONPATH="/app:/app/lib"
    And now Prefect core cannot find it's own packages? But experimenting with the alternative "extension" of
    PYTHONPATH
    instead of overwrite:
    ENV PYTHONPATH="/app:/app/lib:${PYTHONPATH}"
    simply yields a
    PYTHONPATH
    with a colon on the end (
    /app:/app/lib:
    ), so I opted not to extend, since there is no default. My suspicions about
    PYTHONPATH
    could very well be a red herring... Any ideas?
    c
    15 replies · 2 participants
  • e

    Emmanuel Klinger

    06/18/2020, 8:05 AM
    Hi, we're using and evaluating prefect and are loving it. Our question is: How to handle continuous deployments with prefect? We're using gitlab pipelines to deploy code and register prefect flows any time we push in the master branch. Can we tell prefect which python environment and which version of our custom code to use for each flow? So that we don't break running flows during a deployment? Thanks in advance!
    👀 1
    l
    j
    +1
    7 replies · 4 participants
  • a

    Andrey Alekseev

    06/18/2020, 8:40 AM
    Hey, guys! Not sure if understand it. I'm sure that prefect can work with loops. But I don't understand how prefect can help me organize my current pipeline. I work with time series forecasting and for some problems I need to iteratively predict future timestamps. Like on the pic. Can I make prefect be able to have that loop and have my feature generator functions and model as separate tasks?
    e
    j
    7 replies · 3 participants
  • r

    Rafal

    06/18/2020, 9:15 AM
    Hello, due to security I need to run all prefect containers as non-root. Are there any not superuser images versions?
    j
    4 replies · 2 participants
  • h

    Howard Cornwell

    06/18/2020, 2:06 PM
    Is there a way to access a flow’s start time (or any other distinct-per-flow timestamp) from within a flow?
    👀 1
    l
    6 replies · 2 participants
  • h

    Howard Cornwell

    06/18/2020, 2:06 PM
    Or any other kind of one-per-flow identifier?
  • w

    wilsojb

    06/18/2020, 2:09 PM
    Hello! 👋 I'm evaluating prefect server for a new project and I'm curious - what is the suggested way to deploy
    prefect server start
    where its not running in the foreground? I was expecting to find a
    -d
    option (similar to docker-compose). I'm sure this has been asked before - just looking to be pointed in the right direction! Thanks! 🙇
    👀 3
    l
    p
    3 replies · 3 participants
  • j

    Jackson Maxfield Brown

    06/18/2020, 4:13 PM
    Just curious. After updating to the most recent version of prefect
    0.12.0
    , all tasks on
    dask
    /
    bokeh
    are labeled as "run_task" instead of the function name. Did the
    0.12.0
    update make it so I need to explicitly attach a name to each task?
    bokeh==2.0.2
    dask==2.18.1
    distributed==2.18.0
    👀 1
    d
    m
    +2
    18 replies · 5 participants
  • l

    Lewis Bails

    06/18/2020, 5:11 PM
    Hello! I'm trying to get Prefect going on an on-prem server and I'm having some trouble getting the API to connect (it never connects, just keeps spinning) I assume I'm doing something wrong in my config.toml?
    n
    j
    +1
    48 replies · 4 participants
  • k

    Kevin Weiler

    06/18/2020, 5:58 PM
    hey all - is there a way to use a python environment generated by conda with prefect? I’m using the docker agent, and it performs health checks by checking pip
    👀 2
    d
    4 replies · 2 participants
  • c

    Chris Vrooman

    06/18/2020, 6:39 PM
    Hi there! Our task concurrency limits feature was recently enabled on prefect cloud. I am just doing some testing and getting a UI error as I attempt to create a new tag. Is there something I am missing here for getting this to work?
  • c

    Chris Vrooman

    06/18/2020, 6:39 PM
    d
    8 replies · 2 participants
Powered by Linen
Title
c

Chris Vrooman

06/18/2020, 6:39 PM
d

Dylan

06/18/2020, 6:41 PM
Hi @Chris Vrooman, Thanks for letting us know! We’re aware of this issue, you should see a fix in shortly.
c

Chris Vrooman

06/18/2020, 6:43 PM
Gotcha, thanks Dylan!
d

Dylan

06/18/2020, 6:45 PM
In the meantime, this is a great chance to check out the Interactive API! You can create a new task concurrency limit with:
mutation {
  update_task_concurrency_limit(input: {name: "your_tag_name", limit: 5}) {
    id
  }
}
Then you should see it on the page you referenced above ☝️
@Chris Vrooman if you hard refresh the page and try again it should be working
👍 1
c

Chris Vrooman

06/18/2020, 8:44 PM
Looks like it is working now, thank you Dylan! I didn’t realize this could also be done through the API. That may be a better way for us to implement 👍
d

Dylan

06/18/2020, 9:23 PM
No problem! Your Task Concurrency limits are global to your Prefect Cloud Team, so they’ll work for all Task Runs with Tasks with a given Tag
That being said, all of the GraphQL mutations that the UI uses are available to you! 😄
View count: 1