https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • t

    Tim Wright

    04/29/2022, 1:43 PM
    I have a question related to Prefect Cloud. We have our jobs running in AKS - with our Flows using a custom Image. Our flows are all built into the image using Docker Storage. We've been trying to mitigate the possibility of adverse effects if we were somehow to leak or Prefect API keys. To that end we've locked down our cluster a bit with a security policy to disallow any flows trying to run on images that we have not authored (e.g. in our registry). Is there any possibility that someone (with Prefect Cloud API Keys) could register a flow that uses our image, but an alternate flow storage mechanism? We were considering subclassing the FlowRunner class to handle some checking/validation there - validating some of the metadata set at registration time against some of the metadata provided at Runtime. Is this possible, does it make sense? Is there a way we can ensure that our Custom FlowRunner is the default (and only) FlowRunner permitted? To be more concise our goal is to ensure that the only flows running in our infra (AKS) run with our images (this we have solved), and only the flows that are built into those images can be run (this we have not solved).
    h
    a
    7 replies · 3 participants
  • w

    William Durksen

    04/29/2022, 2:19 PM
    👋 Hello, team! I just signed on and hope to find answers for accessing task results.
    👋 3
    c
    a
    6 replies · 3 participants
  • c

    Constantino Schillebeeckx

    04/29/2022, 2:24 PM
    When I used
    FilterTask
    with my own filter func, does the task still filter out the default
    NoResult
    None
    and exceptions?
    a
    14 replies · 2 participants
  • f

    Florian Guily

    04/29/2022, 2:58 PM
    Hey, does anyone in the community did incremental etl with Prefect ? We are considering doing it and maybe there are mistakes that can be avoided ^^ Thanks !
    n
    a
    +1
    10 replies · 4 participants
  • g

    Greg Wyne

    04/29/2022, 5:38 PM
    Hi there! Another question. We recently did an upgrade to our prefect instance, and post upgrade some logs are no longer getting pulled in from the running task. We use a very lightweight wrapper around the DbtShellTask: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask And we have the log_stdout=True variable set which according this this should let the stdout show up: https://docs.prefect.io/core/idioms/logging.html#logging-stdout I’ve ssh’d into the running task and verified that its emitting the stdout we are expecting, is there a step I should be looking at somewhere? Thank you!
    a
    1 reply · 2 participants
  • e

    ek

    04/29/2022, 5:39 PM
    Is there a way to use Prefect Client to get a latest flow_id within a project?
    a
    6 replies · 2 participants
  • m

    Madison Schott

    04/29/2022, 5:42 PM
    Does anyone have Slack alerts set up with dbt tests on Prefect? Is it possible to send an alert with dbt source tests that are warning or failed?
    a
    50 replies · 2 participants
  • j

    Jonathan Seery

    04/29/2022, 8:08 PM
    hey folks - any mods out there who could help me change the email associated with this workspace? i’m losing access to this email here
    a
    3 replies · 2 participants
  • g

    Geoffrey Keating

    04/29/2022, 8:55 PM
    Is there a way to force a task to rerun even if it's cached?
    k
    1 reply · 2 participants
  • w

    Walter Cavinaw

    04/29/2022, 9:34 PM
    hi. I recently came across prefect and it looks amazing. I am trying it out with a few of our jobs, but I'm stuck on something! (Sorry for the simple question!) When registering a flow with Bitbucket storage, how do I ensure that shared functions (from a project utilities module) are available to the flow when it runs on an agent (in this case a local agent)? In the project folder flows are in a flows folder, utilities in another folder, etc. I guess the flow is not being run from within the repo project directory because it can't find the utils and libs modules?
    k
    9 replies · 2 participants
  • t

    Tim Wright

    04/30/2022, 12:24 AM
    Is there any way to specify a custom FlowRunner (which has subclassed Prefect's FlowRunner) as the default FlowRunner with PrefectCloud? I would like to enforce some checks on the Flow Run configuration/settings for all flows. I was thinking of subclassing the FlowRunner and adding some validation in its
    __init__
    method
    k
    1 reply · 2 participants
  • r

    Raviraja Ganta

    04/30/2022, 11:10 AM
    I have logger class which i am using for my code. But the logs printed using that is not coming in prefect UI. Only the ones used with
    prefect.context.get("logger")
    is coming in the UI. How to surface my custom logger method logs also into the UI?
    k
    5 replies · 2 participants
  • r

    Raviraja Ganta

    04/30/2022, 8:09 PM
    I have the code structure as follows
    main-project
      | 
      src
         |
         | __init__.py
         | module1.py
         | module2.py
      |
      configs
         | 
         | config.yaml
      |
      flows
         |
         | sample_flow.py
    module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2. • What is the best way to package the code? • What is the way to run agent that uses GPU on EC2?
    a
    b
    11 replies · 3 participants
  • z

    Zach Schumacher

    05/01/2022, 3:26 PM
    Is there an easy way to rerun all the flows in a project for a cloud backend?
    a
    5 replies · 2 participants
  • c

    Carlos Paiva

    05/01/2022, 5:49 PM
    Hi all, I am currently getting a `urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)`on my EC2 Prefect Server setup. I stumbled upon this: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L64 Although it seems a Cloud configuration. Can someone please point me out on the right direction?
    a
    k
    39 replies · 3 participants
  • c

    Cole Murray

    05/01/2022, 9:52 PM
    Hi All 👋, Relatively new to Prefect and looking into it for running my team's data operations. We have previous experience with Celery for running our distributed tasks, and are looking to upgrade to either Airflow or Prefect. With Airflow, we found several undesirable aspects & complexity which has caused us to investigate Prefect. Our workload: • 1000s of tasks • Lowest granularity is every 5 mins • Primarily python-based code • Homogenous dependencies, but may change in the future as we move to more ML based workflows. I have read through the prefect docs, and have some questions about how Prefect actually schedules tasks (in a non-prefect cloud context). From the docs, one criticism of Airflow is:
    • the centralized nature of the Airflow scheduler provides a single point of failure for the system
    In a typical master/worker deployment, you have a single-point-of-failure schedule (yes you can run it highly available with locking / other mechanisms) responsible for reading from a schedule DB, to invoke tasks into a queue to be processed by workers. Can someone clarify how Prefect server solves this issue? From the docs, it seems Prefect server is also a SPOF in this architecture. Based on code here: https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/scheduler.py#L21, we would not be able to run several instances of the server simultaneously, as there is no locking taking place against the DB, and would cause double execution
    a
    m
    7 replies · 3 participants
  • e

    Eddie Atkinson

    05/02/2022, 5:58 AM
    I am currently running my jobs on Fargate but have bumped into their (weirdly low) limits on the number of cores I can use. What would be the easiest upgrade pathway for running jobs provided I wanted to keep using serverless? My guess would be it’s time to switch to a
    DaskExecutor
    and then use a temporary cluster as described here: https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-local-cluster Just want to make sure I’m not missing a more obvious solution before I move to a cluster.
    a
    2 replies · 2 participants
  • e

    Efraim Globus

    05/02/2022, 8:44 AM
    Hi, I am trying to configure a flow that will run in a docker image and will be executed by a docker agent. The flow was configured successfully. I started a docker image from my machine and triggered a run via the ui in the prefect cloud. I got the following error message: “`run_config` of type
    DockerRun
    , only
    LocalRun
    is supported ” what does it mean?
    with Flow("iris_batch", storage=Local(path="/app/batch/flow.py",stored_as_script=True),
              run_config=DockerRun(image="test:latest")
              ) as batch_prediction_flow:
    What i am missing? Amy help will be appreciated:) Thank you!
    a
    5 replies · 2 participants
  • a

    Alexander van Eck

    05/02/2022, 11:21 AM
    Just out of curiosity, is there an official release date for prefect 2.0? 🙂 I’m aware of the public beta, but when is stable estimated to be released?
    a
    a
    16 replies · 3 participants
  • h

    Horatiu Bota

    05/02/2022, 11:35 AM
    hi community! i'm trying to deploy prefect server on EC2 but can't get past "Couldn't connect to Prefect Server at <address>:4200/graphql", however i can access
    <address>:4200/graphql
    in the browser - i've followed the troubleshooting guide, but no success
    a
    11 replies · 2 participants
  • n

    Naga Sravika Bodapati

    05/02/2022, 11:36 AM
    Hi all, we are seeing all our flows on prefect 1.0 fail with error : cannot allocate memory. Can u name a scenario/reason why this could happen? This happens often when the flows are long time running and there is no way for us to debug using the logs on prefect. Please help.
    a
    12 replies · 2 participants
  • r

    Ron Levi

    05/02/2022, 11:52 AM
    Hi, is there a chance that an ECS Flow runner will be added in Prefect 2.0? Thanks!!
    a
    3 replies · 2 participants
  • b

    Bob Colner

    05/02/2022, 12:24 PM
    I don’t see task
    state_handlers
    functionality in Orion/Prefect2.0. Is is on the roadmap? (or I’m I missing something..) thanks
    a
    5 replies · 2 participants
  • m

    Matthew Seligson

    05/02/2022, 12:41 PM
    @Kevin Kho great to see you on stage at PyCon!
    🙌 4
    🔥 3
    k
    3 replies · 2 participants
  • m

    Mars

    05/02/2022, 3:16 PM
    Hi! What’s the correct way to set the
    PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN
    variable in the k8s manifest created by
    prefect agent kubernetes install
    ? Do I have to use the
    --env
    switch or can I provide the value by editing the generated deployment container’s
    env:
    list directly?
    k
    a
    +1
    36 replies · 4 participants
  • d

    Daniel Tavares

    05/02/2022, 4:28 PM
    Hi! is there a possibility to manually retry a failed task on prefect 2.0 UI?
    a
    d
    +1
    9 replies · 4 participants
  • s

    Slackbot

    05/02/2022, 4:42 PM
    This message was deleted.
    j
    k
    3 replies · 3 participants
  • c

    Chris Reuter

    05/02/2022, 4:54 PM
    See you all this Wednesday on PrefectLive! https://prefect-community.slack.com/archives/C036FRC4KMW/p1651510465396669
    🚀 1
    :upvote: 3
    🔥 1
  • l

    Leon Kozlowski

    05/02/2022, 5:53 PM
    How long will a flow stay in Paused status while it waits for approval? And is the best way to decline just to cancell the flow run?
    k
    a
    13 replies · 3 participants
  • p

    Poosh Adhikary

    05/02/2022, 7:08 PM
    Hi everyone! I am trying to execute a flow on two different docker containers with different number of CPU cores. Is it possible to dynamically set
    num_workers
    for LocalDaskExecutor based on the number of cores available on the container?
    k
    6 replies · 2 participants
Powered by Linen
Title
p

Poosh Adhikary

05/02/2022, 7:08 PM
Hi everyone! I am trying to execute a flow on two different docker containers with different number of CPU cores. Is it possible to dynamically set
num_workers
for LocalDaskExecutor based on the number of cores available on the container?
k

Kevin Kho

05/02/2022, 7:09 PM
Are you using a task to start the Docker container? Or are you using subflows with
DockerRun
?
p

Poosh Adhikary

05/02/2022, 7:12 PM
I am using
DockerRun
.
k

Kevin Kho

05/02/2022, 7:12 PM
Oh I get what you are saying now. You have one flow but you have two execution environments with different resources. What happens though if you leave it blank?
LocalDaskExecutor()
should infer the number of cores
p

Poosh Adhikary

05/02/2022, 7:14 PM
I do not want to use all the cores. Instead, I was hoping to do something like
num_of_cores/2
.
k

Kevin Kho

05/02/2022, 7:15 PM
You can’t with
LocalDaskExecutor
, but I think you can with
DaskExecutor
using
LocalCluster
like this
p

Poosh Adhikary

05/02/2022, 7:19 PM
Gotcha. Would have been great if we could do the same with
LocalDaskExecutor
as well. Thank you Kevin!
View count: 9