https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • d

    David Elliott

    08/17/2022, 5:05 PM
    Hey folks, we’re seeing very strange flow registration behaviour when trying to register a long-standing existing flow to Prefect Cloud (1.0). My question is, is there a 2500 max task limit in Prefect Cloud on statically defined DAGs? We’ve been registering this flow for months (and it’s been growing) and we’ve just surpassed 2500 nodes, and it’s now throwing an error, so want to check if you have an internal limit defined somewhere…
    m
    z
    +1
    18 replies · 4 participants
  • j

    Josh Paulin

    08/17/2022, 5:48 PM
    Hello. Starting to look at migrating to Prefect 2.0, and I’m a little confused about how to structure storage blocks for flows. Is it best practice to create a single block for all flows, or to create one block per flow? I’m not seeing any way when building the deployment to specify a subfolder, which makes me think option 1 is kind of forced…
    👀 2
    o
    i
    +1
    10 replies · 4 participants
  • k

    Khuyen Tran

    08/17/2022, 7:12 PM
    Join us for PrefectLive this Wednesday, live on Twitch right now. @Taylor Curran will be showing us the cool stuff you can do with AWS Blocks! 😛arty-parrot:
    :twitch: 2
  • e

    emo loic

    08/17/2022, 7:47 PM
    Hello. I have a question. I want to deploy my flow in local subprocess from my VM hosted in AWS. The flow code is on my local computer with all the dependencies. When I execute the command "prefect deplyment buid ..." in my local computer with s3 as my block , should I normally start the agent on my VM in AWS. How to configure the agent so that before executing my flow on my VM, he will install the dependicies required. I read this article on how to deploy prefect 2 on AWS but it doesn't mentioned how the agant will execute the flow. Does the agent install dependicies before running the task inside the flow ? Here is the link https://discourse.prefect.io/t/how-to-deploy-prefect-2-0-flows-to-aws/1252
    k
    4 replies · 2 participants
  • t

    Tony Yun

    08/17/2022, 8:07 PM
    Hi, could anyone suggest how do we set up ssh keys in prefect kubernetes agent? we need to connect to a SFTP server that we can only let local development env work but not after register the flow.
    ✅ 1
    b
    3 replies · 2 participants
  • i

    Ilya Galperin

    08/17/2022, 8:31 PM
    What methods are folks using to send custom Kubernetes job templates to a
    kubernetes-job
    infrastructure block in Prefect 2.0? In 1.0, we could accomplish this by using the
    KubernetesRun
    module and passing it like so:
    KubernetesRun(job_template_path="my/local/path.yaml")
    It seems like we can try to mimic the functionality in 2.0 by using some hacks to pipe the output from our job template .yaml file, represent it as a one-line JSON string then force that output into
    --override job=
    but this doesn’t seem ideal.
  • r

    Ross Teach

    08/17/2022, 8:56 PM
    This work queue uses a deprecated tag-based approach to matching flow runs; it will continue to work but you can't modify it
    I noticed this warning on the Prefect Cloud UI. Is there any documentation on the recommended approach to use?
    o
    j
    3 replies · 3 participants
  • k

    Kathryn Klarich

    08/17/2022, 9:30 PM
    Hi all, I have some questions about prefect 2.0 versus 1.0. I use to use prefect 1.0 at a previous company, and am trying to get prefect up and running at a new company. I started down the road of prefect 2.0 but didn’t quite realize how much has changed, and am now wondering if I should stick with prefect 1.0. It looks like there may have been a useful discourse post about this topic (as reference here: https://docs-v1.prefect.io/), but that post no longer exists. Are there other resources on this? Specifically I am most worried about the new UI for prefect 2.0 as it doesn’t seem that useful for my use case - I don’t see the option to start flow runs, view and retry tasks and task graphs, etc.
    k
    a
    +3
    10 replies · 6 participants
  • a

    Aaron Goebel

    08/18/2022, 12:10 AM
    is there a way to use
    Parameters
    or something analogous as input to a
    KubernetesRun
    to request a specific amount of resources on the fly? I'm working w/ Fargate and want to give users flexibility to run flows with differing amounts of compute
    👀 1
    e
    7 replies · 2 participants
  • m

    Michael Levenson

    08/18/2022, 2:47 AM
    Hi all why can we not create service accounts in the ui for v1 - we are on the standard plan
    👀 1
  • m

    Michael Levenson

    08/18/2022, 3:00 AM
    im getting intermittent 404s on the service account page…
    👀 1
    e
    1 reply · 2 participants
  • a

    ash

    08/18/2022, 6:23 AM
    Hello everyone, I have a prefect 1.0 Server running on K8 cluster with Git as storage. Is it possible for me to run a flow on a remote server with all the scripts and dependencies being on remote server and only using prefect on K8 to track the progress of the flow run.
  • m

    Marcin Grzybowski

    08/18/2022, 7:32 AM
    Hi, bumping this one - maybe someone has an idea if I'm doing something wrong or if there really is a problem in ConcurrentTaskRunner/prefect-snowflake
  • r

    Rainer Schülke

    08/18/2022, 9:40 AM
    Hello 🙂 I have a problem with a newly created flow. It is a relative straightforward one and pretty easy but it only works locally (flow.run()) and not on the cloud 😞 The logs don't show any information, just reference task fail. The task is using a custom class for processing and loading the data like this:
    @task(log_stdout=True)
    def etl(
        connector,
        exec_file,
        environment: Optional[str] = None
    ):
        master_data = MasterDataSet(connector, exec_file)
        master_data.get_dataset(environment)
        master_data.load_dataset()
    My flow looks like this:
    .... as flow:
        source = SOME_SOURCE
        destination = SOME_DESTINATION
    
        file_list = ['SOME/PATH/TO/FILE', 'ANOTHER/PATH/TO/FILE']
    
        master_dataset = etl.map(
            connector=unmapped(source),
            exec_file=file_list
        )
    The cloud logs look like this:
    Task 'etl_master_data': Starting task run...
    Task 'etl_master_data': Finished task run for task with final state: 'Mapped'
    Flow run FAILED: some reference tasks failed.
    There is no failed reference task. Does anybody has any idea why this is not working on the cloud but locally? It is configured like all the other flows, even the data handling by class is already implemented for other flows. I really have absolutely no clue why 😄
    👀 1
    ✅ 1
    b
    5 replies · 2 participants
  • h

    Ha Pham

    08/18/2022, 10:09 AM
    (Prefect 2.0 question) my flow has a
    start_date
    parameter. In the flow code the default of this value is set like this
    def sync_circleci(start_date: datetime.date = datetime.datetime.today().date()):
       ...
    and it will come out in the YYYY-MM-DD format. Now when I deploy the flow, it seems that the default value is not recognized by Prefect UI. When I run this flow "using default param values", the flow fails with this error
    prefect.exceptions.SignatureMismatchError: Function expects parameters ['start_date'] but was provided with parameters []
    which means the default value is not recognized (?). And when I select a date value using the date picker, looks like the date format is not recognized by my flow code with this error
    - start_date: invalid date format
    How should I handle this?
    ✅ 1
    a
    j
    8 replies · 3 participants
  • s

    Sven Aoki

    08/18/2022, 10:11 AM
    Upgraded to 2.1.0, ran locally works fine but deployments don't work anymore as libraries could not be find. Do you have to somehow install libraries now to run the deployments???
    ✅ 1
    a
    5 replies · 2 participants
  • j

    Jamie Blakeman

    08/18/2022, 11:07 AM
    apologies if this is a silly question, but I’ve been going around and around for an hour now How can we trigger a flow run from the REST API (Prefect REST API - Prefect 2.0) using a POST
    /api/flow_runs/
    request from an external source? This was something we’d do regularly with the GraphQL api in Prefect 1.0
    ✅ 1
    a
    6 replies · 2 participants
  • o

    Oscar Björhn

    08/18/2022, 11:34 AM
    2.1.0 seems great, I especially like being able to set schedules when building! However, as far as I can tell, there's no way to set the timezone, unless I'm missing something. If I'm not missing something I'll go ahead and create a github issue, just figured I'd check first.
    ✅ 1
    a
    a
    +2
    13 replies · 5 participants
  • s

    Saman

    08/18/2022, 12:12 PM
    Hi all, we are running the Prefect 2.1.0 agent on Kubernetes. The API in the deployment has no errors, but the agent throws the following error: prefect.exceptions.PrefectHTTPStatusError: Client error ‘404 Not Found’ for url ‘https://api.prefect.cloud/account/[my accountid]/workspace/[my workspace id]/block_types/’ Can someone please explain what is going on?
    ✅ 1
    👀 1
    s
    b
    8 replies · 3 participants
  • o

    Oscar Björhn

    08/18/2022, 1:01 PM
    I have an issue running deployments in 2.1.0. My dbt flows take two parameters that have default values set: loglevel and n_dbt_threads. For some reason I am now getting the following message when running the deployments: prefect.exceptions.SignatureMismatchError: Function expects parameters ['loglevel', 'n_dbt_threads'] but was provided with parameters []. You can see the parameters listed on my deployment page in the screenshot attached. If I run the deployment with custom parameters, manually setting the values for the two parameters, everything's fine. Has anyone found a workaround for this problem, or am I meant to be doing something different compared to 2.0.4?
    ✅ 1
    a
    4 replies · 2 participants
  • l

    Lucien Fregosi

    08/18/2022, 1:55 PM
    Hi 👋 For
    KubernetesJob
    is it possible to specify a
    node-selector
    and
    tolerations
    to be able to assign the worker pod to a specific node pool ? I can’t see it in the doc Thanks for your help
  • c

    Clint M

    08/18/2022, 2:11 PM
    I’ve got a question RE: deployments with
    null
    (local) storage
    o
    16 replies · 2 participants
  • t

    Tim Enders

    08/18/2022, 2:23 PM
    Is there a way to stop a Dask cluster's queue to do work when an exception happens?
    2 replies · 1 participant
  • r

    Rio McMahon

    08/18/2022, 2:27 PM
    Are there performance implications between • starting several agents each with their own single work queue • starting one agent with several work queues (e.g. moving all work queues to one agent like
    prefect agent start -q queue1 -q queue2 …
    within the script below) I am bootstrapping agents within a docker container with this python script:
    import prefect
    import asyncio
    import subprocess
    import time
    import sys
    
    async def bootstrap_agents():
    
        # get list of work queues
        pc = prefect.get_client()
        work_queues = await pc.read_work_queues()
    
        # create agents by ID 
        for wq in work_queues:
            proc = subprocess.Popen(['prefect', 'agent', 'start', str(wq.id)])
            print(f'Starting agent associated with {wq.name}. PID number is {proc.pid}', file=sys.stdout)
    
    if __name__ == '__main__':
        asyncio.run(bootstrap_agents())
        while True:
            time.sleep(1)
    If there is a better deployment pattern I’d love to get insight on that.
    r
    j
    6 replies · 3 participants
  • r

    Ryan Lattanzi

    08/18/2022, 2:49 PM
    good morning everybody - i have an "execution environment" question: is it possible to have an agent running on a micro EC2 instance that deploys flow runs as ECS tasks? i know in 1.x we could spin up an ECS agent that would automatically do that, but in 2.x it seems like we would manually have to stitch that piece together? this repo shows how to spin up an agent running on ECS in 2.x which is cool, but the infra in the deployment is still
    Process
    - which means the flow run will be executed in the same container that the agent is running - am i understanding that correctly?
    ✅ 2
    d
    a
    8 replies · 3 participants
  • j

    Jared Robbins

    08/18/2022, 2:54 PM
    Looking to get back into prefect now that it is more stable. I have multiple deployments each with their own flow. Can i prevent the same flow from running twice at the same time while using just one work queue? In the early betas the solution was one work queue or agent per deployment. Edit: dang. Just noticed #5623 is still open
    ✅ 1
    c
    a
    +1
    21 replies · 4 participants
  • t

    Tim-Oliver

    08/18/2022, 3:07 PM
    I am running a flow in Prefect 1.0 with a
    DaskExecutor
    with
    cluster_class="dask_jobqueue.SLURMCluster"
    . Everything works fine, unless SLURM kills the resources (e.g. due to time limit). Then the computation is naturally terminated, but in the Prefect cloud I still see the flow and task running. I did an experiment where SLURM kills the resouces after 2min, but the flow is still shown as running for 1h 38min and counting. I thought that the heartbeat option should kick in at some point to flag the flow as
    Failed
    . Has anyone experience with a similar setup?
    a
    1 reply · 2 participants
  • c

    Charlie Henry

    08/18/2022, 3:40 PM
    Hi everyone, I'm trying to use Docker Storage with our dockerhub account and I keep getting an error that I'm not given permission to push to our account:
    InterruptedError: denied: requested access to the resource is denied
    . If I
    docker push
    outside of prefect it'll push no problem. Maybe it's an incorrect
    registry_url
    ?
    flow.storage = Docker(   
        registry_url="<http://registry.hub.docker.com|registry.hub.docker.com>", 
        image_name="atddocker/atd-microstrategy",
        image_tag=ENV,
    )
    ✅ 1
    a
    8 replies · 2 participants
  • c

    Clint M

    08/18/2022, 3:47 PM
    when you call
    prefect agent start -q misspelledname
    it shouldn’t auto create the queue
    ✅ 1
    j
    13 replies · 2 participants
  • s

    Sam Garvis

    08/18/2022, 3:51 PM
    It would be pretty nice to be able to share certain blocks across workspaces. Like if you're using a dev and prod workspace that share the same secrets, having two different sets of secrets adds the possibility for there to be discrepancies between the two
    ✅ 1
    a
    3 replies · 2 participants
Powered by Linen
Title
s

Sam Garvis

08/18/2022, 3:51 PM
It would be pretty nice to be able to share certain blocks across workspaces. Like if you're using a dev and prod workspace that share the same secrets, having two different sets of secrets adds the possibility for there to be discrepancies between the two
✅ 1
a

Anna Geller

08/18/2022, 3:55 PM
this could be dangerous since workspaces often serve to isolate environments nothing stops you, though, from creating blocks via code and reusing this code to create the same blocks across workspaces - e.g. here
👍 1
I think putting this together into something like Terraform or CloudFormation can largely solve the issue for you
if you need a more complex setup, I'd encourage you to reach out to sales@prefect.io and we could investigate a more advanced customization for your use case
View count: 1