https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • r

    rectalogic

    07/21/2022, 3:03 PM
    so flows work locally but
    __file__
    is not defined when prefect cloud runs the flow. Any idea what the issue is?
    k
    • 2
    • 12
  • e

    Ellie Redding

    07/21/2022, 3:19 PM
    👋 Hi, I’m new here, trialing Prefect 2.0, and have a couple of question. I’m writing a flow to copy data into a bunch of Snowflake tables. I wanted to use the Snowflake connection instead of managing a connection myself. Seems like
    snowflake_query
    and
    snowflake_multiquery
    are their own tasks, so I can’t use them as part of a different task? Which means that my flow looks like this:
    do_some_stuff()
    for table_name in tables:
        queries = build_queries(table)
        snowflake_multiquery(queries)
    These tasks are all running sequentially, but there are a lot of tables so I’d like the
    snowflake_multiquery
    tasks for each table to run concurrently. How can I make that happen?
    k
    m
    • 3
    • 12
  • e

    Ellie Redding

    07/21/2022, 3:23 PM
    Second question:
    snowflake_multiquery
    was throwing
    Object of type SecretStr is not JSON serializable
    errors from this line of code in the connector, where it dumps the request body, including connection auth info, into the request. I got around this for now by changing the password here from type
    SecretStr
    to
    str
    , but that’s obviously not a great solution 😅 Has anyone else run into this issue?
    k
    • 2
    • 2
  • k

    Kha Nguyen

    07/21/2022, 3:25 PM
    Hi, I am in the process of evaluating Prefect Cloud for my pipeline. I have a setup where there is a flow definition with parameters, and I have about 1000 parameter sets from database (and a lot more when the pipeline is live). What I can do is to deploy the flows with parameters for each parameter set via API. I want to know how this scenario can be handled: If I update the code of my flow, such as using a different algorithm, or add another step, how can 1000+ deployments be updated?
    k
    • 2
    • 18
  • j

    Jai P

    07/21/2022, 3:48 PM
    👋 i see a new prefect version (
    2.0b10
    ) was released yesterday, but don't see associated release notes. was this just a minor bug fix?
    k
    • 2
    • 2
  • y

    Yupei Chen

    07/21/2022, 4:01 PM
    When using prefect 2.0, how do I verify that agents are able to communicate with the cloud? On 1.0 there was an agents screen thats showed all connected agents.
    k
    • 2
    • 4
  • t

    Tim Enders

    07/21/2022, 5:32 PM
    On Prefect 1.0, does the agent run each flow in its own process? I think the answer is yes
    k
    • 2
    • 1
  • j

    Jai P

    07/21/2022, 6:08 PM
    👋 another qq: is there any kind of documentation on the type of data the prefect agent might send back if we were to use prefect cloud vs self-hosted?
    a
    • 2
    • 3
  • b

    Billy McMonagle

    07/21/2022, 6:15 PM
    On 1.0, is there a way to run a ShellTask where the stdout/stderr is streamed in its raw format, bypassing the prefect logger?
    ✅ 1
    a
    j
    • 3
    • 10
  • c

    Christian Nuss

    07/21/2022, 6:22 PM
    Hey all! can i make an
    @task
    that across all concurrent flows is a Singleton (e.g only one of that task can run at a time)?
    ✅ 1
    a
    k
    • 3
    • 3
  • j

    Jason

    07/21/2022, 6:34 PM
    Any chance the API had some issues last night? It seems like the flow shut down before it started because it couldn't pull secrets. No errors before or after this one bad run:
    Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    ✅ 1
    a
    • 2
    • 1
  • m

    Matt Alhonte

    07/21/2022, 6:59 PM
    Heya! How do we add tags to AWS ECS containers launched by Prefect? Like, what arguments do we pass to
    DaskExecutor
    and/or
    ECSRun
    ?
    k
    • 2
    • 2
  • c

    Chu

    07/21/2022, 7:11 PM
    Hi Community, I’m using Prefect 1.0, and I want to achieve a parallel running using Dask. Here is the process: We have a flow of flows framework as below, each A, B C, D E is a flow to schedule some transformation jobs. For flow C, D, E, we wanna pass some org_id as a list of variables to the flow (which scheduled dbt jobs), and with parallelism for example, org_1, org_2 and org_3 can run in parallel and accelerate our ETL process. Question: 1. how and where can I set the list of org_id, since in the future we wanna add more org_id, so it cannot be hard coded, must have some certain flexibilities 2. to achieve the parallel running, where do I set DaskExecutor? in single flow or in the flow orchestrator?
    k
    • 2
    • 39
  • d

    datamongus

    07/21/2022, 7:37 PM
    Are there any docs on how to use the Postgres Task while connecting via an ssh tunnel ? AKA A bastion server
    k
    k
    • 3
    • 2
  • y

    Yupei Chen

    07/21/2022, 7:53 PM
    P2.0, running
    prefect work-queue ls
    I get below:
    Response: {'detail': 'Request specified API version 0.6.0 but this server only supports version 0.7.0 and below.'}
    For more information check: <https://httpstatuses.com/400>
    An exception occurred.
    k
    • 2
    • 4
  • s

    Salim Doost

    07/21/2022, 9:28 PM
    When using Prefect 1 or 0.15, how can I know for a task with
    always_run
    trigger and
    skip_on_upstream_skip=False
    , whether any of the upstream-tasks failed or were skipped? I need this information in the task-run method.
    k
    • 2
    • 5
  • x

    Xavier Babu

    07/21/2022, 9:47 PM
    Team, Can someone share an example (Prefect 2.0) to have a flow with kwargs and pass parameter in the DeploymentSpec for the flow? So that I can pass many parameters dynamically. Thank you for your help.
    a
    • 2
    • 5
  • s

    Shaoyi Zhang

    07/21/2022, 10:29 PM
    Prefect 1.0 - Why Prefect don’t recommend forcing version update when the flow metadata didn’t change? Having an updated timestamp on the flow helps user understand when the flow was updated without going back to git repos or CI/CD pipelines logs.
    ✅ 1
    a
    • 2
    • 9
  • j

    Jeff Hale

    07/21/2022, 11:37 PM
    Hi all! 👋 As we get close to Prefect 2.0 GA next week the Prefect team is making lots of releases. Yesterday’s 2.0b10 release fixed some minor bugs. 2.0b11 was released about an hour ago. We don’t have formal release notes, but highlights include the following changes: • Infrastructure types are added and preferred instead of flow runners • Default storage and storage blocks are removed • Results are always persisted locally instead Be prepared for more 2.0 changes in the coming days. We’ll be updating docs and examples in Discourse as quickly as possible. Don’t hesitate to reach out here with questions. 🙂 Finally, a reminder that if you use 1.0 and don’t want to install prefect 2.0 when GA lands next week, make sure you’ve pinned your version (for example:
    pip install prefect==1.2.4
    .
    :marvin: 5
    :marvin-duck: 4
    :thank-you: 2
    🙏 5
  • r

    Rahul Kadam

    07/22/2022, 3:53 AM
    Hi All, We are running prefect in ECS cluster where the flows run as fargate tasks and its working just fine.. The only problem we are seeing is overall time taken by a flow to process, which again is not something prefect is causing, but the fargate task itself is taking around 60 sec to just startup and be ready to execute the code. We want to switch to a better way. Idea is to have flow also call an 'always running' container which will be a EC2 based ECS service with 1 or more tasks, and the flow should access those containers only, rather than creating a fargate task for each flow. Is this possible and supported by prefect ? If yes, could you please direct me to the correct documentation ?
    ✅ 1
    a
    • 2
    • 2
  • b

    bin zhang

    07/22/2022, 3:56 AM
    Hi All, I’m using https://cloud.prefect.io/, meet some issues, pls help add ‘1’ in sys path as ‘2’ use some libs under ‘1’ this works fine when I use flow.run(); but when I use flow.register(), the agent(already restart) meet this: It looks like the agent not understand sys.path added.
    ✅ 1
    a
    • 2
    • 3
  • d

    davzucky

    07/22/2022, 6:01 AM
    S3StorageBlock is not working with Minio as we cannot specify the endpoint_url. This is something I added on the collection-aws. Will this block move to the collection-aws ? If not, how can I do a push request to the branch ?
    ✅ 1
    a
    • 2
    • 3
  • l

    Lucien Fregosi

    07/22/2022, 7:42 AM
    Hi 👋 I’m testing Prefect 2.0 kubernetes/Helm deployment Everything seems fine expected that the agent got an issue when launching the job because the namespace and the service account are set with default value. Is it possible to override these value ? I didn’t found it in the doc
    ✅ 1
    a
    • 2
    • 3
  • a

    Andreas Nigg

    07/22/2022, 9:36 AM
    Hey. I just upgraded to prefect 2.0b11 and I've an agent running in a rather restrictive kubernetes environment - meaning I can't run it as root. This results in having this warning at the Agent startup:
    /usr/local/lib/python3.9/site-packages/prefect/context.py:461: UserWarning: Failed to create the Prefect home directory at /.prefect
    with SettingsContext(profile=profile, settings=new_settings) as ctx:
    While Agent still starts up and picks up flow-runs, I get the following exception during flow-run
    File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
    FileNotFoundError: [Errno 2] No such file or directory: '/.prefect/storage'
    I guess this has something to do with removing storage/default storage in the latest release? I configured my deployment with a RemoteFileSystem - Packager - not sure why prefect still needs to write something locally? Find my flow + deployment in the Thread.
    ✅ 1
    a
    m
    m
    • 4
    • 9
  • a

    Abin Joseph

    07/22/2022, 9:57 AM
    Does prefect enterprise support onpremise deployment?
    ✅ 2
    a
    • 2
    • 2
  • c

    Carlos Paiva

    07/22/2022, 11:14 AM
    Hey, On the last Discourse announcement it says that "*If you currently use Prefect 1.0, please make sure you’ve pinned your Prefect version explicitly* (including your Docker images!)". What does this mean exactly?
    ✅ 1
    a
    • 2
    • 3
  • v

    Vadym Dytyniak

    07/22/2022, 12:10 PM
    Hi. Prefect Cloud UI should support 1.x and 2.x flows after the Prefect 2.0 GA?
    ✅ 1
    c
    • 2
    • 5
  • b

    Benjamin.bgx

    07/22/2022, 12:24 PM
    Hi ! I was testing Prefect Cloud 2.0 and tonight, all my block storage for cloud (Azure and AWS S3) disappear from the interface. I cannot create anymore now. Anyone with the same problem ?
    ✅ 1
    a
    a
    • 3
    • 14
  • e

    Ed Burroughes

    07/22/2022, 12:33 PM
    Hey, we're using the prefect mapping functionality in Prefect 1. Whereby one of the 5 tasks fails, this however in the UI it is appearing as a successful flow run even though one fails. Is there a neat way to fail the flow if any of the mapped tasks fail?
    ✅ 1
    a
    • 2
    • 1
  • b

    Britt Evans

    07/22/2022, 2:01 PM
    Hi, I’m using 2.0b9 with orion and the prefect agent running on the same host. My prefect agent is dying frequently (it rarely stays alive for an hour) with exceptions like
    anyio.BrokenResourceError
    and
    httpcore.ReadError
    when the agent is using the orion client. Has anyone else had similar issues?
    ✅ 1
    k
    • 2
    • 6
Powered by Linen
Title
b

Britt Evans

07/22/2022, 2:01 PM
Hi, I’m using 2.0b9 with orion and the prefect agent running on the same host. My prefect agent is dying frequently (it rarely stays alive for an hour) with exceptions like
anyio.BrokenResourceError
and
httpcore.ReadError
when the agent is using the orion client. Has anyone else had similar issues?
✅ 1
k

Kevin Kho

07/22/2022, 2:38 PM
Is this for local Orion or Cloud 2.0?
b

Britt Evans

07/22/2022, 2:39 PM
local, running on an EC2 instance
k

Kevin Kho

07/22/2022, 2:42 PM
Oh my bad I missed that. That does seem weird. Does it happen when attempting to run a flow or even without running a flow?
b

Britt Evans

07/22/2022, 2:47 PM
Lots of times it’s trying to get jobs from the queue, get_runs_in_work_queue, https://github.com/PrefectHQ/prefect/blob/orion/src/prefect/client.py#L875
k

Kevin Kho

07/22/2022, 2:48 PM
Gotcha ok will ask that team
b

Britt Evans

07/22/2022, 2:50 PM
thank you
View count: 3