https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Mohamed Ayoub Chettouh

    08/18/2022, 4:41 PM
    Question: I want to have a number that increases every time I run a deployment and access it for logging to know which files correspond to which deployment run. How to go about it? Using remote storage (GCS) I tried using a pickle but it appears that it doesn't save on the bucket. Maybe use another bucket or manually pull and push the object to the bucket. Figured there should be an easier way with Blocks.
    ✅ 1
    a
    • 2
    • 2
  • v

    Vishy ganesh

    08/18/2022, 5:40 PM
    I was trying to explore solutions for our use case.... We have a use case where we want to run our jobs in hybrid agents (#PrefectIsPerfect 🙂 ) So I'm trying to accomplish this in prefect 2.0 knowing that we have a little bit of a runway before the solution hits primetime... The cloud options seem to be well documented so I'm less concerned about it Our on prem infrastructure is primarily Openshift. We could spin up VMs to host Docker agents but, was curious how have people gone about running production loads on-prem
    ✅ 1
    a
    • 2
    • 3
  • j

    Jimmy Le

    08/18/2022, 5:42 PM
    Ran into this bug earlier. The UI tells me to use the flag
    --work-queue
    but the CLI says no such option exists. Removing the
    --work-queue
    flagged resolved the issue.
    o
    j
    a
    • 4
    • 5
  • s

    Shanhui Bono

    08/18/2022, 6:21 PM
    Is Prefect Cloud HIPAA compliant? If so, does it have to be Enterprise plan tier?
    a
    • 2
    • 1
  • j

    Jared Robbins

    08/18/2022, 8:19 PM
    A function to check if a block already exists would be helpful
    👍 1
    a
    • 2
    • 5
  • t

    Tim Enders

    08/18/2022, 8:20 PM
    What replaces the old
    flatten
    call in Prefect 2.0? I have a list of lists after mapping tasks and I used to be able to use
    flatten
    to coalesce that into a single large list.
    👀 1
    t
    a
    • 3
    • 2
  • m

    Mars

    08/18/2022, 8:46 PM
    Hi, just thought I’d say that the logging configuration in 2.0 is nice. I have a flow where I need to debug a
    requests-oauthlib
    connection, and now all I need to do is: 1. Copy
    logging.yaml
    2. Set my logger:
    requests_oauthlib:
            level: DEBUG
            handlers: [console]
            propagate: no
    3. Set
    PREFECT_LOGGING_SETTINGS_PATH=logging.yaml
    in my
    .env
    file. And it works!
    🎉 4
    k
    • 2
    • 2
  • m

    Mars

    08/18/2022, 9:16 PM
    What happened to
    create_markdown_artifact
    in Prefect 2.0? I don’t see markdown or artifacts mentioned in the migration guide.
    ➕ 2
  • j

    Josh

    08/18/2022, 9:56 PM
    Is it possibly to dynamically assign tags to task runs based on parameters? I’m using Prefect 1.x and want to do something like the following across multiple flows.
    my_task.map(databases)
    But I don’t want to overload my databases, so I want to set concurrency limits at the task level to make sure there are only ever
    N < limit
    tasks running against a given database at one time. I know you can set task tags when I’m creating the task object, but can I do so as the flow is executing against mapped tasks?
    m
    d
    +2
    • 5
    • 10
  • b

    Blake Hamm

    08/18/2022, 10:48 PM
    Thanks for the new release today! Very exciting. I'm trying to use the new python
    Deployment
    class. Previously, I defined a
    KubernetesJob
    infrastructure block in a different python script and used the
    .save()
    method to instantiate a block in the UI. It looks like this was removed in 2.1... How can I do this now? Moreover, I'm trying to understand what I pass in for the
    Deployment
    class in the
    infrastructure
    parameter. With the
    storage
    parameter, I just load the
    storage
    block that I previously defined. Can I do the same with the
    infrastructure
    block? If so, how do I load an existing
    KubernetesJob
    block? Or, is the new workflow to just input the
    KubernetesJob
    dictionary directly in your
    Deployment
    class? If that's the case, how do I tie this to a work queue? In general, I'm a big fan of the
    KubernetesJob
    block. idc whether it's an actual block or just a python dictionary I pass into the deployment. Either way, it's been extremely helpful to manage compute depending on the deployment. Would love to streamline it with CICD. I know that's in the works and I'm eager to see best practices with this and implement it.
    a
    i
    • 3
    • 13
  • y

    yusuf yel

    08/19/2022, 1:13 AM
    Hi everyone, 👋 I'm participant ongoing data talks/mlops-zoomcamp and we're using prefect for workflow orchestration. I'm aware there new release. Fully readed changelogs and prefect 2.0 guides. But I'm having problem prefect deployment with mlflow artifacts. When I run my training script(with prefect task/flow) artifacts saves without any problem. But when I create deployment and start agent mlflow doesn't saves artifacts. Agent shows my all tasks and flow without any error. Happy new release! 🎉
    👀 1
    b
    • 2
    • 3
  • k

    kojisuganuma.dog

    08/19/2022, 5:15 AM
    Hi everyone! Congrats on the release of Prefect Cloud v2 🎉 I have a question about the lifecycle of Prefect Cloud. When is it a time limit (EOL) to use Prefect Cloud v1? I am currently using v1 and would like to move to v2 at some point.
    ✅ 1
    b
    • 2
    • 1
  • t

    Thomas Pedersen

    08/19/2022, 6:02 AM
    Hi, Not sure if I missed a setting a something in v1, please let me know if that's the case... But, I was wondering if v2 is better (or will be) at managing its database size - in particular in regards to db partitions and retention policies? With about 1,500 flow runs / 100,000 task runs a day, we seem to be accumulating data rather quickly. About 3 months of running on v1 have given us:
    +--------------+-------------+------------------+-------------------+
    |table_name    |relation_size|indexes_sizes_size|total_relation_size|
    +--------------+-------------+------------------+-------------------+
    |log           |15 GB        |9060 MB           |24 GB              |
    |task_run      |5765 MB      |17 GB             |23 GB              |
    |task_run_state|13 GB        |6308 MB           |19 GB              |
    |flow_run      |241 MB       |472 MB            |713 MB             |
    |flow_run_state|438 MB       |118 MB            |557 MB             |
    +--------------+-------------+------------------+-------------------+
    👀 1
    g
    • 2
    • 1
  • c

    Cyber Pyro

    08/19/2022, 6:52 AM
    I've been searching for a python workflow management and I've found the Prefect website. can I use Prefect 2.0 locally without cloud services for free without any user limitations? Is the paid subscription for using the Prefect cloud?
    ✅ 1
    a
    o
    • 3
    • 4
  • b

    Byron Kilian

    08/19/2022, 7:10 AM
    Morning, quick question does prefect have to be installed where the agent is running from? If so does it not create 2 database instances?. When prefect is installed on host running the webservice the .prefect directory and orion.db instance gets created. And then when installing prefect again on the host from where the flows are being run then another .prefect directory and orion.db instance gets created?
    ✅ 1
    b
    • 2
    • 7
  • y

    Yaron Levi

    08/19/2022, 7:43 AM
    Hi (-: A very quick one. I am running a Local Agent in a Render service:
  • y

    Yaron Levi

    08/19/2022, 7:43 AM
  • y

    Yaron Levi

    08/19/2022, 7:43 AM
    The agent seems to start successfully and listen to flows on a queue I’ve defined.
  • y

    Yaron Levi

    08/19/2022, 7:44 AM
    the queue name is “work-queue”. I’ve also defined the needed PREFECT_API_KEY and PREFECT_API_URL with my account and workspace in Render’s env.
  • y

    Yaron Levi

    08/19/2022, 7:44 AM
    (if you don’t know Render, it’s an Heroku alternative)
  • y

    Yaron Levi

    08/19/2022, 7:46 AM
    But when I start a flow via Prefect Cloud UI 2.0, the agent won’t receive the flow:
  • y

    Yaron Levi

    08/19/2022, 7:46 AM
  • y

    Yaron Levi

    08/19/2022, 7:48 AM
    this local agent runs on Render on a web service. Maybe I need to open some ports? or install it on a different type of service in Render? On the other hand if you look at the logs above it dose say “Agent started! Looking for work from queue(s): work-queue...” meaning it might succeeded contacting the Prefect Cloud.
    r
    y
    • 3
    • 15
  • y

    Yaron Levi

    08/19/2022, 7:48 AM
    Any help would me much appreciated.
  • a

    Alexander Belikov

    08/19/2022, 8:39 AM
    Concerning the latest changes in versions >2.0.4 : previously agents were able to find and place flows into queues by tags. From version > 2.0.4 the doc says queue names are used instead. Specifically here (https://docs.prefect.io/concepts/deployments/) it s said:
    A deployment additionally enables you to:
    - Assign a work queue name to delegate deployment flow runs to work queues.
    But in the deployment section there is no info on how to declare a queue to be associated with a deployment. Could you please clarify?
    o
    j
    • 3
    • 6
  • e

    Erik Tiengo

    08/19/2022, 8:50 AM
    In future versions it is expected to pass parameters on the command line
    prefect deployment run command
    ? as plain text or reading them from file? thanks in advance
  • o

    Oscar Björhn

    08/19/2022, 8:52 AM
    I may have bumped into a new bug: It seems none of my schedules are triggering anymore on 2.1.1, neither ones I've created with the --cron argument nor ones I've created manually in the GUI. The flow runs show up as scheduled and then late. They are never processed by any agents (nothing shows up in their logs) and they do not show up under "upcoming runs" for the work queues. I can run all of my deployments by triggering them manually, it's only the scheduled runes that don't get picked up. Does anyone have working schedules in 2.1.1?
    ✅ 1
    j
    a
    k
    • 4
    • 10
  • m

    max

    08/19/2022, 9:38 AM
    hey people! really need your help 🙏 I am trying to build a
    flow
    , that should execute some tasks at the same cluster. It means that if the
    flow
    get a new message, it should use the same workers (cluster) as for previous messages instead of creating new ones. do you have any idea how to solve my case (coz I don't)?
    :blob-attention-gif: 1
  • j

    José Duarte

    08/19/2022, 10:00 AM
    Hey all, I’m currently looking into using Prefect to leverage dbt’s model graph. Ideally, we want to use Prefect’s 2 dynamic flow creation to generate a flow for each node of the dbt graph, however, we’re not being able to launch several flows to run at the same time. We’re definitely missing something, but couldn’t find it in the new docs. Can someone provide some insight? I’m running Prefect locally
    o
    • 2
    • 34
  • d

    David Hlavaty

    08/19/2022, 10:05 AM
    Are there any plans for versioning deployments in Prefect 2 similar to how registered flows are versioned in Prefect 1? Currently the previous deployment just gets overridden, and that is including all source files uploaded to storage. Related to this - what about having two deployments of the same flow, but with different schedules. Currently each deployment gets uploaded to the storage, which means duplicate file transfer and 2 versions in S3 if bucket versioning is enabled
    ✅ 1
    a
    • 2
    • 6
Powered by Linen
Title
d

David Hlavaty

08/19/2022, 10:05 AM
Are there any plans for versioning deployments in Prefect 2 similar to how registered flows are versioned in Prefect 1? Currently the previous deployment just gets overridden, and that is including all source files uploaded to storage. Related to this - what about having two deployments of the same flow, but with different schedules. Currently each deployment gets uploaded to the storage, which means duplicate file transfer and 2 versions in S3 if bucket versioning is enabled
✅ 1
a

Anna Geller

08/19/2022, 11:34 AM
It already works and this is one of my favorite features in the new deployment UX - you can finally easily version your deployments with the GITHUB_SHA from CI just add a flag --version to your
prefect deployment build
example https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main_simple.yaml#L84
what about having two deployments of the same flow, but with different schedules.
this works two - you create 2 deployments of one flow
perhaps consider disabling bucket versioning for deployment buckets then? might be useful since you probably already version-control your code with Git so versioning on S3 seems a little redundant, but up to you
d

David Hlavaty

08/19/2022, 1:12 PM
Thanks Anna. The version argument for
prefect deployment build
command only adds metadata to the deployment unless I am missing something. It has no effect on where the files are stored in the storage - e.g. S3 in my case. So if my storage points to
<s3://example/flow>
then: 1.
prefect deployment build --version 1 ...
will upload the files to
<s3://example/flow>
2.
prefect deployment build --version 2 ...
will override files in
<s3://example/flow>
What I would expect (or be able to do) is for the version to be appended to the storage configuration: 1.
prefect deployment build --version 1 ...
will upload the files to
<s3://example/flow/1>
2.
prefect deployment build --version 2 ...
will upload the files to
<s3://example/flow/2>
So that building the deployment again cannot interfere with any flows that are currently scheduling - e.g. what if version 2 added extra dependencies to a base image, but another flow is already starting that fetched the image, but has not downloaded the source code yet. I appreciate that the odds of this are very low, but it is a consequence of the build artefacts not being immutable - which is often a desirable property (the same as it is often desirable to make docker images immutable once uploaded to the repository).
Sounds like https://prefect-community.slack.com/archives/CL09KU1K7/p1660907617682639?thread_ts=1660904374.225469&amp;cid=CL09KU1K7 will provide an alternative as I will be able to build and upload docker image with all code, and then reference it from deployments
View count: 4