https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Andreas Nord

    08/12/2022, 3:12 PM
    Hi! Decided to try to migrate to Prefect 2 but have some trouble getting a Docker agent going. I get the error:
    Flow could not be retrieved from deployment.
    On Prefect 1 I would package all my python dependencies in an Docker image, as well as the flow code (using storage=Module()). What would be the equivalent for Prefect 2? I saw the option of putting the flow definitions on AWS but I don't really see the point
    ✅ 1
    k
    o
    • 3
    • 6
  • s

    Stéphan Taljaard

    08/12/2022, 4:08 PM
    Hey! Random question - with v1, is it still possible to use the Cloud UI, and connect it up to my apollo/graphql endpoint? With server, you can change the endpoint when clicking on this button
    ✅ 1
    a
    • 2
    • 1
  • p

    Philip MacMenamin

    08/12/2022, 4:13 PM
    If I have a flow, with a task
    task_name
    , which returns "some_value", and I do:
    state = flow.run()
    How to I get the return of
    task_name
    ? (ie "some_value")
    ✅ 1
    a
    • 2
    • 6
  • j

    Jimmy Le

    08/12/2022, 4:31 PM
    I was mapping through 1000 tasks, Orion logger can get mad and throw an error, but I just realized that the flow will still continue. I was cancelling the flow and trying to debug what I did wrong but I just had to be a little more patient 🤦‍♂️
    ✅ 1
    a
    j
    • 3
    • 3
  • h

    Heather DeHaven

    08/12/2022, 4:42 PM
    Hi, I'm quite new to Prefect, very much still learning. I'm using Prefect 1.0. I have a flow that launches another flow using create_flow_run. I'd like to be able to access a Parameter's value and use it to set the run_name of the subsequent flow. I read this doc and saw the use case of using the state_handler to rename the current flow. Is there a way to get the Parameter value in the flow and use it in my create_flow_run for run_name? Can state_handler be used for that? Here's an example of what I'm trying to achieve:
    with Flow(name="my_parent_flow") as flow:
        name = Parameter('name')
        ...
        create_flow_run.map(
            flow_name=unmapped('child_flow')
            ...
            run_name=unmapped(f'<use Parameter 'name''s value> child run')
        )
    ✅ 1
    b
    • 2
    • 2
  • a

    Alex Tam

    08/12/2022, 5:02 PM
    I'm creating a deployment via prefect deployment build & prefect deployment apply. When I check prefect cloud, I can see the deployment but it says 'This deployment is deprecated.' Any tips? I've tried deleting via UI and trying again.
    ✅ 1
    o
    j
    • 3
    • 3
  • s

    Sean Malone

    08/12/2022, 5:05 PM
    In 2.0, is there a way to see a list of running/not-running agents similar to that in Cloud 1.0?
    ✅ 2
    a
    • 2
    • 1
  • d

    Deepak Pilligundla

    08/12/2022, 6:23 PM
    we have installed required packages that need to connect to the redshift cluster but still when we try to register the flows we are seeing the following error
    Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  ModuleNotFoundError("No module named \'redshift_connector\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
    e
    • 2
    • 4
  • i

    Isaac Kargar

    08/12/2022, 6:38 PM
    Hello folks! I got a bit confused about hosting Prefect on a cloud VM explained here: https://discourse.prefect.io/t/hosting-a-remote-orion-instance-on-a-cloud-vm/967 Is my understanding true? We have our code locally in vscode, and we create a VM. Then we do all the settings mentioned in the link. Now, if we run the flow locally, it will be executed on the VM and not the local machine, right?
    k
    m
    a
    • 4
    • 13
  • p

    Panos V.

    08/12/2022, 6:41 PM
    Hey everyone! I’m currently evaluating different orchestrators like Airflow, Prefect and Metaflow for an MLOps pipeline in a large enterprise. I’ve already done a little bit of research and came across even with combinations of Prefect as a general orchestrator and Metaflow for ML DAGs. I’m currently leaning more towards Prefect. Has anyone worked with any of the aforementioned tools to offer some insights? (edited)
    r
    • 2
    • 1
  • m

    Matt Delacour

    08/12/2022, 6:43 PM
    Thanks for the migration guide from 1.0 to 2.0 Can we have a similar guide of the GraphQL APIs that change from 1.0 to 2.0 if any ?
    ✅ 1
    b
    • 2
    • 2
  • a

    Alex Shea

    08/12/2022, 7:10 PM
    Has anyone had an issue with env vars not getting passed from the agent to the job with a kubernetes run leveraging
    PREFECT__CLOUD__AGENT__ENV_VARS
    agent variable?
    b
    • 2
    • 6
  • c

    Chris Marchetti [Datateer]

    08/12/2022, 7:46 PM
    Hello, all! I am having a issue when using github actions to deploy a pipeline.
    [2022-08-12 19:42:34+0000] INFO - prefect.Docker | Pushing image to the registry...
    The error is:
    InterruptedError: EOF
    Complete traceback in thread. Anyone seen this before?
    a
    • 2
    • 3
  • t

    Tim Helfensdörfer

    08/12/2022, 8:30 PM
    Hi, why is the switch for this deployment disabled?
    k
    • 2
    • 2
  • e

    Emil Ordoñez

    08/12/2022, 9:52 PM
    hey everyone. Here with some maybe dummy question: I wanted to be sure about the concept of "Service Accounts" in the "Growth" Plan. Are those like shared users that we can share, for example across the company or certain areas? For example: • DEV user • QA user • PROD user ? https://www.prefect.io/pricing/
    ✅ 1
    a
    • 2
    • 2
  • e

    Edmondo Porcu

    08/12/2022, 9:54 PM
    Anyone who created two tasks but only got one running in Prefect 1.0?
    ✅ 1
    a
    • 2
    • 3
  • b

    Blake Hamm

    08/12/2022, 10:24 PM
    Hey all! I'm very impressed with prefect 2! It's been a learning curve, but it definitely feels more pythonic to create flows! Question regarding parameters: I'm passing in a parameter as a python dictionary. It works great with the default settings (although a little unclear on the docs how to define a dictionary parameter in the deploy yaml). However, I would like to have the ability to change the dictionary parameter through the UI on runs. I can't seem to get it to run as expected. Even though I have defined the parameter as a
    dict
    , it still reads it in as a string in the UI and can't seem to parse it out. I'm pretty much stuck with the default dictionary parameter or I can trigger it from the CLI. Do I essentially have to use
    pydantic
    and explicitly specify the model when working with dictionary parameters? Some other funny parameter behavior, the UI displays the default parameter as
    [object Object]
    when I'm creating a custom flow run. But on the flow run (with defaults), it looks like more like a dictionary.
    🙌 2
    a
    • 2
    • 7
  • b

    Brad

    08/12/2022, 10:59 PM
    hey team - I'm having some trouble with my agents picking up work from my work queue, what is the best way to debug?
    ✅ 1
    m
    a
    • 3
    • 8
  • r

    Rohan Chutke

    08/13/2022, 12:25 AM
    Hi All, I am using Prefect Cloud. In one of the flows, its trying to upload a file to s3 which is about ~900MB. And the flow is stuck since hours trying to upload that file to s3. Any solutions/ suggestions would really be appreciated. Thanks
    r
    a
    • 3
    • 4
  • m

    MasatoShima

    08/13/2022, 1:38 PM
    Hi. I’m trying Prefect Orion. I tried to run deployment. However, the flow status stays ‘pending’ and is not executed. Does anyone have any idea what the reason might be? I would be happy to know.
    t
    a
    • 3
    • 23
  • a

    Artur Spatari

    08/13/2022, 2:44 PM
    Hi team, Someone have experience with programatic scheduling next flow running? Example: Flow A > execute a some logic > take a computed next flow run datetime > schedule execution
    t
    • 2
    • 2
  • m

    Moises Vera

    08/13/2022, 6:10 PM
    noob question for prefect 2, • is it necessary to run the build command each time I change my code? • looks like the build command changes my deployment file with my old config so I have to re write the old config I had again each time
    ✅ 1
    o
    a
    j
    • 4
    • 14
  • m

    Marwan Sarieddine

    08/13/2022, 6:40 PM
    Hi folks, a very minor comment which might help others like myself trying to parse the good content on migrating to prefect 2.0 the youtube channel playlist “Prefect 2.0 Upgrade” does not present the individual videos in the correct order
    ✅ 1
    a
    c
    • 3
    • 5
  • r

    Rio McMahon

    08/13/2022, 7:11 PM
    I am attempting to create a custom storage block (Digital Ocean uses an
    https://
    prefix but
    s3://
    is hardcoded in the S3 filesystem object). I did this by copying the S3 storage object code and removing the hardcoded prefix but leaving everything else the same (except name and credential arguments). The block successfully registers to prefect cloud. However when I try to attach it as the storage block I get the error in the comments. From what I can tell this is happening during the validation step. At some point prefect is looking at the registry for the
    Block
    type via:
    from prefect.utilities.dispatch import lookup_type, get_registry_for_type
    from prefect.blocks.core import Block
    registry = get_registry_for_type(Block)
    print(registry)
    which returns all the prefect defined blocks. If I try to build a deployment it will fail because the custom filesystem class I created does not show up in that registry. Currently the remote file system object stores all ‘settings’ as plain text which isn’t ideal because I’m storing access credentials. Thoughts on the best way to proceed?
    ✅ 1
    a
    • 2
    • 12
  • d

    Dev Dabke

    08/13/2022, 7:43 PM
    Hi y’all, I’m having a bit of trouble wrapping my head around a couple prefect concepts. I’m trying to set up a simple system for me to run prefect flows. I have a development computer (my laptop) where I write all of my code. Simultaneously, I have a local workstation with a couple GPUs that I want everything to actually run on. What’s the correct way to set up a system with these two machines? Ideally, my workflow would look like this: 1. Make some code changes on my laptop 2. Fire off a workflow run from my laptop that runs on my workstation 3. Enjoy 😎 I’m getting a little confused as to what work queues, agents, deployments, and the dask runner all do. I think where I’m getting a little confused is to how the code from my laptop gets transmitted to my workstation. I use git already, so I would like somehow for me to fire off a workflow run and be able to track exactly which commit ran on my workstation.
    ✅ 1
    a
    • 2
    • 3
  • m

    Marwan Sarieddine

    08/13/2022, 8:32 PM
    Hey folks, a question about prefect 2.0 and pydantic. I certainly appreciate using pydantic at the flow level to allow for complex parameter types and to enable type validation/conversion out of the box. However it seems that pydantic supports starts to fail at the task level. More specifically it seems pickling pydantic models using cloudpickle is troublesome. See the thread for a small example. I am curious if this will remain a known issue with certain storage types going forward…
    a
    f
    o
    • 4
    • 26
  • o

    Oscar Björhn

    08/14/2022, 11:22 AM
    I seem to be unable to create cron-based schedules using the GUI. Anyone know if this is a known issue or if there is some sort of work-around? I just get the message "Failed to create schedule", regardless of what I input. Interval-based schedules appear to work. Edit: Nevermind, seems to be a known issue. https://github.com/PrefectHQ/prefect/issues/6374
    👍 1
    a
    g
    • 3
    • 8
  • j

    James Brady

    08/14/2022, 1:40 PM
    I'm using DaskTaskRunner, karpenter, and the
    dask_kubernetes.KubeCluster
    class – trying to get a flow to run on a GPU-enabled node. What's happening: • The pod_template I'm using for the flow specifies
    <http://nvidia.com/gpu|nvidia.com/gpu>: 1
    , per the docs • karpenter starts a new node, which has a GPU (🙌) • However, the new node can't accommodate the dask client because it doesn't have the right resource annotation ("0/3 nodes are available: … , 3 Insufficient nvidia.com/gpu") I realise this might be a question better-suited to the dask community, but would appreciate any stories of people successfully running Prefect 2 workloads on GPU-accelerated nodes and/or help figuring out the above specific issue.
    a
    • 2
    • 2
  • j

    James Phoenix

    08/14/2022, 9:02 PM
    Why am I receiving this error please? 🙏
  • j

    James Phoenix

    08/14/2022, 9:02 PM
    File system created with scheme 'gcs' from base path '<gcs://prefect_source_data>' could not be created. You are likely missing a Python module required to use the given storage protocol.
    a
    • 2
    • 2
Powered by Linen
Title
j

James Phoenix

08/14/2022, 9:02 PM
File system created with scheme 'gcs' from base path '<gcs://prefect_source_data>' could not be created. You are likely missing a Python module required to use the given storage protocol.
a

Anna Geller

08/14/2022, 9:47 PM
Pip install gcsfs should fix it
Can you move all other messages to this thread?
View count: 4