• Kilian

    Kilian

    1 year ago
    Hi all, been using prefect for a while now, and big fan of it. I have the following use case and am not sure if it's supported by prefect. I have a flow that first downloads quite a large file and then processes it using a GPU. I would like to set this up in a parallel cluster of some sort. However, I wonder how can I ensure that the file is downloaded and processed on the same node. To my understanding, a task can be executed on any machine if it is ready to be processed and only waiting for the upstream tasks to finish. Meaning, it might happen that the file downloaded on one machine, the gpu processing is waiting but might start on another machine and then can obviously not find the file. One workaround would be to make this one big task and request a gpu for it, but then if the download takes quite long the GPU could have been utilised elsewhere. Are there any other options. Basically enforcing that two tasks are executed on any yet the same machine? Thank you
  • Lior

    Lior

    1 year ago
    Hey, I am looking to persist to local storage tasks that output pandas dataframes, do you have a relevant example?
    Lior
    Jim Crist-Harif
    +1
    8 replies
    Copy to Clipboard
  • Adam

    Adam

    1 year ago
    Any uptime issues at the moment? The Prefect Cloud UI is not loading anything and I see errors in the console like
    Access to fetch at '<https://api.prefect.io/graphql>' from origin '<https://cloud.prefect.io>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled
    Adam
    Jeremiah
    3 replies
    Copy to Clipboard
  • Enrique

    Enrique

    1 year ago
    Hi prefect team! I have an issue when registering a flow using docker storage. I have a Github Action that downloads the latest version of prefect and register the new flow when the branch changes. I haven't introduced any change since October 30th using Prefect Version 0.13.13. Today when I tried to update my flow this error has appeared. I've tried to do the same from my computer (using version 0.13.13) and it works fine. In our team we follow the update announcements of new versions of Prefect but we haven't seen anything that could affect to our flow. Do you have any idea of what the problem could be? We are using the following command to register if it can help to find the problem:
    flow.register(project_name=env_label, add_default_labels=False)
    We use also the
    flow.storage = Docker(...)
    command. Thanks for your time, and sorry if it is an obvious question 😅
    Enrique
    Jim Crist-Harif
    +1
    4 replies
    Copy to Clipboard
  • Saulius Beinorius

    Saulius Beinorius

    1 year ago
    Hi, it seems that when cancelling a flow run when there are tasks within a resource manager segment, the resource manager cleanup is not called, is that intentional (or can be enabled somewhere)?
    Saulius Beinorius
    Jim Crist-Harif
    +1
    5 replies
    Copy to Clipboard
  • p

    Peter Peter

    1 year ago
    Hello, We are currently looking at shelltask and dbtshelltask and we are wondering why helper_script is only in the constructor and why it cannot set when we call run. There must be a reason for this. If someone could shed some light on this. Thanks Peter
    p
    Jim Crist-Harif
    +1
    4 replies
    Copy to Clipboard
  • p

    Peter Peter

    1 year ago
    Hello, In the documentation it show that when a flow is in pause state that it will show up in the dashboards notifications. We are not seeing this, Is there some setting we need to change or enable notification in our flow? here is where we see it.https://docs.prefect.io/core/idioms/pause-for-approval.html#pause-for-approval Thanks
    p
    Jenny
    3 replies
    Copy to Clipboard
  • r

    Riley Hun

    1 year ago
    Hi everyone - This question pertains to the helm chart for deploying Prefect Server on k8s. I'm trying to use an external database as was recommended in the docs. However, I am dealing with an issue when trying to connect to a postgres cloudsql instance on GCP.
    r
    Michael Adkins
    50 replies
    Copy to Clipboard
  • v

    Vinod Sugur

    1 year ago
    Hi everyone - How to convert prefect python dict parameter to JSON string? I am looking at JSONCompatible. Please point me to some examples.
    v
    Jim Crist-Harif
    +1
    5 replies
    Copy to Clipboard
  • j

    jack

    1 year ago
    Heya - can anyone point me in the right direction/documentation/example if I wanted to register a flow on Prefect Cloud by pushing changes to a repo in Azure DevOps?