• s

    Sven Teresniak

    1 year ago
    hmmm. when you are creating a project and the project already exists, prefect cli will just print a message and return without error. so no need for
    --skip-if-exists
    because this is the default behaviour. but when you're creating a tenant a 2nd time (same name and slug) you will get an exception and return value 1 from the prefect cli. it seems
    create-tenant
    needs a
    --skip-if-exists
    and
    create project
    not
    s
    Amanda Wee
    2 replies
    Copy to Clipboard
  • Harshal Rane

    Harshal Rane

    1 year ago
  • Harshal Rane

    Harshal Rane

    1 year ago
    Hi everyone, i am trying out hybrid model approach of prefect, where i am using prefect.io cloud and i have created PAT token and using that generated a Runner token for Docker agent which is on the virtual machine in cloud. when i am scheduling my flow, the docker agent accepts job it pulls the agent image but flow fails with this error
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
    Attached are logs from Docker agent Has anyone faced this issue before?
    Harshal Rane
    j
    6 replies
    Copy to Clipboard
  • Michael Hadorn

    Michael Hadorn

    1 year ago
    Hi there Is there a way to use dynamic flows with the Prefect Server (Cloud or on-premise)? This register stuff, is not useful sometimes. With dynamic flow i mean: the tasks will be generated at runtime. Not possible with mapping, because they have dependencies. I already asked here, but nobody answered about this: https://prefect-community.slack.com/archives/C014Z8DPDSR/p1612986542265800 Thanks a lot for every help!
    Michael Hadorn
    ale
    +1
    12 replies
    Copy to Clipboard
  • j

    Jonathan Wright

    1 year ago
    Hi. I am interested in how Prefect approaches the concept of nested or sub flows. That is running a flow inside another. I’ve not found anything in the docs about this? Or is this a design pattern that Prefect discourages? Thank you.
    j
    Amanda Wee
    2 replies
    Copy to Clipboard
  • m

    Milly gupta

    1 year ago
    Hi, I am looking to deploy Prefect agent either on Kubernetes or Service Fabric? Does anyone know what are the resource requirements for Prefect Agent?
    m
    j
    11 replies
    Copy to Clipboard
  • Josselin Girault

    Josselin Girault

    1 year ago
    Hi. I'm curious about how Dask and Prefect behave together. If I have a dask.dataframe as the input of my prefect flow and want to both narrow and wide operations on the dataframe, how do I go about implementing it ? Since all prefect tasks are submitted as dask.delayed objects, I can't directly use the dask API on my dask.dataframe inside the tasks as it's unreliable to use dask calls in dask calls. If I make so the narrow operations are mapped tasks, how do I write a groupby efficiently ? The results of the mapped tasks will be reduced and the groupby will be done on a single node. Both Dask documentation (https://examples.dask.org/applications/prefect-etl.html) and Prefect documentation (https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html) do not answer this issue. I haven't found an example of how to use dask objects inside a prefect task, or how to implement wide operations in pure prefect. It seems that by using Dask as its engine, Prefect prevents the use of the Dask API to do transformations.
    Josselin Girault
    Chris White
    2 replies
    Copy to Clipboard
  • m

    Matthew Blau

    1 year ago
    Hello all, my boss has asked a question about Prefect that I don't quite know the answer to. We are using Prefect server and the question was: can we host Prefect server on 1 server, but have our program code, including the flows, on another server? I don't quite know whether or not that is something Prefect Server can do. We are wanting a dedicated server for Prefect, but the program code hosted elsewhere in our infrastructure
    m
    Chris White
    8 replies
    Copy to Clipboard
  • Puja Thacker

    Puja Thacker

    1 year ago
    @Kyle Moon-Wright #prefect-community -- I am new to Prefect.. Can someone help me understand the difference between Prefect agent and Prefect executors?
    Puja Thacker
    j
    4 replies
    Copy to Clipboard
  • Nikul

    Nikul

    1 year ago
    Hi, I'm running a mapped task over ~400 elements on Kubernetes using DaskExecutor + KubeCluster, but I quickly run out of memory. The data I'm using is <5GB and the nodes I'm using have ~60GB of RAM. The job pod (running the Dask scheduler) reaches >40GB memory usage just before the mapped task starts and the node runs out of memory before any of the mapped tasks start. I was wondering if anyone knows what the issue is. Thank you
    Nikul
    m
    4 replies
    Copy to Clipboard