• Edmondo Porcu

    Edmondo Porcu

    4 months ago
    Still coming to Databricks and I guess in general with Tasks. Databricks has added git support for jobs, and the current DatabricksSubmitMultitaskRun doesn't support it. I am in doubt among the possible approaches: • Create a DatabricksSubmitMultitaskRun custom implementation. However, I will access underscore methods in prefect package • Use the databricks CLI python library to create a job and then just run it via Prefect • Other? The real problem is that the Task does not allow dependency injection (i.e. the databricks client is created within the Run function, so it's not easy to override it). I guess the design of the Task is concerning in the sense that is not extensible, one needs to rewrite it from scratch
    Edmondo Porcu
    Kevin Kho
    +1
    20 replies
    Copy to Clipboard
  • Ben Ayers-Glassey

    Ben Ayers-Glassey

    4 months ago
    In old-school Prefect (not Orion), is there a workaround for adding build args (i.e. Dockerfile's ARG instead of ENV)? Currently Prefect's
    Docker
    class takes an
    env_vars
    dict, but no
    build_args
    dict, which would be nice. Here is the place in the
    Docker
    class where ENVs are generated:https://github.com/PrefectHQ/prefect/blob/8e04ccad071d1127afc7ca3580f1fe6f4e884f27/src/prefect/storage/docker.py#L437-L442 ...the ARGs could go right above there. In any case, ultimately what I'm trying to do is get the
    Docker
    class to install
    python_dependencies
    from our private PyPi server. Is there a good way to do that?
    Ben Ayers-Glassey
    Kevin Kho
    +1
    55 replies
    Copy to Clipboard
  • Ben Ayers-Glassey

    Ben Ayers-Glassey

    4 months ago
    Thread about loading flows from modules instead of .py files or cloudpickle-serialized blobs =>
    Ben Ayers-Glassey
    Anna Geller
    5 replies
    Copy to Clipboard
  • Raviraja Ganta

    Raviraja Ganta

    4 months ago
    I have registered the flows via GitHub Actions and the agent is there. When I run the flow, it looks like it got stuck. Can someone help me on debugging this? or what is happening here?
    Raviraja Ganta
    2 replies
    Copy to Clipboard
  • b

    Bernard Greyling

    4 months ago
    Morning All We are using prefect via a kubernetes agent. We've noticed that failed flows don't clean up prefect jobs and in some cases the dask-excecutor. Is there any passive mechanism to clean up these dead jobs ?
    etl                prefect-job-29a6d24a-hwnzx                         0/1     ImagePullBackOff             0          6d20h
    etl                prefect-job-30bd9ad7-zsc6z                         0/1     CreateContainerConfigError   0          11h
    etl                prefect-job-39468921-lt95t                         0/1     CreateContainerConfigError   0          6d20h
    etl                prefect-job-3c1aee81-hvbv4                         0/1     Completed                    0          7h49m
    etl                prefect-job-3c84d152-wfqw9                         0/1     Completed                    0          7h53m
    etl                prefect-job-3fb2397e-7dc5b                         0/1     ImagePullBackOff             0          11h
    etl                prefect-job-5ba165eb-rfxhq                         0/1     CreateContainerConfigError   0          6d20h
    etl                prefect-job-62ae93a3-ptbcr                         0/1     CreateContainerConfigError   0          6d20h
    etl                prefect-job-6cfd964c-p7xpl                         0/1     ImagePullBackOff             0          6d20h
    etl                prefect-job-86e07407-nmpwb                         0/1     ImagePullBackOff             0          6d20h
    etl                prefect-job-9478c771-bfg26                         0/1     ImagePullBackOff             0          11h
    etl                prefect-job-950ba45b-tv4f2                         0/1     CreateContainerConfigError   0          11h
    etl                prefect-job-a199b1d6-6vwmd                         0/1     Completed                    0          7h49m
    etl                prefect-job-a4976ce3-gxqv6                         0/1     Completed                    0          6h5m
    b
    Anna Geller
    +1
    7 replies
    Copy to Clipboard
  • Julian Brendel

    Julian Brendel

    4 months ago
    Good morning prefect community 😊. We are evaluating a switch to either Prefect or Dagster. On the surface the feature sets look quite similar. Do you have any resources aiding with the decision which solution to choose?
    Julian Brendel
    Anna Geller
    9 replies
    Copy to Clipboard
  • Ievgenii Martynenko

    Ievgenii Martynenko

    4 months ago
    Morning, we've set up new Prefect env in K8s, but some old dataflow have DockerRun() as run_config. These dataflows are failing with error: "Flow run 46a30aa1-11f0-4c77-bb56-afab879db8bc has a
    run_config
    of type
    DockerRun
    , only
    KubernetesRun
    is supported" Suppose that Prefect knows it runs in K8s and now all dataflow should use K8s agent, right?
    Ievgenii Martynenko
    2 replies
    Copy to Clipboard
  • m

    Mini Khosla

    4 months ago
    Hi All We are using prefect server 0.15.7 and running the flows on yarn Cluster. I have 20 mapped tasks and each mapped task loops over a list , does some processing and saves output to hadoop. I am getting the below error intermittently. Any help would be appreciated.
    m
    Anna Geller
    3 replies
    Copy to Clipboard
  • Raviraja Ganta

    Raviraja Ganta

    4 months ago
    While registering the flow it is not storing the configs file which is present in a different directory. But some modules needs them. How to solve this?
    main-project
      | 
      src
         |
         | __init__.py
         | module1.py
         | module2.py
      |
      configs
         | 
         | config.yaml
      |
      flows
         |
         | sample_flow.py
    Raviraja Ganta
    Anna Geller
    12 replies
    Copy to Clipboard
  • Rasmus Lindqvist

    Rasmus Lindqvist

    4 months ago
    Hi all, we are using
    prefect server
    for local development and
    prefect cloud
    for production which is quite neat in terms of speed. When doing this we set the --external-postgres flag in order to use our development database, which works great! However we are using Hasura for our other backend service which creates some git-pain as Prefect overwrites our local Hasura metadata files. Have anyone run into the same problem ?
    Rasmus Lindqvist
    Kevin Kho
    4 replies
    Copy to Clipboard