• a

    Andrey Tatarinov

    5 months ago
    Hi, we're looking into ways to reduce latency between Flow Run submission and getting results back. At the moment we're using KubernetesAgent which spawns k8s Jobs for each flow run. Job initialization is quite slow in our case due to image size. Question: if we setup permanent Dask cluster with appropriate image and set executor to DaskExecutor - would we be able to skip Job initialization step? i.e. is it Agent who sends specific commands to Dask cluster or k8s Job cannot be avoided?
    a
    Kevin Kho
    +1
    14 replies
    Copy to Clipboard
  • Hafsa Junaid

    Hafsa Junaid

    5 months ago
    Hey team! I am trying to deploy prefect-server on gke At this point--> vim ~/.prefect/config.toml I entered apollo IP and pressing esc-> :wq or esc-->:x ... Then I am getting the following message. Any guide on how to deal with it?
    Hafsa Junaid
    Kevin Kho
    4 replies
    Copy to Clipboard
  • Hafsa Junaid

    Hafsa Junaid

    5 months ago
    Hey.. I am registering my task on prefect-server to demo project. The task has spark and sparks context. The flow.run() is successful but flow.register() gives following errorRuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063. Any guide on this spark context setting.
    Hafsa Junaid
    Rhys Mansal
    2 replies
    Copy to Clipboard
  • Dave

    Dave

    5 months ago
    Hello, Have anyone experienced problems when using:
    prefect.engine.signals.LOOP
    , where the task is suddenly is missing the context on retries. e.g:
    index = prefect.context.get('task_loop_result', {}).get('index', 0)
    See more code in the thread 🧵
    Dave
    Anna Geller
    6 replies
    Copy to Clipboard
  • Faheem Khan

    Faheem Khan

    5 months ago
    Hi, I'm running local prefect server in docker, and to register my flow I run a docker client. I am using this method(https://github.com/flavienbwk/prefect-docker-compose). I can see the code run successifully in docker container client and flow is registered but when I rerun the flow from UI I getting no module Pandas found. I have Pandas installed on host and also in the client container, my question is where else should the module be installed for the flow to run successfully from the UI?
    Faheem Khan
    Anna Geller
    +1
    3 replies
    Copy to Clipboard
  • Rhys Mansal

    Rhys Mansal

    5 months ago
    Hi all, I'm having difficulty with the following code.
    param_a = flow.add_task(Parameter("param_a", default=True))
    with case(param_a, False):
        task_a = some_task(arg)
    task_b = some_task(task_a)
    some_task
    is being run multiple times with different arguments. When
    param_a
    is false some of these should be skipped. If I do not use
    @task(skip_on_upstream_skip=False)
    in the task decorator on
    some_task
    then all downstream tasks from the first skipped task are skipped, whether they are inside a case or not. If I do, none of them are skipped regardless of what
    param_a
    is set to. Does anyone have any idea how to get only the tasks inside the with block to skip (and only when
    param_a
    is true)?
    Rhys Mansal
    Anna Geller
    2 replies
    Copy to Clipboard
  • Florian Guily

    Florian Guily

    5 months ago
    Hey guys, i have a quick question on good practice. How scalable is the mapping task ? are the children tasks "queued" if the map is pretty big ?
    Florian Guily
    emre
    11 replies
    Copy to Clipboard
  • x

    xyzz

    5 months ago
    Regarding storage the Orion docs say "you can also configure a self-hosted key-value store for testing.".
    But what kind of key-value store is compatible and how to set it up for testing purposes?
  • x

    xyzz

    5 months ago
    I'd like to test orion cloud together with kubernetes running on my laptop via docker
  • x

    xyzz

    5 months ago
    would redis (https://hub.docker.com/r/ubuntu/redis) be compatible?
    x
    Anna Geller
    +1
    18 replies
    Copy to Clipboard