https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • g

    GitHub

    06/29/2019, 6:45 PM
    was added to this conversation
  • j

    Jeremiah

    06/29/2019, 7:30 PM
    Hey @Zach Angell 👋
  • z

    Zach Angell

    06/29/2019, 7:51 PM
    Hey @Jeremiah 👋
  • w

    wilsojb

    07/01/2019, 7:28 PM
    👋
    👋 1
  • j

    Jeremiah

    07/01/2019, 10:55 PM
    Welcome everyone! Sorry it’s a bit quiet while we get some things ready :)
    o
    • 2
    • 2
  • j

    Jay Leach

    07/02/2019, 4:23 AM
    Hello 🙂
  • c

    Chris White

    07/02/2019, 5:12 AM
    Hello @Jay Leach !
  • j

    Jamie Catherwood

    07/02/2019, 1:12 PM
    Is this thing on?
    📻 2
    o
    • 2
    • 1
  • j

    Jeremiah

    07/02/2019, 1:17 PM
    Jamie we were hoping you’d lead a conversation on the details of the execution model
    🤣 3
    c
    • 2
    • 1
  • j

    Jamie Catherwood

    07/02/2019, 1:18 PM

    https://media.tenor.com/images/02a2c2bd9eeafa9f15a1b072678ac59d/tenor.gif▾

    👏 3
    😂 2
  • d

    Dylan Baker

    07/02/2019, 3:16 PM
    Hello 👋 Am I in the right place? @Jeremiah’s Twitter feed led me to believe this is a place for high quality pun content.
    👋 2
    j
    o
    • 3
    • 6
  • c

    Chris Hart

    07/02/2019, 6:19 PM
    hey peeps, any preferences for organizing projects/namespaces based on prefect flows?
    j
    d
    +2
    • 5
    • 10
  • c

    Chris White

    07/02/2019, 9:55 PM
    Introducing, the official prefect party parrot ---> :marvin:
    😋 2
    😂 2
    :marvin: 4
    t
    • 2
    • 5
  • r

    Romain

    07/03/2019, 12:25 PM
    Hi folks, I am starting to consider using Prefect in a project, but one thing remains to be clarified : is that currently possible to specify the worker ressource requirement for a task running on a Dask cluster? E.g.: I have a flow with two tasks A and B. I would like to specify that A should run only on GPU workers while B should run on NO GPU workers. I did not find anything about that in the docs.
    o
    c
    +2
    • 5
    • 9
  • a

    Alistair Hey

    07/05/2019, 12:28 PM
    Im trying to use Prefect with a dask kubernetes cluster (installed with helm stable/dask) - however whenever i install prefect on the dask worker pods it upgrades distributed and then the cluster falls apart - any one got prefect working in this way?
    c
    m
    • 3
    • 15
  • j

    Joselin

    07/08/2019, 6:58 AM
    With_kwargs_as_params.py
  • j

    Joselin

    07/08/2019, 6:58 AM
    Hi, i am using imperative API's. in need of a clarification on using kwargs in overridden Task class run method. Am using two classes and i want to pass kwargs to one and the modified kwargs to the next task. Am not sure if i had done right in putting pieces together but am getting "TypeError: too many positional arguments" from the second class's run method. I have attached the code as well. Thanks in Advance
    c
    m
    • 3
    • 8
  • c

    Chris Hart

    07/08/2019, 7:00 PM
    heyyo general design question about boundaries between tasks and level of separation of concerns and idempotency... I am writing my first project with Prefect and have come up on a situation where I want to pre-process a bunch of data in one task before persisting it in another task
  • c

    Chris Hart

    07/08/2019, 7:02 PM
    but in the task 2, I need to do post-processing on the selfsame data structure as outputted by task 1.. in airflow it would seem this is a no-no without persisting it somewhere first.. I guess just wondering: is there is a general preference for mixing purposes and keeping tasks orthogonal?
    j
    • 2
    • 4
  • d

    David Norrish

    07/09/2019, 12:17 AM
    Hey all! A question I couldn't find addressed in the docs: is there a way to run a flow from a particular "entry point"? E.g. if you have a flow with sequential tasks A -> B -> C -> D, could you invoke the flow from task B or C? A concrete use case is a data processing pipeline where task A is time-consuming and most of the time doesn't need to be run.
    j
    c
    m
    • 4
    • 18
  • j

    Jason

    07/09/2019, 12:37 PM
    @Chris White Regarding state and metadata (e.g. log of task runs), you mentioned a while back that Prefect Cloud will have this setup by default. If we wanted to roll our own DB, how custom is this kind of setup and is there any Airflow-like equivalent to setting up your "metadata db" with a script?
    c
    • 2
    • 10
  • s

    Sherman

    07/09/2019, 4:36 PM
    Hello, I am running through the Docker Prefect example (https://docs.prefect.io/guide/examples/imperative_docker.html) and it says that it is using the Imperative API to build out the example. Is there a reason why the Functional API couldn't be used for this purpose?
    c
    • 2
    • 6
  • d

    David Ojeda

    07/10/2019, 3:58 PM
    Hi there, I was trying to get the dask scheduler UI to be a bit more expressive. Normally, I have all my running tasks grouped under the run_task and run_fn names, but what I would like is to have the real task name. I managed to extend
    prefect.engine.executors.DaskExecutor
    to add the task name as the
    key
    parameter sent to the dask client:
    import uuid
    from typing import Any, Callable
    
    from distributed import Future
    from prefect.engine.executors import DaskExecutor
    
    
    class MyDaskExecutor(DaskExecutor):
    
        def submit(self, fn: Callable, *args: Any, **kwargs: Any) -> Future:
            if 'key' not in kwargs and 'task' in kwargs:
                kwargs['key'] = kwargs['task'].__class__.__name__ + '-' + str(uuid.uuid4())
            return super().submit(fn, *args, **kwargs)
    So far this is working as expected, with the exception of tasks that use the
    .map(...)
    idiom, where the run_fn still appears with that name. I have looked further into prefect’s code but it seems that to achieve this I would need to create my own task runner. Since that seems a bit complicated, I just wanted to ask if there is another solution or if this is just a bad idea?
  • c

    Chris White

    07/10/2019, 3:59 PM
    this is a super interesting idea! I love it
  • c

    Chris White

    07/10/2019, 4:00 PM
    for mapping, you should look at the
    map
    method of the
    DaskExecutor
    -> when we map tasks, we use
    executor.map
    instead of
    executor.submit
  • d

    David Ojeda

    07/10/2019, 4:01 PM
    I tried that, but the thing is that
    DaskExecutor.map
    does not receive
    kwargs
    , and I am using the “task” kwarg to guess the task name.
  • c

    Chris White

    07/10/2019, 4:03 PM
    ah i see
  • c

    Chris White

    07/10/2019, 4:04 PM
    ah
  • c

    Chris White

    07/10/2019, 4:04 PM
    inside of
    DaskExecutor.map
    , try:
    task_name = prefect.context.get("task_full_name", "some-default-value")
  • d

    David Ojeda

    07/10/2019, 4:13 PM
    👍 That works for me ! Thanks! My only regret is that my custom executor now repeats the
    DaskExecutor
    map code, because I cannot do
    super().map(...)
    given that the executor does not admit kwargs. I can live with that, but I will have to keep an eye on any new changes on that class on future versions.
Powered by Linen
Title
d

David Ojeda

07/10/2019, 4:13 PM
👍 That works for me ! Thanks! My only regret is that my custom executor now repeats the
DaskExecutor
map code, because I cannot do
super().map(...)
given that the executor does not admit kwargs. I can live with that, but I will have to keep an eye on any new changes on that class on future versions.
View count: 46