https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • j

    Joe Schmid

    12/04/2019, 3:18 PM
    I found this talk on Dask interesting and thought the Prefect community might, too. (Even though the title is "Is Spark still relevant?" it's really about Dask.) As Prefect and Dask users, we're very interested in things that drive enterprise Dask adoption: 

    https://www.youtube.com/watch?v=obKZzFRNTxo▾

    :upvote: 6
  • b

    Ben Doremus

    12/12/2019, 8:51 PM
    Ya’ll have pretty awesome customer service. Just saying.
    ➕ 6
    😁 5
    :prefect: 6
    :marvin: 6
    z
    • 2
    • 1
  • m

    Mark McDonald

    12/18/2019, 2:04 AM
    The new logs UI looks amazing! Nice work!
    :marvin: 8
    💯 3
    🙌 3
  • b

    Braun Reyes

    12/26/2019, 10:51 PM
    Typer https://typer.tiangolo.com/ Interesting cli library built on top of click
    👍 1
    👀 2
    a
    • 2
    • 1
  • j

    Jeremiah

    12/26/2019, 10:51 PM
    👀
  • j

    Jeremiah

    12/26/2019, 10:52 PM
    We use FastAPI / Starlette in Cloud and it’s been great - looks like this is from the same people!
  • r

    rich

    01/13/2020, 2:47 PM
    https://www.nomadproject.io/
    :upvote: 2
    c
    • 2
    • 2
  • v

    Vitor Avancini

    01/14/2020, 9:06 PM
    Hello y'all, will the UI be open sourced at somepoint?
    n
    d
    a
    • 4
    • 4
  • c

    Chris Eberly

    01/15/2020, 4:14 AM
    unrelated to the product but related to that last comment, thank you guys for drawing my attention to MagicMock() in your incredible tests, saving me hours of frustration :)
    :success-kid: 7
    c
    j
    f
    • 4
    • 4
  • b

    Braun Reyes

    01/29/2020, 3:19 PM
    will be interesting to see how something like https://www.python.org/dev/peps/pep-0554/ affects Dask if implemented
  • a

    Aiden Price

    01/30/2020, 12:45 AM
    I did remember hearing on either Python Bytes or Talk Python To Me (can’t remember which or which episode sorry) that it would actually be helpful for their scheduler, but couldn’t replace their cluster computing stuff.
  • a

    Alex Cano

    02/02/2020, 12:20 AM
    Hey Prefect team, was curious if you guys can share any insights onto how the cloud platform is tested (specifically the GQL interface). I’m building a demo app with GQL for the first time and realized it could be a lot different testing the thing. I’m specifically struggling with getting my head around testing the almost unlimited number of layers of a call. I’ve been thinking about testing the relationships that should be there just to make sure everything works okay. For example against the Prefect API:
    query {
        flow {
            name
            task {
                slug
            }
        }
    }
    vs
    query {
        flow {
            name
        }
    }
  • z

    Zachary Hughes

    02/02/2020, 12:33 AM
    Hi @Alex Cano, great question! 100% agree that there's a lot of surface area to test, here are some tips from what we do: • we use Hasura to manage all of queries. This is helpful because it handles a lot of the grunt work for us, but we still need to test it. To that end, we have tests to ensure each of our entities (for example, flows and flow runs) are properly relating to each other, that they're displaying the proper fields, and that they can only be viewed by the appropriate users. • beyond what's handled by Hasura, we have several additional layers of testing. We test to ensure that we're surfacing the correct fields, that we sanitize and handle erroneous input, that any non-Hasura permissions are correctly applied, our models look like we expect them to, etc. We start small with unit tests for each individual model/piece of functionality, then work our way up to larger integration tests. (And
    pytest.mark.parametrize
    is definitely our friend) TL;DR Test that fields are being surfaced as expected, test object relationships, and test permissions. Happy to go deeper/broader if you'd like (we have an inordinate number of tests), but these were the items I found most useful when starting out with GraphQL.
    :upvote: 4
    :prefect: 1
    👨‍🎓 4
    a
    • 2
    • 1
  • b

    bardovv

    03/18/2020, 6:57 PM
    Is there any open source alternative to prefect cloud?
    a
    p
    d
    • 4
    • 3
  • a

    Alex Cano

    03/25/2020, 8:10 PM
    So…. anyone read any good blog posts lately 👀 👀
    😂 1
    l
    b
    • 3
    • 6
  • i

    itay livni

    03/26/2020, 4:05 AM
    Hi - How does one address the 9.8 User warning directly without suppressing warnings?
    UserWarning: DEPRECATED: the result_handler Task option will be deprecated in 0.11.0, and removed in 0.12.0, in favor of the `result` option instead.
      UserWarning,
    c
    • 2
    • 1
  • a

    Armin

    03/27/2020, 8:50 AM
    Hi, I'm having trouble checkpointing a task. Given a task definition:
    class SomeTask(Task):
     def __init__(self, **kwargs):
      super().__init__(**kwargs)
      # SomeTask specific init code
     def run(self, input):
      output = process(input)
      return output
    And flow:
    some_task = SomeTask(checkpoint=True, output_handler=LocalResultHandler(dir=cache_location))
    
    def some_flow():
     with Flow():
      output_handler(input)
    
    if __name__ == '__main__':
     some_flow()
    The
    cache_location
    remains empty and on multiple runs of the flow (with same input to
    some_task
    ) the result is recomputed every time. Could somebody please suggest what am I missing?
    c
    • 2
    • 12
  • z

    Zach

    04/01/2020, 4:20 PM
    Can I get the output of a result_handler on a task from within a state_handler on that same task?
    c
    • 2
    • 3
  • t

    Thoth Jedi

    04/03/2020, 12:37 AM
    Is there any NO-docker version of prefect to install on-prem (we don't have any docker support)? I did
    conda install
    and now getting this
    No such file or directory: 'docker-compose'
    for
    prefect server
    .
    c
    s
    • 3
    • 4
  • o

    ocet

    04/07/2020, 7:37 PM
    Hello. How can I trigger Prefect Flows programmatically? Is there any Prefect Server API I could call? I'm looking to implement a case similar to one described in https://docs.prefect.io/core/PINs/PIN-14-Listener-Flows-2.html
    j
    • 2
    • 3
  • a

    Alex Cano

    04/07/2020, 9:00 PM
    Truly a random question, but I’d like people who know more about asyncio to chime in. I’ve had experience in path building things using it, which is all fine and dandy. I’m curious if there’s any downside to having everything execute within the asyncio event loop, even if you know things are going to be blocking. Like I know you wouldn’t gain any of the advantages of asyncio, but I was just thinking kind of bigger picture. I’m sure there’s some overhead with running code inside an event loop, but I don’t know enough on the topic to know whether it is significant enough to prevent ^.
    j
    • 2
    • 7
  • k

    Kostas Chalikias

    04/08/2020, 10:25 AM
    I imagine that if you never actually await, there won't be a performance hit outside of getting that first entry point function called by the loop. However you will have much longer stack traces for no good reason and will just generally cause confusion as to what your intentions are?
  • l

    Laura Lorenz (she/her)

    04/08/2020, 1:51 PM
    @Alex Cano your Q reminded me of this slide from a talk I saw last year at djangocon, which plots some empirical data on the case of gunicorn vs twisted requests processed/second on localhost... so a similar situation where requesting localhost/low latency can’t leverage async much at all so the difference is mostly from overhead this chart is from

    https://www.youtube.com/watch?v=1LwKKyyATvo▾

    at 31:32. this whole talk goes pretty seriously into async and the whys and whens so the whole thing might be of interest to you if you haven’t seen it already
    :upvote: 2
  • l

    Leo Meyerovich (Graphistry)

    04/08/2020, 3:59 PM
    ^^^ yeah we run asyncio http for non-prefect via a multiprocessing setup -- multiple processes, quart instead of flask, etc -- b/c of ^^^. so it matters as soon as you have parallel latency-sensitive tasks. without, something like a pandas call on main thread will kill avg session latency, even if not for an indiv request. but no idea wrt prefect; we were advised to batch tasks to 30s-1min level so don't care as much here.
  • a

    Alex Cano

    04/08/2020, 4:13 PM
    Yeah definitely. I’ve built out and am maintaining a Fast API app for work, so I’m definitely familiar with trying to keep as much async as possible when you’re already living in async world. It makes you very conscious of your blocking calls and where it’s “correct” to put them. We’ve been using aiohttp, but we’ve heard good things about httpx as well. Mirroring the requests API is really helpful because it can get other devs on board faster
    s
    j
    • 3
    • 5
  • a

    An Hoang

    04/13/2020, 4:15 PM
    Hi everyone, does anyone know of a way to visualize the relationship between pandas dataframes into a Directed Acyclic Graph? Like
    df_B
    is obtained from
    df_A
    by passing it through some function
    A_to_B_func()
    .
    df_D
    is made by merging
    df_B
    with
    df_C
    etc. If it’s a complex function, yes we can write prefect Tasks to visualize it, but usually it’s an aggregate of small pandas functions and merge, so it would be a pain to wrap each of those into its own prefect task. Maybe I would need to write a complex context manager/decorator or a meta class wrapper around dataframe to record these operations automatically into a prefect task graph? Sorry the idea is not so clear in my head right now…
    a
    l
    • 3
    • 6
  • l

    Leo Meyerovich (Graphistry)

    04/13/2020, 5:08 PM
    @An Hoang email leo@graphistry.com -- can get you going w/ https://github.com/graphistry/pygraphistry
  • l

    Leo Meyerovich (Graphistry)

    04/13/2020, 5:08 PM
    can handle pretty fun-sized graphs 🙂
  • l

    Lashwin

    04/16/2020, 12:13 AM
    Hi, I see that secrets can be used by anyone within the team. Is there any way of locking down secrets per project or per role?
    j
    • 2
    • 4
  • j

    Jacob Blanco

    04/16/2020, 1:22 PM
    What's the right way to load auxiliary non-Python files inside of a module? I can use
    __file__
    and "navigate" from there, but I feel like that's not the right way.
    d
    • 2
    • 3
Powered by Linen
Title
j

Jacob Blanco

04/16/2020, 1:22 PM
What's the right way to load auxiliary non-Python files inside of a module? I can use
__file__
and "navigate" from there, but I feel like that's not the right way.
d

Dan DiPasquo

04/16/2020, 4:40 PM
https://pypi.org/project/importlib-resources/ is an alternative that I've found useful.
allows usage e.g.
with importlib.resources.path('my.package.resource_dir', config_file) as config_path:
🙇‍♂️ 1
j

Jacob Blanco

04/17/2020, 1:30 AM
oh awesome thanks!
View count: 3