https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-contributors
  • s

    Shaun Cutts

    08/24/2020, 3:36 AM
    Implementing experimental server helm chart; question 5 continued (“5b”) -- Now overriding values via env settings in graphql and towel containers — eg: PREFECT__API__URL PREFECT__DATABASE__HOST PREFECT__DATABASE__PORT PREFECT__DATABASE__USERNAME PREFECT__DATABASE__PASSWORD PREFECT__HASURA__HOST PREFECT__HASURA__PORT EDIT — should these be “PREFECT_SERVER__…” rather than “PREFECT__…“? --- I can feed some values through to chart “values.yaml” — “queued_runs_returned_limit”, etc. --- Question 5c) But what about this:
    # deploy environment, which impacts default values for some settings.
    # Make sure this is lower case because config keys are lower case.
    env = “local”
    Where are the defaults that it impacts, and should I use (/create) a new
    env
    setting? (“kubernetes” — for instance?) Question 5d) What about the config.toml [services] block? With “host=0.0.0.0”, is this outbound config? I was thinking that this is config for graphql and for towel. Is it config for apollo to? Do I eg. need to set PREFECT__SERVICES__APOLLO__PORT in the apollo container? … hmm… afaikt … this isn’t read there. Is this then obsolete config? Or in fact inbound config? EDIT — so apollo env in docker-compose has
    PREFECT_SERVER__TELEMETRY__ENABLED: ${PREFECT_SERVER__TELEMETRY__ENABLED:-true}
    . Is there something in the apollo that is reading the toml and these interpolations? (If so, why use different form for HASURA etc here?)
  • s

    Shaun Cutts

    08/25/2020, 1:06 AM
    Implementing experimental server helm chart; making some progress. I have db initializing and most services starting properly — but not apollo:
    {"status":"ok","version":"UNKNOWN"}
    GraphQL service healthy!
    
    > @ serve /apollo
    > node dist/index.js
    
    2020-08-25T00:43:27.136Z Building schema...
    2020-08-25T00:43:27.155Z Error: Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"}
      at getSchemaFromIntrospection (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:981:15)
      at introspectSchema (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:990:12)
      at process._tickCallback (internal/process/next_tick.js:68:7) Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"} Error: Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"}
      at getSchemaFromIntrospection (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:981:15)
      at introspectSchema (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:990:12)
      at process._tickCallback (internal/process/next_tick.js:68:7)
    2020-08-25T00:43:27.155Z 
    Trying again in 3 seconds...
  • s

    Shaun Cutts

    08/25/2020, 1:11 AM
    I’m not sure if its environment setup issue; I have:
    - name: HASURA_API_URL
         value: <http://prefect-server-hasura.prefect-server.svc.cluster.local:3000/v1alpha/graphql>
        - name: PREFECT_API_URL
         value: <http://prefect-server-graphql.prefect-server.svc.cluster.local:4201/graphql/>
        - name: PREFECT_API_HEALTH_URL
         value: <http://prefect-server-graphql.prefect-server.svc.cluster.local:4201/health>
        - name: GRAPHQL_SERVICE_HOST
         value: <http://prefect-server-graphql.prefect-server.svc.cluster.local>
        - name: GRAPHQL_SERVICE_PORT
         value: "4201"
    which I think are correct:
    prefect-server-graphql        ClusterIP  172.20.137.57  <none>    4201/TCP  38m
       prefect-server-hasura        ClusterIP  172.20.185.76  <none>    3000/TCP  38m
    Do I also need PREFECT_SERVER__* settings here? Any other debugging info? chart code is in https://github.com/PrefectHQ/server/pull/57 @Chris White @Jeremiah — any thoughts?
    c
    j
    • 3
    • 7
  • s

    Shaun Cutts

    09/02/2020, 4:50 PM
    @josh — Wanted to get your opinion on how to integrate auth into the helm chart. What I’m planning is to allow the chart to take configuration for extra containers in the apollo and ui pods, which could run auth proxies, and fiddle with the port options so that we could bypass auth internally. (Forwarding scopes around, etc I guess I can wait on. :)). I would try to configure with louketo-proxy, as we have keycloak running, but should be possible to run other proxies as well. Does that sound like a good idea, or would you suggest something else?
    j
    • 2
    • 2
  • i

    itay livni

    09/03/2020, 6:09 PM
    https://github.com/PrefectHQ/prefect/discussions/3257
  • m

    Michael Ludwig

    09/05/2020, 10:20 PM
    Hi all, I am would be curious if you at one point maybe thought about providing a completly serverless Prefect agent option? We currently had one tiny service in ECS Fargate with one container running the Fargate agent around the clock. As I am very much a serverless fan and I might only have a daily scheduled flow I thought it would be nice to only activate the agent when needed or even have a push-based system. On AWS side I would see two options e.g. a Lambda function which is triggered via a webhook over AWS Api Gateway or even a AWS SNS webhook. Alternatively if push is not possible I could imagine a poll-based cron scheduled lambda to check for flow runs. Maximum frequency is unfortunately only one minute but might be ok for certain cases but that would be anyways not really “on-demand”. To my knowledge the agent is anyways doing the flow runs in fire and forget style therefore I thought this could be quite doable. Anybody with more insights to the agent having an opinion or did think about something similiar already? 🙂 (I hope it is not wrong to post this question in this channel 😄 )
    j
    • 2
    • 4
  • r

    Robin

    09/08/2020, 8:17 AM
    Dear prefects, I have created a pull request that is rather just a quick fix (just adding kwargs to a run method). Two questions: 1. Is this the right approach to suggest quick fixes? 2. I get an error that
    import prefect
    takes more than 1s. However, this can't be due to my small changes, can it? Background: I had problems with the
    NoRegionError
    for the boto client. It seems like setting the
    region_name
    as kwarg of the secret manager's run method is the most elegant way of solving this issue. Therefore, I added the kwargs to the run method.
    :upvote: 3
    ✔️ 1
    j
    • 2
    • 2
  • a

    Alex Cano

    09/10/2020, 8:27 PM
    Howdy friends… what are
    flow_groups
    ? I can’t seem to find docs on what they exactly are. As a guess, they look like they can be used to apply settings across many flows? I’m specifically thinking about
    flow_groups
    with regards to implementing Flow Concurrency Limits into the new server codebase, so it’s looking like (based on some other code) we look at the
    flow_group
    labels first, then default to the
    environment
    labels? Is that how it should be implemented for this as well? Or should it be checking both sets of labels? I’d guess both sets, but not sure
    d
    n
    • 3
    • 8
  • s

    Shaun Cutts

    09/12/2020, 1:15 AM
    # Co-routine tasks I was considering creating a “PIN” (or two) concerning “co-routine” tasks. I thought I’d write here first to get some initial feedback. --- A co-routine task runs corresponds to a co-routine in python. It runs multiple times, consuming marked parameters, before optionally transitioning back to a pending state, ready to run again if/when new versions of the parameters are available. It may be implemented in python as an actual co-routine, but also could be implemented using signals. ## Use cases ### Loops over complex sub-graphs in flows. Currently, it is impossible to loop over a subgraph of a flow. A “co-routine” is a primitive that could function as a building block for while loops with complex substructure. Tasks downstream of a “loop controller” co-routine task would run after each execution of the loop body, but could have edges that fed parameters back up to the loop control task. ### Processing external streams The current architecture is not well suited to processing large event streams -- such as a Kafka or NATS stream that could have 100s or 1000s of events per second. Processing such streams should not have to spin up an individual flow per event; but at the same time, it should not be limited to a single long-running event that handles all processing internally. A co-routine task can be triggered for each event, waiting in a pending state until an event arrives. ## Implementation Both internal “loop back” edges, and external edges, would be specially marked as co-routine inputs, so they could be consumed per-iteration, and waited on for the next iteration. A Pending->Coroutine State would be added to represent running coroutines. Internally, co-routine tasks could use dask pub/sub. For external links, the server/cloud could provide a webhook. As a further extension, this could optionally support upgrade to a websocket; also as further extensions, connectors to standard event queues could be provided. --- I like to think of Prefect as “an OS for FAAS processes” — or at least FAAS written in python and running in Dask. I don’t know to what extent this overlaps with your vision? Co-routine tasks could be used as a primitive to create complex flow processes containing event loops. They would be a big jump toward generalizing what Prefect could do.
    :upvote: 1
    👀 2
    j
    • 2
    • 3
  • i

    itay livni

    09/15/2020, 8:45 PM
    https://github.com/PrefectHQ/prefect/discussions/3321
  • a

    alex

    09/18/2020, 12:50 AM
    Hello! I have a flow as follows:
    results = []
    for op in operations:
      op = task(op)
      most_recent = op
      
      .. a few more conditional tasks defined at compile time
     
      if do_recovery: 
         recovery_task = recover(trigger=any_failed_else_raise_skip_signal)
         most_recent = recovery_task
      results.append(most_recent)
    
    backup_task = Backup('mydatabasename', upstream_tasks=[results], skip_on_upstream_skip=False)
    mapped = Aggregate.map(
                results,
                target=unmapped('mydatabasename')
    )
    mapped.set_upstream([backup_task])
    As you can see from the image, instead of getting one list, I have 2 lists and due to this, my Aggregate task is being skipped even though it is downstream of a
    skip_on_upstream_skip
    task. Is there a way to cleanly resolve this (ie. only get 1 list) ?
    n
    • 2
    • 5
  • a

    Alex Cano

    09/22/2020, 7:04 PM
    Hey yall, question for ya: What is the
    labeler
    in github actions? It looks like the action is failing on this PR: https://github.com/PrefectHQ/server/actions/runs/266743999
    c
    • 2
    • 15
  • t

    Tenzin Choedak

    10/03/2020, 11:08 PM
    Hi there 👋 In an attempt to get more familiar with Prefect, I thought I’d look at open issues and just add what little I can. I fixed an issue but I am wondering what the standard way to contribute is. Following the development guide, I’m just at the last step of opening a pull request from my forked repo back into Prefect. Do I make the base branch
    master
    or is there a different branch I should use? For context i’m fixing issue 3318 and my fixes are in my branch.
    c
    • 2
    • 3
  • m

    max

    10/06/2020, 6:53 PM
    hey everyone, my name is max (madelgi on github). i've been contributing to prefect off and on for the past month, mostly just adding tasks that have been useful for my purposes at my current company, but i've enjoyed contributing and i'm looking to work on something maybe slightly more substantial to get familiar with other parts of the code base. i've considered a couple of options -- adding a new storage option, maybe this issue, about persisting results in memory, etc
    👋 1
  • m

    max

    10/06/2020, 6:53 PM
    i just wanted to drop a message in here before i start on anything to see if any of the folks @ prefect have an opinion on something they'd like done. e.g., if there's something that's been on the backlog for a while that isn't necessarily super difficult but no one wants to do it, well, feel free to share it with me haha. no worries if no one has any suggestions or requests, i'll just choose some open issue that looks interesting
    j
    j
    • 3
    • 8
  • a

    ale

    10/08/2020, 4:23 PM
    Hi folks, I’m currently working on improving S3List to support
    last_modified_start
    and
    last_modified_end
    to leverage the
    last_modified
    property of keys, to make it possible to filter keys based on time. I’m struggling to find a way to mock boto3 put_object to write a meaningful test. Any suggestions?
    d
    • 2
    • 6
  • l

    Laura Lorenz

    10/15/2020, 9:10 PM
    set the channel topic: Core Contributor Cantina 11/6 @ 4pm EST (meet.google.com/aqv-gxru-zsi)
    👋 8
    :marvin: 1
  • b

    Braun Reyes

    10/26/2020, 7:20 PM
    Hello yall! Long time no contribute....we were wanted to make a couple enhancements to the fargate agent. 1 - add an
    extra_containers
    argument to support adding sidecars to the flow run container 2 - add the ability to skip the task definition registration based a run context key of
    task_arn_override
    . This would allow folks using a map to submit flow run(subflow pattern) to skip the task registration issue that @Darragh and @Lukas were running into
    🚀 2
    👀 2
    d
    j
    +2
    • 5
    • 36
  • b

    Braun Reyes

    10/26/2020, 7:20 PM
    number 1 allows you to add a datadog agent to your flows
  • b

    Braun Reyes

    10/26/2020, 7:22 PM
    number 2 would require a special run context....Is there a schema example of how the run context is presented to the agent as
    GraphQLResult
    ?
  • b

    Braun Reyes

    10/26/2020, 7:37 PM
    figured it out
    serialized_state.context
  • l

    Lonneke

    11/02/2020, 9:24 AM
    But also, one thing we'd really like is a user written example on how you use and deploy prefect to do real work! (Realize that's not quite what you were asking about so no worries if you'd rather focus on adding to the code!). We also really welcome additions and contributions to the docs!
    @Jenny This is exactly what I was trying to find, do you know if anyone else started on this?
  • j

    Jenny

    11/02/2020, 2:18 PM
    Hi @Lonneke and welcome! I don't think we have that exactly yet but our cause studies might be helpful here: https://www.prefect.io/why-prefect/case-studies Our twitter account is also a good place to check out what has been written about Perfect: https://twitter.com/PrefectIO?s=09
    l
    • 2
    • 2
  • h

    Henry

    11/05/2020, 5:38 PM
    I noticed a spelling mistake on the DaskExecutor code in the docs
  • h

    Henry

    11/05/2020, 5:38 PM
    do you have a github repo i can post a correction to?
    n
    • 2
    • 1
  • l

    Laura Lorenz

    11/06/2020, 8:48 PM
    Hey contribs! I need to reschedule our cantina that is supposed to start in 10 mins 😬 Look out for the next invite instead!
    👍 2
  • l

    Laura Lorenz

    11/06/2020, 8:49 PM
    set the channel topic: Core Contributor Cantina reschedule TBD
  • l

    Laura Lorenz

    11/09/2020, 9:15 PM
    set the channel topic: Core Contributor Cantina 11/20 4pm ET / 1pm PT (meet.google.com/qxg-towp-ymx)
  • l

    Laura Lorenz

    11/09/2020, 9:15 PM
    ^ so sorry about that last friday contribs, but looking forward to seeing you all 11/20 🙂
    😄 3
  • l

    Laura Lorenz

    11/20/2020, 9:12 PM
    My connection dropped! Be right back
Powered by Linen
Title
l

Laura Lorenz

11/20/2020, 9:12 PM
My connection dropped! Be right back
View count: 2