Implementing experimental server helm chart; question 5 continued (“5b”) -- Now overriding values via env settings in graphql and towel containers — eg: PREFECT__API__URL PREFECT__DATABASE__HOST PREFECT__DATABASE__PORT PREFECT__DATABASE__USERNAME PREFECT__DATABASE__PASSWORD PREFECT__HASURA__HOST PREFECT__HASURA__PORT EDIT — should these be “PREFECT_SERVER__…” rather than “PREFECT__…“? --- I can feed some values through to chart “values.yaml” — “queued_runs_returned_limit”, etc. --- Question 5c) But what about this:

# deploy environment, which impacts default values for some settings.

# Make sure this is lower case because config keys are lower case.

env = “local”

Where are the defaults that it impacts, and should I use (/create) a new

env

setting? (“kubernetes” — for instance?) Question 5d) What about the config.toml [services] block? With “host=0.0.0.0”, is this outbound config? I was thinking that this is config for graphql and for towel. Is it config for apollo to? Do I eg. need to set PREFECT__SERVICES__APOLLO__PORT in the apollo container? … hmm… afaikt … this isn’t read there. Is this then obsolete config? Or in fact inbound config? EDIT — so apollo env in docker-compose has

PREFECT_SERVER__TELEMETRY__ENABLED: ${PREFECT_SERVER__TELEMETRY__ENABLED:-true}

. Is there something in the apollo that is reading the toml and these interpolations? (If so, why use different form for HASURA etc here?)

Implementing experimental server helm chart; making some progress. I have db initializing and most services starting properly — but not apollo:

{"status":"ok","version":"UNKNOWN"}
GraphQL service healthy!

> @ serve /apollo
> node dist/index.js

2020-08-25T00:43:27.136Z Building schema...
2020-08-25T00:43:27.155Z Error: Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"}
  at getSchemaFromIntrospection (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:981:15)
  at introspectSchema (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:990:12)
  at process._tickCallback (internal/process/next_tick.js:68:7) Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"} Error: Could not obtain introspection result, received: {"path":"$","error":"resource does not exist","code":"not-found"}
  at getSchemaFromIntrospection (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:981:15)
  at introspectSchema (/apollo/node_modules/@graphql-tools/wrap/index.cjs.js:990:12)
  at process._tickCallback (internal/process/next_tick.js:68:7)
2020-08-25T00:43:27.155Z 
Trying again in 3 seconds...

I’m not sure if its environment setup issue; I have:

- name: HASURA_API_URL
     value: <http://prefect-server-hasura.prefect-server.svc.cluster.local:3000/v1alpha/graphql>
    - name: PREFECT_API_URL
     value: <http://prefect-server-graphql.prefect-server.svc.cluster.local:4201/graphql/>
    - name: PREFECT_API_HEALTH_URL
     value: <http://prefect-server-graphql.prefect-server.svc.cluster.local:4201/health>
    - name: GRAPHQL_SERVICE_HOST
     value: <http://prefect-server-graphql.prefect-server.svc.cluster.local>
    - name: GRAPHQL_SERVICE_PORT
     value: "4201"

which I think are correct:

prefect-server-graphql        ClusterIP  172.20.137.57  <none>    4201/TCP  38m
   prefect-server-hasura        ClusterIP  172.20.185.76  <none>    3000/TCP  38m

Do I also need PREFECT_SERVER__* settings here? Any other debugging info? chart code is in https://github.com/PrefectHQ/server/pull/57 @Chris White @Jeremiah — any thoughts?

@josh — Wanted to get your opinion on how to integrate auth into the helm chart. What I’m planning is to allow the chart to take configuration for extra containers in the apollo and ui pods, which could run auth proxies, and fiddle with the port options so that we could bypass auth internally. (Forwarding scopes around, etc I guess I can wait on. :)). I would try to configure with louketo-proxy, as we have keycloak running, but should be possible to run other proxies as well. Does that sound like a good idea, or would you suggest something else?

Hi all, I am would be curious if you at one point maybe thought about providing a completly serverless Prefect agent option? We currently had one tiny service in ECS Fargate with one container running the Fargate agent around the clock. As I am very much a serverless fan and I might only have a daily scheduled flow I thought it would be nice to only activate the agent when needed or even have a push-based system. On AWS side I would see two options e.g. a Lambda function which is triggered via a webhook over AWS Api Gateway or even a AWS SNS webhook. Alternatively if push is not possible I could imagine a poll-based cron scheduled lambda to check for flow runs. Maximum frequency is unfortunately only one minute but might be ok for certain cases but that would be anyways not really “on-demand”. To my knowledge the agent is anyways doing the flow runs in fire and forget style therefore I thought this could be quite doable. Anybody with more insights to the agent having an opinion or did think about something similiar already? 🙂 (I hope it is not wrong to post this question in this channel 😄 )

Dear prefects, I have created a pull request that is rather just a quick fix (just adding kwargs to a run method). Two questions: 1. Is this the right approach to suggest quick fixes? 2. I get an error that

import prefect

takes more than 1s. However, this can't be due to my small changes, can it? Background: I had problems with the

NoRegionError

for the boto client. It seems like setting the

region_name

as kwarg of the secret manager's run method is the most elegant way of solving this issue. Therefore, I added the kwargs to the run method.

Howdy friends… what are

flow_groups

? I can’t seem to find docs on what they exactly are. As a guess, they look like they can be used to apply settings across many flows? I’m specifically thinking about

flow_groups

with regards to implementing Flow Concurrency Limits into the new server codebase, so it’s looking like (based on some other code) we look at the

flow_group

labels first, then default to the

environment

labels? Is that how it should be implemented for this as well? Or should it be checking both sets of labels? I’d guess both sets, but not sure

# Co-routine tasks I was considering creating a “PIN” (or two) concerning “co-routine” tasks. I thought I’d write here first to get some initial feedback. --- A co-routine task runs corresponds to a co-routine in python. It runs multiple times, consuming marked parameters, before optionally transitioning back to a pending state, ready to run again if/when new versions of the parameters are available. It may be implemented in python as an actual co-routine, but also could be implemented using signals. ## Use cases ### Loops over complex sub-graphs in flows. Currently, it is impossible to loop over a subgraph of a flow. A “co-routine” is a primitive that could function as a building block for while loops with complex substructure. Tasks downstream of a “loop controller” co-routine task would run after each execution of the loop body, but could have edges that fed parameters back up to the loop control task. ### Processing external streams The current architecture is not well suited to processing large event streams -- such as a Kafka or NATS stream that could have 100s or 1000s of events per second. Processing such streams should not have to spin up an individual flow per event; but at the same time, it should not be limited to a single long-running event that handles all processing internally. A co-routine task can be triggered for each event, waiting in a pending state until an event arrives. ## Implementation Both internal “loop back” edges, and external edges, would be specially marked as co-routine inputs, so they could be consumed per-iteration, and waited on for the next iteration. A Pending->Coroutine State would be added to represent running coroutines. Internally, co-routine tasks could use dask pub/sub. For external links, the server/cloud could provide a webhook. As a further extension, this could optionally support upgrade to a websocket; also as further extensions, connectors to standard event queues could be provided. --- I like to think of Prefect as “an OS for FAAS processes” — or at least FAAS written in python and running in Dask. I don’t know to what extent this overlaps with your vision? Co-routine tasks could be used as a primitive to create complex flow processes containing event loops. They would be a big jump toward generalizing what Prefect could do.

Hello! I have a flow as follows:

results = []
for op in operations:
  op = task(op)
  most_recent = op
  
  .. a few more conditional tasks defined at compile time
 
  if do_recovery: 
     recovery_task = recover(trigger=any_failed_else_raise_skip_signal)
     most_recent = recovery_task
  results.append(most_recent)

backup_task = Backup('mydatabasename', upstream_tasks=[results], skip_on_upstream_skip=False)
mapped = Aggregate.map(
            results,
            target=unmapped('mydatabasename')
)
mapped.set_upstream([backup_task])

As you can see from the image, instead of getting one list, I have 2 lists and due to this, my Aggregate task is being skipped even though it is downstream of a

skip_on_upstream_skip

task. Is there a way to cleanly resolve this (ie. only get 1 list) ?

Hey yall, question for ya: What is the

labeler

in github actions? It looks like the action is failing on this PR: https://github.com/PrefectHQ/server/actions/runs/266743999

Hi there 👋 In an attempt to get more familiar with Prefect, I thought I’d look at open issues and just add what little I can. I fixed an issue but I am wondering what the standard way to contribute is. Following the development guide, I’m just at the last step of opening a pull request from my forked repo back into Prefect. Do I make the base branch

master

or is there a different branch I should use? For context i’m fixing issue 3318 and my fixes are in my branch.

hey everyone, my name is max (madelgi on github). i've been contributing to prefect off and on for the past month, mostly just adding tasks that have been useful for my purposes at my current company, but i've enjoyed contributing and i'm looking to work on something maybe slightly more substantial to get familiar with other parts of the code base. i've considered a couple of options -- adding a new storage option, maybe this issue, about persisting results in memory, etc

i just wanted to drop a message in here before i start on anything to see if any of the folks @ prefect have an opinion on something they'd like done. e.g., if there's something that's been on the backlog for a while that isn't necessarily super difficult but no one wants to do it, well, feel free to share it with me haha. no worries if no one has any suggestions or requests, i'll just choose some open issue that looks interesting

Hi folks, I’m currently working on improving S3List to support

last_modified_start

and

last_modified_end

to leverage the

last_modified

property of keys, to make it possible to filter keys based on time. I’m struggling to find a way to mock boto3 put_object to write a meaningful test. Any suggestions?

set the channel topic: Core Contributor Cantina 11/6 @ 4pm EST (meet.google.com/aqv-gxru-zsi)

Hello yall! Long time no contribute....we were wanted to make a couple enhancements to the fargate agent. 1 - add an

extra_containers

argument to support adding sidecars to the flow run container 2 - add the ability to skip the task definition registration based a run context key of

task_arn_override

. This would allow folks using a map to submit flow run(subflow pattern) to skip the task registration issue that @Darragh and @Lukas were running into

number 1 allows you to add a datadog agent to your flows

number 2 would require a special run context....Is there a schema example of how the run context is presented to the agent as

GraphQLResult

?

figured it out

serialized_state.context

But also, one thing we'd really like is a user written example on how you use and deploy prefect to do real work! (Realize that's not quite what you were asking about so no worries if you'd rather focus on adding to the code!). We also really welcome additions and contributions to the docs!

@Jenny This is exactly what I was trying to find, do you know if anyone else started on this?

Hi @Lonneke and welcome! I don't think we have that exactly yet but our cause studies might be helpful here: https://www.prefect.io/why-prefect/case-studies Our twitter account is also a good place to check out what has been written about Perfect: https://twitter.com/PrefectIO?s=09

I noticed a spelling mistake on the DaskExecutor code in the docs

do you have a github repo i can post a correction to?

Hey contribs! I need to reschedule our cantina that is supposed to start in 10 mins 😬 Look out for the next invite instead!

set the channel topic: Core Contributor Cantina reschedule TBD

set the channel topic: Core Contributor Cantina 11/20 4pm ET / 1pm PT (meet.google.com/qxg-towp-ymx)

^ so sorry about that last friday contribs, but looking forward to seeing you all 11/20 🙂

My connection dropped! Be right back