cleared channel topic

set the channel topic: Core Contributor Cantina 8/14 @ 4pm EDT (https://meet.google.com/quz-zuea-kix)

Hi everyone!

I am the web page - https://docs.prefect.io/core/development/overview.html . The readme link there is not working. What should be done?

I followed the steps outlined in https://github.com/prefectHQ/ui. After typing npm run serve in the cmd, the following error is shown. What should be done?

set the channel topic: Core Contributor Cantina 8/14 @ 4pm EDT (https://meet.google.com/xxd-nwqi-hqp)

Hey everyone! @josh and I will be at Contributor Cantina in 30 minutes (@ 4pm EDT) — feel free to stop by to hear about the latest developments in Core, Server and the UI or just to hangout and chat data! Link here: https://meet.google.com/xxd-nwqi-hqp

Coming!

Hi @Aditya Mandke - here’s the issue I opened during Cantina: https://github.com/PrefectHQ/ui/issues/110

While working on Prefect, is it a must to have Docker installed? For its installation, I need to do update my Windows, and the last time I did it, my laptop got so slow, it became unusable I had to format it. Please help

hey I ported over the DatabricksSubmitRun operator from Airflow but there is already another PR related to it https://github.com/PrefectHQ/prefect/pull/3084. I've addressed the main comment about it in my branch. Should I submit a PR with my changes that addresses the comments of the old one?

@nicholas - yes we spoke on Saturday regarding that. I was thinking, is there some way we can create a completely new repo, or a new folder in the UI repo so that Windows users can contribute? So for example in the Apache Kafka repo (https://github.com/apache/kafka), in the bin folder, they have a separate folder 'windows' which contains windows specific '.bat' files, where as the other files in the bin folder are bash files.

I am ready to work on something like this, with your help. Two questions: 1) Is it worthwhile to work on this? 2) If yes, will somebody work with me on this?

General question about the

server

repo. Are there plans to make it mypy compliant? Just a tad confused because there are mypy configurations in

setup.cfg

but mypy isn’t a required package for dev

Hello 🙂 I'm wondering if someone can explain to me how does prefect core execute mapping with the DaskExecuter. In the Docs, it's said that Prefect automatically creates a copy of the task for each element of its input data. However, looking at the code I'm not understanding where that is being done, and when those new tasks are being submitted to Dask.

Hello -- following up on an invitation from dylan@prefect.io -- I'm interested in helping to create helm charts for the prefect core server.

@Dylan -- any thoughts/tips on getting started? I've cloned the server repo, and am taking a look at it. There are some things created with docker compose, the main server command created with docker, and ... some other services...

I presume we want a stateful set for postgres. I also would like to support an external postgres option with a secret for config. Is anything else stateful?

If there are sub-charts to use for any of the components that would be great.

Also ... should the chart source go in the server repo itself, and where should it be hosted? I've created "private" charts before, which I deploy with a relative path, but this is my first "public" chart.

I’ll also want some guidance on service account setup. I use RBAC and ISRA, but am not terribly sophisticated -- mainly I just copy recipes. You support several cloud providers, but setup for the general case (or various specific cases) might be beyond my capabilities / time constraints.

Perhaps the best place to start would be how to map the description in the doc onto pods and services: • UI: the user inteface that provides a visual dashboard for mutating and querying metadata • Apollo: the main endpoint for interacting with the server • PostgreSQL: the database persistence layer where metadata is stored • Hasura: the GraphQL API that layers on top of Postgres for querying metadata • GraphQL: the server’s business logic that exposes GraphQL mutations • Towel: runs utilities that are responsible for server maintenance • _Scheduler_: schedules and creates new flow runs • _Zombie Killer_: marks task runs as failed if they fail to heartbeat • _Lazarus_: reschedules flow runs that maintain an unusual state for a period of time

It would seem that at least UI and Apollo need externally available services. [I like configuring ingress myself anyway, but I guess could put some default ingress config into the chart…] How do these things talk to each other under the hood?

Another area I’m interested in is how credentials (eg to s3 storage, dask gateway) should be dealt with. “general secrets” can be simply specified at create time and mounted. I’m not sure how one would deal with user-specific secrets.

Hi @Shaun Cutts - our bandwidth is limited to support such an effort but we’d be more than happy to accept a Helm chart PR in the Server repo; I highly recommend starting with the docker compose in the Core repository here and reverse engineering it into a helm chart: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml

Chris — thanks for the reply! I saw the docker-compose but it didn’t seem complete to me. Are all the other things mentioned in the architecture currently run in the “main” docker container? (e.g. Towel — etc… )?

Hi folks, I am trying the S3 flow storage option - using a kubernetes agent and a DaskKubernetesEnvironment execution environment. The issue I am running into is that the prefect agent is creating a prefect-job pod that is throwing an S3 authentication error. I am using a k8s secret and I already have the proper AWS credentials set as environment variables on the agent and worker k8s spec. but for some reason the prefect-job is not being delegated these environment variables... What is the prefect way to running an s3 stored flow with this similar setup ?

Implementing experimental helm chart: • graphql and towel have “PREFECT_SERVER__HASURA__ADMIN_SECRET” in environment. Is this an actual secret that I should be mounting in the environment? What is its form? How come the “hasura” container doesn’t have it itself? Is it written by hasura container — so perhaps it should be a shared volume? (RW hasura, RO by the other two)? • What would be sensible defaults for pod resources for: {hasura, graphql, towel, apollo, ui}?

Implementing experimental server helm chart: 1. health and liveness probes -- • hasura supports /healthz endpoint, which I’m using for both health and liveness checks. (ok?) • graphql: apollo env variable suggests “/health” as healthcheck • apollo: doc suggests

/.well-known/apollo/server-health

• ui .. ? .. just use “/”? • towel: even more problematic, as no containerPort 2. Database password: Can I mount from secret into PGPASSWORD in environment, and leave out of HASURA_GRAPHQL_DATABASE_URL (etc)? Building “HASURA_GRAPHQL_DATABASE_URL” from secret would take a start hook (unless there is another idea?) 3. init containers • If the database doesn’t exist, it should be created using admin pass. This could be init container on hasura, I guess? • After optional db creation — what to run to create schema, if necessary? • Then there are database upgrades (alembic) — these are idempotent, right — can be run on every startup? All that can be put in hasura, I guess. For other pods, init might amount to waiting for other services to be available? If we had good health checks we could just have kubernetes kill and restart until other services were up? 4. Default database size (for pvc if not external) • 8Gi?

Implementing experimental server helm chart; more questions: 5. src/config.toml -- Where is this used? Should I be rebuilding it using a config map? It would seem that it duplicates some settings passed as environment variables. hmm… is this just for the cli? [NB perhaps the server cli and the server itself should be split into separate repos.] EDIT — seems to be used in migrations — so initcontainer needs, at least. I guess I can use PREFECT__ … vars to overwrite anything necessary rather than having to compose in config map? E.g. PREFECT__DATABASE__PASSWORD can be configured as env mounted from secret. 5a. what about services/hasura/config.yaml … does this endpoint need to be overwritten with configured containerPort? Should it use FQDN or is it purely for loopback? Hmm… I guess this is baked into the Dockerfile for hasura, so presumably should be left as is, and the containerPort should be fixed at 3000, and we should use the service port to configure external access. 6. postgresql connection config — (see #2, above) Could we go to using PGHOST, PGPORT, etc, and have the various db drivers use a “default connection” or have the various applications construct their own connection string if that doesn’t work? I can compose a connection url in a helm template, but I shouldn’t be putting the password in it. And if you are also constructing from pieces (e.g. in config.toml) then this would be the easiest default way to converge.