https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-contributors
  • l

    Laura Lorenz

    08/11/2020, 3:09 PM
    cleared channel topic
  • l

    Laura Lorenz

    08/11/2020, 10:40 PM
    set the channel topic: Core Contributor Cantina 8/14 @ 4pm EDT (https://meet.google.com/quz-zuea-kix)
  • a

    Aditya Mandke

    08/13/2020, 2:04 PM
    Hi everyone!
    đź‘‹ 6
  • a

    Aditya Mandke

    08/13/2020, 2:26 PM
    I am the web page - https://docs.prefect.io/core/development/overview.html . The readme link there is not working. What should be done?
    d
    • 2
    • 5
  • a

    Aditya Mandke

    08/14/2020, 4:40 AM
    I followed the steps outlined in https://github.com/prefectHQ/ui. After typing npm run serve in the cmd, the following error is shown. What should be done?
    n
    • 2
    • 1
  • c

    Chris White

    08/14/2020, 7:29 PM
    set the channel topic: Core Contributor Cantina 8/14 @ 4pm EDT (https://meet.google.com/xxd-nwqi-hqp)
  • c

    Chris White

    08/14/2020, 7:31 PM
    Hey everyone! @josh and I will be at Contributor Cantina in 30 minutes (@ 4pm EDT) — feel free to stop by to hear about the latest developments in Core, Server and the UI or just to hangout and chat data! Link here: https://meet.google.com/xxd-nwqi-hqp
    ❤️ 1
  • j

    Jeremiah

    08/14/2020, 8:01 PM
    Coming!
  • c

    Chris White

    08/14/2020, 9:21 PM
    Hi @Aditya Mandke - here’s the issue I opened during Cantina: https://github.com/PrefectHQ/ui/issues/110
    ❤️ 1
    a
    • 2
    • 1
  • a

    Aditya Mandke

    08/15/2020, 5:40 PM
    While working on Prefect, is it a must to have Docker installed? For its installation, I need to do update my Windows, and the last time I did it, my laptop got so slow, it became unusable I had to format it. Please help
    j
    n
    • 3
    • 3
  • a

    Ashton

    08/15/2020, 10:07 PM
    hey I ported over the DatabricksSubmitRun operator from Airflow but there is already another PR related to it https://github.com/PrefectHQ/prefect/pull/3084. I've addressed the main comment about it in my branch. Should I submit a PR with my changes that addresses the comments of the old one?
    j
    • 2
    • 2
  • a

    Aditya Mandke

    08/17/2020, 5:02 AM
    @nicholas - yes we spoke on Saturday regarding that. I was thinking, is there some way we can create a completely new repo, or a new folder in the UI repo so that Windows users can contribute? So for example in the Apache Kafka repo (https://github.com/apache/kafka), in the bin folder, they have a separate folder 'windows' which contains windows specific '.bat' files, where as the other files in the bin folder are bash files.
    n
    • 2
    • 2
  • a

    Aditya Mandke

    08/17/2020, 5:05 AM
    I am ready to work on something like this, with your help. Two questions: 1) Is it worthwhile to work on this? 2) If yes, will somebody work with me on this?
  • a

    Alex Cano

    08/20/2020, 8:04 PM
    General question about the
    server
    repo. Are there plans to make it mypy compliant? Just a tad confused because there are mypy configurations in
    setup.cfg
    but mypy isn’t a required package for dev
    c
    • 2
    • 8
  • r

    Ruben Silva

    08/20/2020, 9:44 PM
    Hello 🙂 I'm wondering if someone can explain to me how does prefect core execute mapping with the DaskExecuter. In the Docs, it's said that Prefect automatically creates a copy of the task for each element of its input data. However, looking at the code I'm not understanding where that is being done, and when those new tasks are being submitted to Dask.
    d
    j
    • 3
    • 5
  • s

    Shaun Cutts

    08/21/2020, 6:13 PM
    Hello -- following up on an invitation from dylan@prefect.io -- I'm interested in helping to create helm charts for the prefect core server.
  • s

    Shaun Cutts

    08/21/2020, 6:16 PM
    @Dylan -- any thoughts/tips on getting started? I've cloned the server repo, and am taking a look at it. There are some things created with docker compose, the main server command created with docker, and ... some other services...
  • s

    Shaun Cutts

    08/21/2020, 6:17 PM
    I presume we want a stateful set for postgres. I also would like to support an external postgres option with a secret for config. Is anything else stateful?
  • s

    Shaun Cutts

    08/21/2020, 6:18 PM
    If there are sub-charts to use for any of the components that would be great.
  • s

    Shaun Cutts

    08/21/2020, 6:20 PM
    Also ... should the chart source go in the server repo itself, and where should it be hosted? I've created "private" charts before, which I deploy with a relative path, but this is my first "public" chart.
  • s

    Shaun Cutts

    08/21/2020, 6:23 PM
    I’ll also want some guidance on service account setup. I use RBAC and ISRA, but am not terribly sophisticated -- mainly I just copy recipes. You support several cloud providers, but setup for the general case (or various specific cases) might be beyond my capabilities / time constraints.
  • s

    Shaun Cutts

    08/21/2020, 6:30 PM
    Perhaps the best place to start would be how to map the description in the doc onto pods and services: • UI: the user inteface that provides a visual dashboard for mutating and querying metadata • Apollo: the main endpoint for interacting with the server • PostgreSQL: the database persistence layer where metadata is stored • Hasura: the GraphQL API that layers on top of Postgres for querying metadata • GraphQL: the server’s business logic that exposes GraphQL mutations • Towel: runs utilities that are responsible for server maintenance • _Scheduler_: schedules and creates new flow runs • _Zombie Killer_: marks task runs as failed if they fail to heartbeat • _Lazarus_: reschedules flow runs that maintain an unusual state for a period of time
    s
    • 2
    • 2
  • s

    Shaun Cutts

    08/21/2020, 6:34 PM
    It would seem that at least UI and Apollo need externally available services. [I like configuring ingress myself anyway, but I guess could put some default ingress config into the chart…] How do these things talk to each other under the hood?
  • s

    Shaun Cutts

    08/21/2020, 6:44 PM
    Another area I’m interested in is how credentials (eg to s3 storage, dask gateway) should be dealt with. “general secrets” can be simply specified at create time and mounted. I’m not sure how one would deal with user-specific secrets.
  • c

    Chris White

    08/21/2020, 7:07 PM
    Hi @Shaun Cutts - our bandwidth is limited to support such an effort but we’d be more than happy to accept a Helm chart PR in the Server repo; I highly recommend starting with the docker compose in the Core repository here and reverse engineering it into a helm chart: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml
  • s

    Shaun Cutts

    08/21/2020, 7:10 PM
    Chris — thanks for the reply! I saw the docker-compose but it didn’t seem complete to me. Are all the other things mentioned in the architecture currently run in the “main” docker container? (e.g. Towel — etc… )?
    c
    • 2
    • 10
  • m

    Marwan Sarieddine

    08/21/2020, 8:29 PM
    Hi folks, I am trying the S3 flow storage option - using a kubernetes agent and a DaskKubernetesEnvironment execution environment. The issue I am running into is that the prefect agent is creating a prefect-job pod that is throwing an S3 authentication error. I am using a k8s secret and I already have the proper AWS credentials set as environment variables on the agent and worker k8s spec. but for some reason the prefect-job is not being delegated these environment variables... What is the prefect way to running an s3 stored flow with this similar setup ?
    j
    • 2
    • 19
  • s

    Shaun Cutts

    08/22/2020, 8:17 PM
    Implementing experimental helm chart: • graphql and towel have “PREFECT_SERVER__HASURA__ADMIN_SECRET” in environment. Is this an actual secret that I should be mounting in the environment? What is its form? How come the “hasura” container doesn’t have it itself? Is it written by hasura container — so perhaps it should be a shared volume? (RW hasura, RO by the other two)? • What would be sensible defaults for pod resources for: {hasura, graphql, towel, apollo, ui}?
    j
    • 2
    • 3
  • s

    Shaun Cutts

    08/23/2020, 4:57 AM
    Implementing experimental server helm chart: 1. health and liveness probes -- • hasura supports /healthz endpoint, which I’m using for both health and liveness checks. (ok?) • graphql: apollo env variable suggests “/health” as healthcheck • apollo: doc suggests
    /.well-known/apollo/server-health
    • ui .. ? .. just use “/”? • towel: even more problematic, as no containerPort 2. Database password: Can I mount from secret into PGPASSWORD in environment, and leave out of HASURA_GRAPHQL_DATABASE_URL (etc)? Building “HASURA_GRAPHQL_DATABASE_URL” from secret would take a start hook (unless there is another idea?) 3. init containers • If the database doesn’t exist, it should be created using admin pass. This could be init container on hasura, I guess? • After optional db creation — what to run to create schema, if necessary? • Then there are database upgrades (alembic) — these are idempotent, right — can be run on every startup? All that can be put in hasura, I guess. For other pods, init might amount to waiting for other services to be available? If we had good health checks we could just have kubernetes kill and restart until other services were up? 4. Default database size (for pvc if not external) • 8Gi?
  • s

    Shaun Cutts

    08/23/2020, 2:38 PM
    Implementing experimental server helm chart; more questions: 5. src/config.toml -- Where is this used? Should I be rebuilding it using a config map? It would seem that it duplicates some settings passed as environment variables. hmm… is this just for the cli? [NB perhaps the server cli and the server itself should be split into separate repos.] EDIT — seems to be used in migrations — so initcontainer needs, at least. I guess I can use PREFECT__ … vars to overwrite anything necessary rather than having to compose in config map? E.g. PREFECT__DATABASE__PASSWORD can be configured as env mounted from secret. 5a. what about services/hasura/config.yaml … does this endpoint need to be overwritten with configured containerPort? Should it use FQDN or is it purely for loopback? Hmm… I guess this is baked into the Dockerfile for hasura, so presumably should be left as is, and the containerPort should be fixed at 3000, and we should use the service port to configure external access. 6. postgresql connection config — (see #2, above) Could we go to using PGHOST, PGPORT, etc, and have the various db drivers use a “default connection” or have the various applications construct their own connection string if that doesn’t work? I can compose a connection url in a helm template, but I shouldn’t be putting the password in it. And if you are also constructing from pieces (e.g. in config.toml) then this would be the easiest default way to converge.
Powered by Linen
Title
s

Shaun Cutts

08/23/2020, 2:38 PM
Implementing experimental server helm chart; more questions: 5. src/config.toml -- Where is this used? Should I be rebuilding it using a config map? It would seem that it duplicates some settings passed as environment variables. hmm… is this just for the cli? [NB perhaps the server cli and the server itself should be split into separate repos.] EDIT — seems to be used in migrations — so initcontainer needs, at least. I guess I can use PREFECT__ … vars to overwrite anything necessary rather than having to compose in config map? E.g. PREFECT__DATABASE__PASSWORD can be configured as env mounted from secret. 5a. what about services/hasura/config.yaml … does this endpoint need to be overwritten with configured containerPort? Should it use FQDN or is it purely for loopback? Hmm… I guess this is baked into the Dockerfile for hasura, so presumably should be left as is, and the containerPort should be fixed at 3000, and we should use the service port to configure external access. 6. postgresql connection config — (see #2, above) Could we go to using PGHOST, PGPORT, etc, and have the various db drivers use a “default connection” or have the various applications construct their own connection string if that doesn’t work? I can compose a connection url in a helm template, but I shouldn’t be putting the password in it. And if you are also constructing from pieces (e.g. in config.toml) then this would be the easiest default way to converge.
View count: 2