https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
introductions
  • j

    Joseph Lim

    02/22/2022, 2:17 PM
    Hello everyone, does Prefect Cloud has any global status page that is available for public to monitor of it services uptime ? Something like status.databricks.com, status.aws.amazon.com ?
    👋 10
    a
    3 replies · 2 participants
  • t

    Tomer Cagan

    02/24/2022, 12:25 PM
    Hello, I am a software developer from Israel. Been reading and playing some with prefect and it looks like a really great product! I am working with an algorithmic research team and we are looking for a platform to run our experiments. We are currently using an in-house system but we experience a lot of friction using it and looking for an alternative. (more inside this thread)
    👋 12
    a
    c
    +1
    13 replies · 4 participants
  • d

    Dylan

    02/25/2022, 10:46 PM
    Hi, I’m an engineer doing a POC for Prefect. I want to get a good local setup running in docker. I tried out the docker-compose on github and am under the impression that’s used more for doing development on Prefect itself. It didn’t work and after reading back through the readme’s I tried to install the conda prefect package and run it on a simple python docker image. Now that I’ve got everything installed I’m trying to get the various components running in a single container runtime that I’ll then split out later (server, agent, ui) The idea is that this local stack be used for trying out local workflows as part of the development process, and then the docker-compose of those three ‘roles’ can be ported to helm charts and run on K8s. I’m running into a couple issues, and after sleuthing around on the GH issues I figured I’d just ask here. docker-compose
    version: '3'
    services:
      python-repl:
        image: prefect
        entrypoint: "/bin/bash"
        volumes:
          - "workspace:/home/workspace/"
        ports:
          - "4200:4200"
      prefect-backend-only:
        image: prefect
        command: prefect backend server && prefect agent local start && prefect server start
        volumes:
          - "workspace:/home/workspace/prefect-server"
      prefect-full-stack:
        image: prefect
        command: prefect server start
        volumes:
          - "workspace:/home/workspace/prefect-server"
    volumes:
      workspace:
    the dockerfile I used to build the
    prefect
    image above is just a python image I’ve installed conda and prefect on. I can run the
    prefect backend server
    command just fine but it blows up on
    prefect agent local start
    with
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='apollo', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4004761130>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    I could just be missing some docker-compose pieces, and ideally I could re-use the docker-compose pieces your development team uses for the local dev stack.
    👋 7
    k
    38 replies · 2 participants
  • d

    Dylan

    02/25/2022, 10:48 PM
    Would the prefect conda pacakge start up out of the box with none of the external dependencies defined or would I need to essentially do what’s been done here? https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml
    k
    1 reply · 2 participants
  • d

    Dylan

    02/25/2022, 10:49 PM
    I’m really excited to try this platform out btw. The work everyone’s done so far is impressive!
    k
    1 reply · 2 participants
  • d

    Dylan

    02/25/2022, 10:58 PM
    And I guess to add to my confusion, from within the container runtime (interactive shell) if I skip the local agent start and just try to do
    prefect server start
  • d

    Dylan

    02/25/2022, 10:58 PM
    I get an error
  • d

    Dylan

    02/25/2022, 10:58 PM
    root@e2ab16a5e319:/# prefect server start
    ...
    ...
    FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose'
  • d

    Dylan

    02/25/2022, 10:58 PM
    So maybe I’m not supposed to issue that command from inside a container?
    k
    2 replies · 2 participants
  • d

    Dylan

    02/25/2022, 10:58 PM
    I will start to reread the docs as well 😄
    k
    1 reply · 2 participants
  • d

    Darren Burton

    03/03/2022, 3:03 AM
    Looking to get started with Prefect in a Conda env on Windows 10. I encountered errors attempting to Orion so I decided to go forward with Prefect.
    👋 9
    k
    a
    11 replies · 3 participants
  • e

    Evgeniya Sukhodolskaya

    03/04/2022, 3:43 PM
    👋 Hello, team! I am a data evangelist from toloka.ai - a crowdsourcing data labeling platform, I am located in Munich:)
    👋 19
    k
    a
    +5
    7 replies · 8 participants
  • v

    Vijay Kalvakuntla

    03/08/2022, 12:13 AM
    Hi, where can I ask questions about billing?
    k
    2 replies · 2 participants
  • t

    The Chosen One

    03/08/2022, 10:05 PM
    Hello fellow dek stackers..... I am The chosen one and it is my pleasure to joining your fine accompaniment this evening... 🙂 ❤️
    👋 11
    c
    k
    +1
    3 replies · 4 participants
  • k

    kensuke matsuura

    03/10/2022, 10:46 AM
    Hello team! I work for a Japanese company and I am in the process of implementing prefect in my project. I am still new to the subject, so I would appreciate your help.
    👋 15
    a
    c
    +7
    10 replies · 10 participants
  • t

    Tom

    03/10/2022, 3:42 PM
    Hey all, I already asked some questions, but did not introduce myself. I'm currently working as a consultant for a major airline doing a project in the field of discrete optimization. I think that field would benefit from having structured pipelines so checking out Prefect for that 🙂
    👋 12
    l
    k
    +5
    8 replies · 8 participants
  • l

    Lalit Pagaria

    03/11/2022, 12:04 PM
    Hi All, Lalit here. We are currently building Obsei : An open-source low-code automation tool. Kind of ETL tool for unstructured (current focus on text later on audio and image) data. We are experimenting with few open-source orchestrator tools like Perfect, Airflow, Ray and Dagster, to add capability to run data extraction job. Along with using cloud version for commercial offering.
    👋 14
    a
    c
    +4
    7 replies · 7 participants
  • d

    Dillon Lindsay

    03/11/2022, 5:54 PM
    Hey Prefect Community! 👋 I work for a medium sized pest control company as a data engineer. I'm currently spearheading our data team's effort to shift from a no-code workflow orchestration/ETL tool to Prefect! We're excited about what we'll be able to do once we can break free from the chains of our current tool and Prefect seems like the perfect way to do it.
    👋 12
    k
    c
    +4
    7 replies · 7 participants
  • c

    Clayton Smith

    03/14/2022, 1:11 AM
    Hello Prefect team. My org is exploring ETL infrastructure providers and I was hopping I could get a product demo/walkthrough. Would it be possible to setup a call with a SME sometime this week or next to answer some product questions? Thanks !!
    👋 9
    k
    c
    5 replies · 3 participants
  • c

    Christopher Leyon

    03/14/2022, 2:20 PM
    Hello fellow Prefecters. I just wanted to introduce myself - I am responsible for standing up using Prefect in my company to eventually take over running our ETL & periodic reporting jobs. I've seen how great the Prefect community is, and I'm excited to be joining!
    :marvin: 1
    👋 15
    k
    a
    +6
    9 replies · 9 participants
  • h

    hayk c.

    03/15/2022, 12:33 AM
    Hi there. New to prefect.
    👋 15
    k
    a
    +5
    15 replies · 8 participants
  • h

    Henry Saba

    03/16/2022, 12:29 AM
    Hi all! I first used Prefect in Jan 2020 at my old job but haven’t checked it out in almost 2 years. Looks like a lot has changed. Fun fact: I still remember the Prefect tutorial advertising that “Flows could be used to make a cup of coffee” ☕
    :marvin: 4
    👋 12
    k
    l
    +6
    9 replies · 9 participants
  • c

    Chris Turnbull

    03/16/2022, 9:33 AM
    Hello everyone! I'm a consultant data architect, based in Barcelona at the minute. Nice to meet everyone, looking forward to connecting with you all ☕
    👋🏽 1
    👋 13
    a
    c
    +6
    8 replies · 9 participants
  • a

    Andrew Lane

    03/17/2022, 4:51 PM
    Hi all. I’m a data scientist based in London. I’ve recently been tasked with getting a data orchestration framework up and running for my org. I’m new to Prefect (and data orchestration in general) so very happy there seems to be a lively community here!
    👋 17
    k
    c
    +5
    8 replies · 8 participants
  • d

    Danil Dubovik

    03/18/2022, 6:49 AM
    Hi all! I’m Python Engineer from Ukraine. I’ve been working with Airflow for 2 years and now we are planing migrating to Prefect. Nice to meet everyone here!🎉
    👋 23
    n
    m
    +9
    12 replies · 12 participants
  • l

    lialzm

    03/18/2022, 6:53 AM
    👋 hi all !I’m Python Engineer from china
    👋 19
    🙌 1
    n
    c
    +4
    7 replies · 7 participants
  • b

    Benoit de Bernardy

    03/20/2022, 8:56 PM
    Hello everybody I’m a software engineer currently based in Spain. I spent most of my life writing software but I never wrote a modern data pipeline. My stack up until now has been c#, node, and a tiny bit of Python. I love that there’s a Slack channel to help each other out!
    👋 17
    a
    k
    +9
    11 replies · 12 participants
  • f

    Florian Guily

    03/21/2022, 4:27 PM
    Hello everyone, I'm a data engineer junior seeking for knowledge about modern data tools. I stumbled upon you guys when searching for a workflow orchestrator as we are currently building our data stack. The 2.0 version looks very interesting and i'm really looking forward to use it and asks questions ! Thank you 🙂
    👋 19
    :marvin: 6
    k
    a
    +8
    10 replies · 11 participants
  • g

    Gopal Sinha

    03/22/2022, 10:01 AM
    Hello Everyone,
  • g

    Gopal Sinha

    03/22/2022, 10:02 AM
    I am new to prefect cloud and trying to do a PoC on prefect with S3, It would be great if anyone can guide me with this PoC
    👋 14
    👋🏽 1
    a
    c
    +7
    10 replies · 10 participants
Powered by Linen
Title
g

Gopal Sinha

03/22/2022, 10:02 AM
I am new to prefect cloud and trying to do a PoC on prefect with S3, It would be great if anyone can guide me with this PoC
👋 14
👋🏽 1
a

Anna Geller

03/22/2022, 10:08 AM
Hi @Gopal Sinha, welcome to Prefect! Check out those getting started resources and let us know if you have any specific questions https://discourse.prefect.io/t/getting-started-with-prefect/27
g

Gopal Sinha

03/22/2022, 10:10 AM
Thanks @Anna Geller
c

Chris Reuter

03/22/2022, 10:40 AM
Welcome to Prefect @Gopal Sinha!
j

justabill

03/22/2022, 12:19 PM
Hey there @Gopal Sinha, welcome!
s

Stéphan Taljaard

03/22/2022, 12:55 PM
Hey Gopal. Enjoy your stay here 😄
k

Kevin Kho

03/22/2022, 2:20 PM
Hi @Gopal Sinha, welcome to Prefect!
j

Joseph Desjardins

03/22/2022, 2:41 PM
Hi @Gopal Sinha! Glad to have you here!
d

Dylan

03/22/2022, 9:44 PM
Welcome, Gopal!
c

Christina Lopez

03/23/2022, 7:41 PM
Welcome to the community @Gopal Sinha 👋🏽
d

David Abraham

03/24/2022, 12:21 AM
Welcome to the Prefect community @Gopal!
View count: 6