https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    Mike Ninov

    07/24/2020, 11:15 PM
    What are the capabilities of perfect?
    d
    • 2
    • 2
  • g

    Greg Desmarais

    07/28/2020, 1:50 PM
    Is it possible to extend the prefect Flow with my own flow class that implements some common patterns? If I do that, and I use my own flow class in the context manager during a flow creation on the client, which downstream services need that flow class code? I am seeing this in my prefect server ui:
    Last State Message
    [9:35am]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'rightsize'")
    j
    j
    • 3
    • 76
  • g

    Greg Desmarais

    07/28/2020, 8:04 PM
    I'm working on getting Results to be available for registered flows. I have a server deployed that is executing my flows nicely with a Dask cluster, and can run those flows when they are not registered. When I register a flow and create a flow run, I don't seem to be able to get a hold of any Results.
    j
    c
    • 3
    • 22
  • b

    Bivu Raj

    07/30/2020, 4:55 PM
    using the Client class, how do i use the flow name rather that flow uuid? e.g.
    client.create_flow_run("a14f9774-e94d-40cc-b019-cb9044e120d4", run_name=f"manually_run_flow")
    I understand that there can be multiple versions of the same flow. However, is there a way to query the latest version of UUID for a given flow from the client class itself and not using the graphql api? Thanks
    c
    p
    • 3
    • 7
  • r

    Roy

    07/31/2020, 12:49 PM
    Hi! I have defined a flow and can run it with
    flow.run()
    . However, these runs do not show up in the Prefect Dashboard, even when I
    .register()
    them (the flow shows up in the web interface and I can run it from there, but starting it with
    .run()
    does not show them as a Run). Is there a way to make them show up without running an agent?
    j
    • 2
    • 4
  • b

    Bivu Raj

    08/01/2020, 7:59 PM
    If I have the server running locally (with
    prefect backend server
    ), is there a way to run a flow every 10 seconds. Right now, I take an error saying that the cloud runner's minimum time delta is 1 minute? Thanks
    n
    • 2
    • 1
  • e

    emmanuel

    08/07/2020, 9:23 AM
    Hey there, I’ve just started exploring Prefect locally with the Prefect OSS server and I came across a couple of basic issues: • I can’t create any project from the local UI: I get a failure message (with no explanation and nothing specific in the logs the docker containers). I could create the project tho by direct insertion in the PG database. • When I run a local agent locally, it registers properly with the graphql server but it doesn’t show up in the local ui agent dashboard (or I didn’t find that part of the dashboard) • After registering a flow, I can’t get it to start from the UI, it complains about
    tenant
    being
    None
    🤷‍♂️
    • 1
    • 1
  • e

    emmanuel

    08/07/2020, 12:18 PM
    ^^ ok found the issue, since I could not create a project from the UI when creating it directly in PG I had to attach it to a valid tenant (default one actually)… too bad the project creation didn’t work from the UI
    n
    • 2
    • 13
  • l

    Lukas N.

    08/12/2020, 9:43 AM
    Hi everyone, I would like to use specific version of Prefect. When testing things locally
    prefect server start --version 0.13.2
    gives me an error. It's looking for the manifest for
    prefecthq/apollo:0.13.2
    which doesn't exist. Using the
    latest
    works, but I would rather not use the latest tag in a production system as it may change when you don't want to. Looking at tags of prefecthq/server, prefecthq/apollo and prefecthq/ui the latest common version was
    0.12.6
    . Where can I find which versions of those Prefect components are compatible?
    j
    • 2
    • 2
  • j

    Jeremiah

    08/12/2020, 11:30 AM
    set the channel topic: https://github.com/PrefectHQ/server
  • r

    Rafi Rubin

    08/12/2020, 7:04 PM
    Hi. We just tried setting up a clean install with 0.13.2 and the ui is not finding a tenant, and the create-default-tenant command noted in the web ui doesn't seem to be working.
    c
    • 2
    • 14
  • r

    Rafi Rubin

    08/12/2020, 11:09 PM
    Just a comment we have a running list of fixes to apply when spinning up prefect server. 1. docker-compose in the conda environment (as its not a dependency, perhaps not something that should be fixed) 2. sed on the docker-compose.yml to deal with the fact that we use a proxy for the docker repos 3. sed in the running server container to adjust the graphql url. (not sure if handling of an environment variable fixes this, but haven't seen it working so far) That last point also fixed the missing tenant error.
    👍 2
  • a

    Adam

    08/14/2020, 2:48 PM
    Hi friends, is there any helm chart for Prefect Server or a guide to installing on Kubernetes?
    n
    j
    +2
    • 5
    • 5
  • r

    Riley Hun

    08/14/2020, 5:18 PM
    Hi all, Now that I have my flow running on a local server working beautifully, I would like to transition to the Cloud. I already have a remote server up and running on compute engine on GCP. So what are the next steps for registering a flow to a remote Prefect Server hosted on GCP? I should also add that I have my flows stored as docker images on GCS. Thanks in advance.
    n
    • 2
    • 8
  • e

    Elie Hamouche

    08/16/2020, 10:38 AM
    Hi Everyone I've just started trying out the Prefect Cloud and unfortunately I can't seem to get any tasks to run I have a simple task that looks like this:
    from datetime import timedelta
    
    import prefect
    from prefect import Flow, task
    from prefect.schedules import IntervalSchedule
    
    prefect.config.debug = True
    prefect.config.logging.level = "DEBUG"
    
    @task(name="orchestrate", log_stdout=True)
    def orchestrate():
        logger = prefect.context["logger"]
        logger.info("Orchestrating")
        for i in range(1, 60):
            logger.error(f"Counting {i}")
        return True
    
    
    flow = Flow("Cloud Flow", schedule=IntervalSchedule(
        interval=timedelta(minutes=1),
    ))
    with flow:
        orchestrate()
    
    flow.register(project_name="dev")
    flow.run_agent(token="XXXX")
    And I start the agent with:
    python example.py
    Agent seems to come up fine but just hangs:
    (venv) ibraflow (master|✚5…) python example.py
    Result check: OK
    Flow: <https://cloud.prefect.io/ysft/flow/b7f56496-54df-4eeb-a0b6-2e274c39ed80>
    
     ____            __           _        _                    _
    |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
    | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
    |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
    |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                               |___/
    
    [2020-08-16 10:34:38,874] INFO - agent | Starting LocalAgent with labels ['MacBook-Pro-2.local', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage']
    [2020-08-16 10:34:38,874] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    [2020-08-16 10:34:38,875] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
    [2020-08-16 10:34:39,009] INFO - agent | Waiting for flow runs...
    On the cloud's UI I see the correct version and task, so that rules out authentication and network issues? But the tasks never run and are always marked "late" - attachment:
    n
    j
    c
    • 4
    • 30
  • r

    Riley Hun

    08/17/2020, 5:27 PM
    Hello! Hope everyone had a good weekend. Are there any plans to release a tutorial for adding security to the Prefect Core Server that's deployed to GCP? I'm looking into adding an nginx reverse proxy on my GCP Compute Engine VM where the Prefect Server is hosted but no success. Not sure if anyone has done something like that before and can point me to some available online resources? Would be much appreciated.
    k
    c
    a
    • 4
    • 7
  • b

    Bivu Raj

    08/18/2020, 6:26 PM
    Hello prefect team !! With the new prefect UI (that was released maybe 3 weeks ago), I am getting this error: "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:4200/graphql. (Reason: CORS request did not succeed)." Any idea what is going on? Thanks
    j
    j
    • 3
    • 4
  • c

    Craig

    08/19/2020, 8:25 PM
    Hi, I have been asked to deploy Prefect by our Data team. Since we operate in AWS I would like to use Aurora Postgres for the database and Fargate for the containers. Is there some documentation on this deployment?
    j
    g
    • 3
    • 2
  • m

    Marwan Sarieddine

    08/19/2020, 8:29 PM
    Hi Folks, is there a link to monitor the performance of prefect cloud i.e. to track any service issues if/when they arise ?
    c
    • 2
    • 2
  • r

    raman

    08/19/2020, 11:20 PM
    Hello folks, I am new to prefect and trying to build a process to load millions of files from a FTP location to one of the local database. While doing this we have the typical ETL transformation which happens at each file. My question here is on how can i design this using prefect? Right now we have a in house process which leverages Celery to queue and process the files but the way it runs currently, it downloads all the files and then perform the ETL process on each one of it in parallel. Shall we maintain it the same way or download each file and then perform ETL operation? Maintaining error files state even if retry logic fails. Also the current time which this process takes is about 20-24 hours as it processes Billions of records. Any insight on how to design this would be helpful.
    👀 1
    d
    • 2
    • 9
  • c

    Charles Leung

    08/21/2020, 1:56 PM
    Hey Prefect team, I"m having issues using docker storage when the base_image is being pulled from my team's on-prem repository; it seems docker-py doesn't support that at the moment?
    • 1
    • 2
  • m

    Matt Allen

    08/25/2020, 3:34 PM
    I'm getting an error when the graphql service tries to run it's Alembic migrations against an RDS postgres instance. Has anyone seen anything like this before?
    Error: (psycopg2.errors.SyntaxError) syntax error at or near "REFERENCING"
    LINE 61: ...rting_state AFTER INSERT ON public.flow_run_state REFERENCIN...
    j
    • 2
    • 3
  • c

    Charles Leung

    08/25/2020, 9:26 PM
    Hey team! I'm having an issue with running prefect server start, but against my private docker registry. how do i configure this?
    j
    • 2
    • 4
  • c

    Charles Leung

    08/27/2020, 3:39 PM
    Hey Team, i'm having an issue when running "prefect server create-tenant". I am getting the error [requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=30)], however i have set the environment variable GRAPHQL_URL to my hostname where it's hosted; it seems not to be registering the env variable. Any advice on how to properly change the Graphql url?
    d
    m
    • 3
    • 6
  • c

    Charles Leung

    08/27/2020, 6:07 PM
    Question team, if i have multiple agents with the same label, how does the cloud-server / core-server choose which agent to execute on? is it arbitrary?
    d
    • 2
    • 2
  • c

    comet

    08/28/2020, 2:29 AM
    i have trouble running dask on prefect, https://github.com/PrefectHQ/prefect/issues/3226 can anyone helps? i did try similar example using older version of prefect and it was working
    Unexpected error: AttributeError("'NoneType' object has no attribute 'is_finished'")
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 359, in check_upstream_finished
        if not all(s.is_finished() for s in all_states):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 359, in <genexpr>
        if not all(s.is_finished() for s in all_states):
    AttributeError: 'NoneType' object has no attribute 'is_finished'
    any comment?
    m
    • 2
    • 1
  • v

    Vipul

    08/28/2020, 3:35 PM
    Hey Team, I have a query, I am running flow.register() from within the Flask app and I could see as many time Flask app is run, I could see new version from UI. How do I avoid re-registering of the flow? Thanks
    n
    • 2
    • 8
  • c

    Chris Goddard

    08/29/2020, 12:28 AM
    Hey folks - I'm trying to get prefect server set up on google cloud compute. i'm looking through the tutorials on youtube and medium - I'm confused why the graphql url has to be the external IP. if you're just running on one node couldn't it just be on localhost? also, any instructions/tips on how to secure an instance? I could connect to the ui through an ssh tunnel but I'd rather not have anything publicly accessible if I can help it
    n
    • 2
    • 2
  • d

    David Stern

    08/31/2020, 6:32 AM
    Hey everyone, I'm trying to run the cloud tutorial "Running a Flow", I've authenticated using my personal token and then run the agent with its token and this works fine, but "Run Demo Flow" simply results in an error message
    "Failed to start flow run. Please wait a few moments and try again"
    😅 Any ideas with what could be wrong? Searching google and this channel didn't turn anything up :/
    j
    • 2
    • 4
  • m

    Matt Allen

    08/31/2020, 8:32 PM
    Are logs from flow runs stored in the postgres instance for the prefect server? If so, is there any kind of retention policy or cleanup process in place to delete them if they pass a certain age? I'm wondering about running out of space as runs accumulate in the history
    k
    m
    • 3
    • 5
Powered by Linen
Title
m

Matt Allen

08/31/2020, 8:32 PM
Are logs from flow runs stored in the postgres instance for the prefect server? If so, is there any kind of retention policy or cleanup process in place to delete them if they pass a certain age? I'm wondering about running out of space as runs accumulate in the history
k

Kyle Moon-Wright

08/31/2020, 8:49 PM
Hey @Matt Allen, To answer your questions directly: yes and not per se. If you are using the
--use-volume
flag with your instance of Prefect Server, then everything should be stored in your
~/.prefect/pg_data
directory. From here, all of the configurations for your database should be stored in the
postgresql.conf
file, which you should be able to adjust for the logging configurations you need. So no policy in place per se, but you should be able to adjust your db configurations there. Otherwise, deleting a flow should also delete its logs.
m

Matt Allen

08/31/2020, 8:50 PM
Awesome, thanks
I'm running with an RDS postgres, so I imagine I'll have to set something up to delete old flows if this becomes a problem
m

Mikael

09/04/2020, 4:37 AM
@Matt Allen how do you use RDS with prefect? 🤔
m

Matt Allen

09/04/2020, 3:30 PM
I'm using a custom helm chart based on this https://github.com/szelenka/prefect-server-k8 to deploy the different parts of the server to kubernetes
👍 1
View count: 2