https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • j

    Jamie Miller

    02/01/2022, 6:37 PM
    Hey everyone. If I want to have my flow set up in a 'flows' folder, and then have that flow import 2 tasks from a different file, how would I go about that? What kind of import statement would I need to add at the top of my file?
    k
    • 2
    • 2
  • d

    David Mannion

    02/01/2022, 9:36 PM
    Hi all, I'm looking to get started with Prefect. I've installed Prefect Server on an EC2 instance in a private subnet which I can access via a bastion server. I've successfully started the server and can see it running. I'm trying to access the ui by using a putty tunnel to forward a port on my laptop, via the bastion server, to the <machine with prefect>:8080. I configured firefox to use SOCKS5 with localhost and the port forwarded by putty. Unfortunately I'm not getting the ui to load up. It's a difficult one to debug as there's lots of components involved. Has anyone any experience doing something like this? Any tips on prefect configuration that might be pertinent?
    k
    g
    • 3
    • 15
  • s

    Sylvain Hazard

    02/02/2022, 10:15 AM
    Hey there ! Starting to try out GraphQL (first time 😬) to clean up old flow runs in order to reduce the size of Postgre Database. I manage to get flow run ids before a given date as well as deleting a single flow run from its id using
    delete_flow_run
    but is there a way to delete multiple flow runs at once ? Thanks !
    a
    p
    m
    • 4
    • 7
  • f

    frojo

    02/02/2022, 12:11 PM
    Hi everyone! I'm new to Prefect (just a couple of months). I've been using it as provided for some local tests without an issue. I was trying to replicate this tests on a remote machine (ssh mediated) but I have runned into some issues. I'm trying to spin up a local prefect server (just testing some flows) on a remote machine. Some data on versions `prefect diagnostics`:
    {
    "config_overrides": {},
    "env_vars": [],
    "system_information": {
    "platform": "Linux-5.4.0-96-generic-x86_64-with-glibc2.27",
    "prefect_backend": "server",
    "prefect_version": "0.15.13",
    "python_version": "3.8.12"
    }
    }
    I was setting the local server as option and started as usual:
    prefect backend server
    prefect server start
    No special options. Images are pulled and all the services started. However some errors (regarding graphql mainly) start to apper on logs.
    graphql_1   | Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
    graphql_1   | 
    graphql_1   | Could not upgrade the database!
    graphql_1   | Error: Failed to apply Hasura metadata! Error: 503 Server Error: Service Unavailable for url: <http://hasura:3000/v1/query>
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    tmp_graphql_1 exited with code 1
    (I have the full trace until stopping them). I also checked (while waiting the spin up - just displaying the failing one)
    > docker ps 
    7036b92f538d   prefecthq/server:core-0.15.13   "tini -g -- bash -c 
"   About a minute ago   Restarting (1) 1 second ago                                      tmp_graphql_
    which is continously restarting. I have tryied: ‱ previuos Prefect version (0.15.0) with the same effect. ‱ the same env on my local machine with success (so I suspect from the remote configuration, but I could not figure what is missing) Thanks!
    a
    m
    • 3
    • 15
  • m

    Max Watermolen

    02/02/2022, 6:41 PM
    Howdy, Oh man I am struggling to figure out why all my tasks are failing with
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/magnox/'")
    Env: ‱ Local Agent + Local Storage (Running in docker [python:3.10] ... yeah yeah lol) ‱ External Postgres 12 ‱ Prefect Server in Kubernetes (Official HELM) ‱ Django Integrated Code: The Source is located in
    /code/
    We have added
    /code
    to the path and Python path on execution. Notes: ‱ Sometimes a single container will work fine (using the same exact image and everything)(Cant seem to figure out why) ‱ I have tried with and without Dask ‱ I tried to symlink the /code -> /home/magnox to no avail
    k
    • 2
    • 23
  • w

    William Clark

    02/02/2022, 6:58 PM
    Hello, do we have the ability to delete a tenant?
    k
    • 2
    • 9
  • n

    NIKHIL KARADIGUDDA

    02/03/2022, 1:32 PM
    hello
  • n

    NIKHIL KARADIGUDDA

    02/03/2022, 1:32 PM
    I am compltely new
  • n

    NIKHIL KARADIGUDDA

    02/03/2022, 1:32 PM
    to perfect cloud
  • n

    NIKHIL KARADIGUDDA

    02/03/2022, 1:32 PM
    I have a question
  • n

    NIKHIL KARADIGUDDA

    02/03/2022, 1:32 PM
    does prefect cloud provide gpus??
    a
    k
    • 3
    • 24
  • c

    Cristian Toma

    02/03/2022, 2:47 PM
    Hi, Is there a way to append a string to a flow run name?
    k
    • 2
    • 4
  • m

    Manuel Ledesma

    02/03/2022, 11:14 PM
    Hi, how do we setup prefect to use localstack running locally on minikube?
    k
    • 2
    • 7
  • e

    Eric Mauser

    02/04/2022, 12:23 AM
    Hi everyone, quick question on the Airbyte operator. Let's say I have Airbyte running on an EC2 instance. And I have prefect deployed to run on ECS. Will the airbyte operator run the connection on the Airbyte EC2 instance or on the Prefect ECS?
    k
    • 2
    • 2
  • s

    Stéphan Taljaard

    02/04/2022, 6:16 AM
    Hi. I'm trying to guage how much Prefect Cloud would cost. Anyone have a GraphQL query (or anything) on hand to see how many billable task runs I have? (per project/flow, and over time, would be nice)
    a
    • 2
    • 4
  • c

    Christopher

    02/04/2022, 7:26 AM
    Hi, I'm still evaluating prefect as an alternative to Airflow. The docs aren't totally clear to me whether prefect server is intended for production us. Is it open source software that I can run on my own production infrastructure, or is it more supposed to be a test bed for development, with production usage always through prefect cloud?
    a
    • 2
    • 2
  • m

    Matthew Webster

    02/04/2022, 4:59 PM
    Hi! I’m looking at the options for best recording a summary of results (e.g. “2,131 records loaded, 5,000 skipped” etc.) and was wondering if there was an option that would show up in the job tool tips like the screenshot. If not, any other suggestions of where to best/prominently log and display?
    k
    • 2
    • 1
  • h

    Hagai Arad

    02/05/2022, 9:57 PM
    Hello, does Prefect server integrate with Okta or any other similar platforms?
    k
    • 2
    • 1
  • j

    jack

    02/08/2022, 2:48 PM
    What is the correct way to start the agent using an API key? Previously this worked:
    prefect agent local start --token [some-token] --api <https://api.prefect.io>
    But after generating an api KEY (not a token) and running a similar incantation:
    prefect agent local start --token [THE-NEW-KEY] --api <https://api.prefect.io>
    it complains with
    prefect.utilities.exceptions.ClientError: [{'path': ['register_agent'], 'message': 'Unauthorized: permission error.', 'extensions': {'code': 'FORBIDDEN'}}]
    k
    • 2
    • 14
  • s

    Santiago Gonzalez

    02/08/2022, 9:45 PM
    Hey, I have a flow with manual task. I added a state handler, in order to post a Slack Message in a channel, to let know that a task is waiting for approval. Is there a way to print some kind of link to the task is running?
    k
    • 2
    • 7
  • a

    Andrew Stewart

    02/09/2022, 3:05 PM
    Hello! I'm new to prefect and I'm trying to use the prefect cli to register my a flow. Every time I run it, it returns with "0 registered." I even tried to register one of the tutorial programs (command: prefect register --project "misc" --path 03_parameterized_etl_flow.py) and it did the same thing. I was able to get one flow to register using the flow.register() method in my script, but I can't get the cli to detect my flows.
    k
    m
    • 3
    • 22
  • v

    Vladislav Bogucharov

    02/09/2022, 5:48 PM
    Hello to all! I have a question. Using checkpoints and targets, I set up the pipeline in such a way that successfully completed tasks are not re-executed. It’s great! But is there any mechanism that allows you to force the task to be reloaded? For example, when you need to force overwrite a file, even if it has already been created.
    k
    • 2
    • 1
  • f

    Francois van Heerden

    02/09/2022, 8:09 PM
    Is it possible to include a link to a pdf produced as output from a run using the
    Artifacts
    api? I’m currently using the server backed and a local agent.
    k
    • 2
    • 8
  • s

    Stéphan Taljaard

    02/10/2022, 4:55 AM
    The latest Orion announcement has me quite excited for "local" task concurrency limits. It reminded me of the per-agent concurrency work that wasn't possible in Server unfortunately. Any plans to pick back up in in Orion?
    k
    • 2
    • 2
  • s

    Sam Werbalowsky

    02/10/2022, 2:24 PM
    Hey all - I’ve been running prefect in production with no issues on
    kubernetes
    . This morning, no prefect job pods are spinning up - has anyone encountered this?
    k
    • 2
    • 11
  • t

    Tara

    02/11/2022, 6:42 AM
    Hi there, does any one know if there’s a way to disable (turning off) schedule of a running flow via cli ? I’ve been looking through the documentation and there seems to be no option for that.
    k
    • 2
    • 1
  • s

    Stéphan Taljaard

    02/11/2022, 3:18 PM
    Hi. I'm sure the answer will be here if I search 😅 Why am I not seeing logs logged using
    prefect.context.get("logger").warning()
    Surely if the default logging level is
    info
    , it will hide
    debug
    , but should still log and show
    warning
    ,
    error
    , and
    critical
    ?
    k
    • 2
    • 5
  • s

    Stéphan Taljaard

    02/14/2022, 11:35 AM
    Hey. Anyone that can point out if there's a gotcha I missed here? I created a flow to get a flow run (by name)'s logs and email them to a user. I figured out the GQL query in the
    Interactive API
    tab on the Server page. I then moved it to my flow. When running the flow, I'm getting
    ReadTimeoutError: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)
    It's strange to me, because I'm using default port/other settings, and expect it to work since the
    Interactive API
    gives a result almost immediately
    k
    • 2
    • 11
  • a

    Alexis Lucido

    02/14/2022, 1:37 PM
    Another question, but big issue for us here. Sorry folks. We have deployed Prefect Core on an on-premise virtual machine with 4 CPU and 16 Go of RAM. Some flows regularly fail and are retried. Right now we have 10 flows retrying, and they are overloading the CPUs... Those retrying flows slow down the subsequent ones, that get slower and slower, until no flow can run anymore and the whole VM is virtually stopped. The tasks executed by the flows are not memory-intensive and, when no error is raised, they run smoothly in several seconds. But here, we are facing some critical error problems... One solution would be to reboot the whole application regularly, but it was something advised to do with Airflow and its scheduler issues, and one of the reason we switched from Airflow to Prefect. I have also optimized for the number of retries and the retry_timedelta, but that is not an optimal solution. You can find attached a screenshot of my htop command. Any thoughts on how to solve the issue? Btw we are using Prefect for launching an entirely-automated renewable electricity trading and, except for these bugs that I'm sure we can solve, we are very happy with the solution and would happily collaborate for writing blog posts showing our use-case of your solution or anything! Best, and thanks in advance!
    k
    a
    • 3
    • 50
  • h

    Henrietta Salonen

    02/14/2022, 2:32 PM
    Hello, This may be something simple that I’m just totally overlooking but trying to use the s3.s3Upload.run in my flow in the following way:
    with Flow("test") as flow:
        s3.S3Upload.run(data, credentials="AWS_CREDENTIALS", bucket='bucket_name', compression='gzip')
    
    flow.run()
    Data is json string. I keep on getting this error
    AttributeError: 'str' object has no attribute 'bucket'
    k
    • 2
    • 5
Powered by Linen
Title
h

Henrietta Salonen

02/14/2022, 2:32 PM
Hello, This may be something simple that I’m just totally overlooking but trying to use the s3.s3Upload.run in my flow in the following way:
with Flow("test") as flow:
    s3.S3Upload.run(data, credentials="AWS_CREDENTIALS", bucket='bucket_name', compression='gzip')

flow.run()
Data is json string. I keep on getting this error
AttributeError: 'str' object has no attribute 'bucket'
k

Kevin Kho

02/14/2022, 2:34 PM
You shouldn’t call the
run()
method inside the Flow. I don’t know if this will help but try:
with Flow(..) as flow:
    S3Upload(init_stuff_here)(run_stuff_here)
h

Henrietta Salonen

02/14/2022, 2:38 PM
Hey do you mean like this:
with Flow("test") as flow:
    s3.S3Upload(data, credentials="AWS_CREDENTIALS", bucket='bucket_name', compression='gzip')

flow.run()
k

Kevin Kho

02/14/2022, 2:40 PM
no you need 2 parenthesis. either:
s = S3Upload()
with Flow(..) as flow:
    s()
or:
with Flow(..) as flow:
    S3Upload()()
h

Henrietta Salonen

02/14/2022, 2:54 PM
cheers, that was it! Thank you 🙂
k

Kevin Kho

02/14/2022, 3:00 PM
Nice!
View count: 1