https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • c

    Constantino Schillebeeckx

    06/23/2022, 2:34 PM
    is anyone having pretty major issues using the Web UI in firefox?
    k
    m
    +2
    • 5
    • 16
  • r

    rectalogic

    06/23/2022, 3:13 PM
    we're no longer able to login via firefox or chrome. We login then get sent to https://cloud.prefect.io/access-denied In chrome sometimes it can't resolve cloud.prefect.io
    k
    j
    n
    • 4
    • 10
  • s

    Scott Aefsky

    06/23/2022, 5:27 PM
    Hi all. I have a flow with the following tasks defined:
    capture_ID_batch, all_capture_metadata = get_unprocessed_capture_ID_batches(snowflake_tbl, experiment_id=10, batch_size = 500)
        capture_loc_map = get_locations_for_captures(all_capture_metadata, s3_bucket)
        #read the files from local disk given IDs, process and record proc jsons
        process_batch.map(capture_ID_batch, unmapped(snowflake_tbl), unmapped(capture_loc_map), unmapped(intrazone_extractor), unmapped(dynamic_zone_finder))
    The Prefect schematic shows that my first task `get_unprocessed_capture_ID_batches`is getting duplicated, and my flow is eventually failing because of a Dask scheduler timeout. The last task starts running, but won't complete because the timeout kills it. I'm trying to understand a couple of things: • Why is my first task showing up as if it's a mapped task? • Why is the base node of that task not completing if each of the 2 instances of it are complete? • Why is the scheduler timing out? If I run this flow with a small amount of data, it runs fine, but with a larger dataset I get this failure. Thanks for any help you can provide!
    k
    • 2
    • 17
  • n

    Nicholas Kan

    06/23/2022, 5:29 PM
    Hi Everyone, trying to set up prefect on my local machine for the first time, I keep getting this error when the postgres docker container is building. I am just using the docker-compose from docker hub. Has anyone encountered this before or knows how to workaround? Error message in the replies
    k
    • 2
    • 14
  • j

    Javier Ochoa

    06/23/2022, 6:48 PM
    Hello, I am trying to setup a prefect agent into AWS ECS task... it worked, but inside the python scripts this prefect agent is running, I am instancing a boto3 to talk to our SSM Parameters but I am getting an "AccessDenied" Error. It must be related to policies, but I have a hard time solving that issue (editing policies, editing roles... etc and still getting the same thing)
    botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::999999999999:assumed-role/MyRole-dev/12345459ba45458183ed3d1aa5112341 is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:region:999999999999:parameter/XXX/value because no identity-based policy allows the ssm:GetParameter action
    What is your suggestion here, or a possible solution for this?
    k
    • 2
    • 12
  • j

    Jehan Abduljabbar

    06/23/2022, 7:32 PM
    Hello, I have this problem when running a flow on the server using dask executer. (I don't have the problem when I use the server without dask, or when I use dask executer but use flow.run() from vs code rather than run from the server) So only when I use both (which I need). Here is the error I get in the server: Error during execution of task: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') This is happening for task load_data[1], which I think corresponds to when item2 is supposed to be loaded to cosmos db. Any advice is appreciated.
    k
    • 2
    • 4
  • a

    Amit Gupta

    06/23/2022, 7:40 PM
    Hey All, How to set database as cloud sql instead of sqlite in Prefect Kubernetes Setup?
    k
    p
    • 3
    • 18
  • g

    Guoying Qi

    06/23/2022, 8:37 PM
    Hi there. Just install Prefect 2.0b6 on my laptop and API and UI server work perfect fine locally. After I register the cloud, and login successful from the command line, and then tried to run some flows, got the following error:
    prefect.exceptions.PrefectHTTPStatusError: Client error '400 Bad Request' for url 'https://*******************************'
    Response: {'detail': 'Request specified API version 0.5.0 but this server only supports version 0.6.0 and below.'}
    For more information check: <https://httpstatuses.com/400>
    Which version of the Prefect I should install on my local machine? thanks.
    k
    • 2
    • 3
  • g

    Guoying Qi

    06/23/2022, 8:54 PM
    For both 2.0b6 and 2.0b7, when you type http://127.0.0.1:4200/ in your browser, it will direct the url to http://127.0.0.1:4200/runs And then click around for runs, flows and deployments side menus, all work fine. But when you refresh the page with any addresses: http://127.0.0.1:4200/runs http://127.0.0.1:4200/flows http://127.0.0.1:4200/deployments it will return the following json:
    {
      "detail": "Not Found"
    }
    k
    k
    • 3
    • 9
  • g

    Guoying Qi

    06/23/2022, 9:02 PM
    Anywhere you release Docker image for Prefect or Helm chart?
    k
    • 2
    • 5
  • g

    Guoying Qi

    06/23/2022, 9:38 PM
    This is still true? Just want to save my time not to learn Version 1.
    When will Prefect 2.0 be released?
    The Prefect 2.0 beta period is expected to last for at least the second quarter of 2022.
    k
    • 2
    • 1
  • t

    Tilé

    06/23/2022, 10:26 PM
    Hello, I have a meta-question. I have a flow of 10 or more tasks that uses the
    PrefectSecret
    class. The problem is that there is a secret I have to pass on every task. It works properly, but when I try to visualize my flow I get this secret shown as input to every task, which basically ruins the whole diagram and makes it hard to read. So 1. Is there any other way to pass my secret to the tasks when running locally or using the agent from the cloud, or 2. Is there any way to remove the secret box from the visualized flow? Thanks
    k
    • 2
    • 1
  • j

    Josh

    06/23/2022, 11:21 PM
    Is it possible to dynamically run prefect tasks within the same flow in different environments? https://discourse.prefect.io/t/run-different-prefect-tasks-in-different-clusters/1117
    k
    • 2
    • 1
  • j

    Jeff Kehler

    06/24/2022, 4:57 AM
    Is it possible to set the Flow run name using a value from a Parameter passed to the flow?
    k
    • 2
    • 2
  • d

    Darren Fleetwood

    06/24/2022, 6:07 AM
    Hi all. I have a bunch of Ray Actors and functions that are created within a Prefect flow. I don't seem to be able to access the Ray logs when the flow is run in Prefect Cloud (1.x). This works fine when I'm testing locally. I’ve tried setting this env var:
    PREFECT__LOGGING__EXTRA_LOGGERS= "['ray']"
    As well as this within the flow:
    prefect.config.logging.extra_loggers = ['ray']
    Plus a few other things, none of which have worked. This is the case for all logging levels (error, info, etc.) What’s the proper way of doing this? Thanks!
    k
    • 2
    • 3
  • b

    Balveer Singh

    06/24/2022, 6:47 AM
    Hi all. How can we install pip library within code and, import that library for any particular task.
    k
    • 2
    • 1
  • z

    Zheng Cheng

    06/24/2022, 9:16 AM
    k
    • 2
    • 5
  • g

    Guoying Qi

    06/24/2022, 11:06 AM
    $ prefect -v
    2.0b7
    
    $ prefect orion kubernetes-manifest
    Error: No such command 'kubernetes-manifest'.
    
    $ prefect orion --help
    Commands:
      database  Commands for interacting with the database.
      start     Start an Orion server
    The documents needs to be updated? https://orion-docs.prefect.io/tutorials/kubernetes-flow-runner/
    k
    • 2
    • 1
  • j

    Joshua Greenhalgh

    06/24/2022, 11:29 AM
    I have a flow that looks a bit like this;
    k
    • 2
    • 13
  • m

    Maverick Humbert

    06/24/2022, 11:55 AM
    Hey ! I struggle a little bit with DBT Do I need to add dbt-core && dbt-postgres if I added prefect[dbt] ?
    k
    n
    • 3
    • 5
  • b

    Balveer Singh

    06/24/2022, 12:36 PM
    How can we an install and import libraries during runtime in python script.
    m
    k
    • 3
    • 3
  • j

    Jelle Vegter

    06/24/2022, 1:15 PM
    Hi all, does anyone have experience hosting a Prefect Agent (1.0) in an Azure Container Instance? Locally the container runs properly but when attempting to host it in Azure it returns "no content"
    k
    • 2
    • 1
  • s

    Sander

    06/24/2022, 1:38 PM
    Hi, I was wondering how I can handle the following case. Suppose person A sets up a deployment for flow_a and person B want to set up a deployment for flow_b that depends on (recent) results from flow_a that he doesn't maintain. I'm looking to setting this up in prefect 2.0
    k
    • 2
    • 15
  • b

    Benjamin Bonhomme

    06/24/2022, 2:19 PM
    Hi Everyone! Prefect Cloud question: Is it possible to change the Prefect Cloud Authentication method from Google SSO to Login/Password?
    k
    • 2
    • 3
  • r

    redsquare

    06/24/2022, 5:28 PM
    in prefect cloud 2.0 how do I get the PREFECT_API_URL for the k8's config?
    k
    • 2
    • 9
  • r

    redsquare

    06/24/2022, 5:29 PM
    is it https://beta.prefect.io/account/{account_id}/workspace/{workspace_id}
  • r

    redsquare

    06/24/2022, 5:32 PM
    or simply https://beta.prefect.io/api
  • k

    komal azram

    06/25/2022, 8:30 AM
    Hello I have started using prefect for airbyte orchestration task. Below I have attached my code
    import prefect
    from prefect import task, Flow
    
    @task
    def claims_func():
        airbyte_server_host = "localhost",
        airbyte_server_port = 8000,
        airbyte_api_version = "v1",
        connection_id = conn_id,
    
    
    
    with Flow("fhir-flow") as flow:
        claims_func()
    
    
     
    flow.run()
    As per my understanding when when I run this flow it should trigger the connection and automatically sync data in from gcp->snowflake. I don't get any error but no data is synced.
    o
    k
    • 3
    • 10
  • g

    Guoying Qi

    06/25/2022, 3:58 PM
    A issue being submitted to github: https://github.com/PrefectHQ/prefect-helm/issues/6 Helm installed Prefect 2.0 with postgresql as backend, and ingress enabled with host address prefect.example.com
    prefect config set PREFECT_API_URL="<http://prefect.example.com/api>"
    the flow run states being stored in the database successfully, you can query them in the database table. But nothing shown in the UI, because all the API calls from the UI page still go to local server: http://127.0.0.1:4200/api/
    k
    • 2
    • 4
  • z

    Zheng Cheng

    06/25/2022, 4:01 PM
    Hi, this code doesn’t make sense at all
    m
    k
    • 3
    • 13
Powered by Linen
Title
z

Zheng Cheng

06/25/2022, 4:01 PM
Hi, this code doesn’t make sense at all
https://orion-docs.prefect.io/api-ref/rest-api/
of course it doesn’t work
m

Michael Adkins

06/25/2022, 4:22 PM
app = create_app()
should get you there
We refactored the app to use a factory like that and the import got updated but the example did not
You can view the API documentation locally by running
prefect orion start
and visiting
localhost/api/docs
and we have all the API documentation hosted in the docs you’re currently viewing.
👍 2
z

Zheng Cheng

06/25/2022, 4:27 PM
Thanks!!!
m

Michael Adkins

06/25/2022, 5:43 PM
Oh sorry it looks like it’s actually broken in the hosted documentation now, we’ll investigate that
👍 1
z

Zheng Cheng

06/27/2022, 1:34 PM
This is still not fixed
from prefect.orion.api.server import create_app
app = create_app()
openapi_doc = app.openapi()
unreadable but I guess it just json for machine
m

Michael Adkins

06/27/2022, 3:03 PM
What’s still not fixed? The example?
It’s been fixed internally but is not published yet
z

Zheng Cheng

06/27/2022, 3:18 PM
OK, then it should be published.
k

Kevin Kho

06/27/2022, 3:59 PM
It’s not practical to publish every change because we have to sync the docs with the codebase so we release as a version. If we released every change, you’d see a lot of documentation out of sync with the API
View count: 5