https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Michael Urrutia

    10/27/2022, 12:51 PM
    Is it possible to sign in via password auth instead of the magic link? I am no longer seeing that option
    ✅ 1
    k
    • 2
    • 1
  • s

    Suresh R

    10/27/2022, 1:16 PM
    Hi, We have lot of stuff in Prefect 1.0, We are not planning to migrate to Prefect 2.0 anytime soon. is there any plan for deprecation of Prefect 1.0, will there be any impact if we are not migrating to 2.0?
    ✅ 1
    j
    • 2
    • 2
  • d

    David Elliott

    10/27/2022, 1:22 PM
    Hey folks, I believe I’ve found a bug with the
    allow_failure()
    wrapper for the
    KubernetesJob
    infra - it seems to have no effect when running with
    SequentialTaskRunner
    or
    ConcurrentTaskRunner
    . When I run the same flow locally it works as expected (downstream tasks are ran), but when it’s ran on k8s (with
    KubernetesJob
    infrastructure) it ignores the wrapper and the downstream tasks just go to
    NotReady
    as if the
    allow_failure()
    wrapper weren’t used at all. MRE in 🧵
    :gratitude-thank-you: 1
    ✅ 1
    a
    • 2
    • 6
  • q

    Q

    10/27/2022, 1:40 PM
    We're running an orion instance. Since orion is not using an externally hosted API,
    PREFECT_API_URL
    is not set (
    PREFECT_ORION_UI_API_URL
    is).
    printenv | grep -E "PREFECT.*API"
    # PREFECT_ORION_UI_API_URL=<https://domain.tld/api>
    FlowRunNotifications use
    PREFECT_API_URL
    as base api url and since
    PREFECT_API_URL
    is not set
    api_url
    defaults to a rather useless
    <http://ephemeral-orion/api>
    . I'm wondering if it would make sense to change this line to use
    PREFECT_ORION_UI_API_URL
    if
    PREFECT_API_URL
    is not set. I see that
    PREFECT_ORION_UI_API_URL
    defaults to
    PREFECT_API_URL
    , but I think if I set
    PREFECT_API_URL
    to a seemingly external url api calls from ui will stop being routed through loopback and that would be a whole thing.
  • a

    Adam

    10/27/2022, 1:49 PM
    Hi friends, I would like to make a common
    create_deployment.py
    utility that my CI/CD uses that will create and apply deployments when flows change. Instead of complicated bash, it’s easier to do it in Python. However, I’m unclear how I can get the
    flow
    object from a filepath in this case:
    deployment = Deployment.build_from_flow(
        flow=my_flow, # how do i get from a filepath to the flow
        name="s3-example"
    )
    In Prefect v1, there was a useful utility called
    from prefect.utilities.storage import extract_flow_from_file
    — my question is does such a utility exist in v2? Or alternatively, is there a better way of doing this?
    k
    • 2
    • 3
  • p

    Paco Ibañez

    10/27/2022, 1:50 PM
    Hello! is there a way to kill a flow run? I have a flow running as a kubernetes job and it fails because the image does not exist. I deleted the flow run from the UI and killed the job in k8s but a new job keeps being created. Thanks!
  • j

    Jai P

    10/27/2022, 2:00 PM
    👋 hullo! since 2.3.0 it appears we can add flow code directly into your docker image, and prefect won't need a remote storage block for a
    DockerContainer
    . I'm wondering, does that same logic apply to a
    KubernetesJob
    (since we also have to provide an image for that infra block)?
    ✅ 1
    r
    • 2
    • 2
  • s

    Sean Davis

    10/27/2022, 2:15 PM
    Is anyone working on a Google Cloud Batch backend? If not and I were interested in working on one, where would you point me as a starting place?
  • d

    Daniel

    10/27/2022, 2:39 PM
    Hello! I'm working on setting up a demo for prefect to programmatically build a deployment and run it in a docker container. I have a single script (see script in thread) that saves a storage block, an infrastructure block, builds a deployment, applies it, and attempts to run it with
    run_deployment
    . When running the script, I get
    JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    that comes from
    run_deployment
    and is thrown at
    prefect/client/orion.py", line 1329, in read_deployment_by_name: return schemas.core.Deployment.parse_obj(response.json())
    . I edited
    orion.py
    to add
    print(response.json())
    at line 1329 and then that line threw the error, but the json response still prints and it looks like a valid json response object. I'm using prefect 2.6.4. I would appreciate any help in getting the deployment working, thanks!
    • 1
    • 1
  • How to connect EFS to AWS Lambda used together with Prefect?
    v

    Viet Nguyen

    10/27/2022, 2:50 PM
    I have prefect agent running on Lambda that connected to a VPC, at the moment I allow all traffic from everywhere for the inbound Security group setting, but this is not ideal, besides port 4200, might I know what other ports I need to allow to get Prefect running on Lambda with a VPC without issue? Thank you.
    a
    • 2
    • 40
  • t

    Tim-Oliver

    10/27/2022, 2:59 PM
    Quick Q regarding task dependencies in Prefect 2.x: In Prefect 1.x I could pass `upstream_tasks=[task_a, task_b]`` to a task in a flow. What is the equivalent in Prefect 2.x?
    ✅ 1
    • 1
    • 1
  • c

    Chris Gunderson

    10/27/2022, 3:20 PM
    Hi Team, Is it possible to run an agent for two different workspaces for one container? Can we have multiple profiles inside a container? I don't think it is possible, because the Prefect CLI will be pointing to a specific workspace, but I thought I'd ask instead of just spinning up two ECS instances.
    ✅ 1
    a
    • 2
    • 6
  • j

    Jehan Abduljabbar

    10/27/2022, 4:09 PM
    Can I use both .map and .submit on a task?
    ✅ 1
    m
    • 2
    • 2
  • p

    Paco Ibañez

    10/27/2022, 5:09 PM
    Hello again! is there a way to not display flow run parameters of type pydantic.SecretStr in the UI?
    n
    • 2
    • 15
  • d

    dammy arinde

    10/27/2022, 5:28 PM
    Hello Team, is there a reason parameter entered in UI doesn't work in the run? When I make a change to the parameter in the UI, the flow still picks up the default parameter instead of the newly entered one.
    n
    • 2
    • 10
  • j

    Jon Young

    10/27/2022, 6:49 PM
    hey all, im trying to pass a task's result to a function that's not a task. I'm getting:
    TypeError: not all arguments converted during string formatting
    is there a way to get the parameter's value without giving function with a task decorator?
    ✅ 1
    n
    • 2
    • 13
  • c

    Carlo

    10/27/2022, 7:15 PM
    "attempted to send logs to Orion without a flow run id": We are running into an issue where our flow imports libraries that do some logging on import. As outlined here: https://linen.prefect.io/t/3323130/must-get-run-logger-be-called-in-each-task-flow-i-have-a-flo Is there a known work around? We tried to disable logging & re-enable logging post imports, but that didn't help.
    ✅ 1
    • 1
    • 2
  • a

    Andreas Nigg

    10/27/2022, 7:54 PM
    Hey, congrats on releasing 2.6.5. Why is it, that for manual retries via UI we need to enable result persistence? Don't we simply rerun a flow - why do we need the results from the failed flow to be persisted? (Not that it's a problem, I'd just like to understand the underlying architecture)
    👍 2
    🙌 1
    ✅ 1
    m
    o
    a
    • 4
    • 4
  • j

    Jens Freund

    10/27/2022, 11:01 PM
    Hello all, After my last problem was immediately and perfectly solved here in the channel (thank you, again, @Q !), I would like to ask if anyone could help me with a second question, concerning the same example project (https://prefect-community.slack.com/archives/CL09KU1K7/p1666555308227089)? Namely, now that the creation of the deployment on the SFTP remote storage block works, I'm unfortunately unable to start a flow run from it. Specifically, I started a local Prefect agent and afterwards a local flow run from the UI (Run -> Now with defaults). After that, the paramiko library gives the following error:
    File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
        with open(localpath, "wb") as fl:
    PermissionError: [Errno 13] Permission denied: '.'
    23:16:29.384 | INFO    | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.
    To solve the PermissionError, I tried two approaches, which unfortunately both failed: 1. Try to change the
    PREFECT_LOCAL_STORAGE_PATH
    2. Running Powershell, from which I start Prefect, as admin My problem seems to be practically the same as that reported by @Mike He in the discourse forum (https://discourse.prefect.io/t/agent-debugging-agent-execution-error-with-ftp-storage-in-windows/1587) and here in Slack (https://prefect-community.slack.com/archives/C03D12VV4NN/p1663319368100279). (The only difference I see is that in my case, the last error of the traceback comes from the paramiko library instead of fsspec.) I saw that he already developed a workaround, which however includes changing the Prefect source code which I have not tried yet. I would write details, e.g. the full traceback, in the thread. If anyone had an idea or a hint how this problem could be solved, preferably without manually changing the Prefect source code, you would help me again a lot. Thank you very much in advance!
    ✅ 1
    q
    • 2
    • 4
  • a

    Andrei Tulbure

    10/28/2022, 8:25 AM
    Hi. Does anybody had any issues with Prefect 1.x ECS deployment ? Regarding the flow register not having the Fargate machine you requested ? I registered 16vCPU and have in the task definition only a 2vCPU machine. Spoke with AWS support but it was not helpful and our AWS Account does have a vCPU limit at 2000. So enough bandwidth.
    m
    m
    • 3
    • 8
  • n

    Nic

    10/28/2022, 9:21 AM
    Hi - I have my task and flow in seperate files, but would like to use this
    def cache_within_flow_run(context, parameters):
        return f"{context.flow_run_id}-{task_input_hash(context, parameters)}"
    How would i pass the flow_run_id into the task this function and further into the task decorator?
    @task(cache_key_fn=cache_within_flow_run)
    def cached_task():
        print('running an expensive operation')
        return 42
    ✅ 1
    m
    • 2
    • 3
  • c

    Camilo Fernandez

    10/28/2022, 9:49 AM
    Hi, is it possible to add the ECR as a block in Prefect 2? I was wondering if this is doable, I'm was trying in the UI, but the credentials part got me thinking. I'm using IAM Roles to authenticate the flow runs, so I don't even have an ECR username and password which the Docker Registry block in the UI requires. Maybe someone knows a way to achieve that. Thanks in any case
    ✅ 1
    a
    • 2
    • 5
  • d

    Deepanshu Aggarwal

    10/28/2022, 11:48 AM
    i had a conceptual question about the deployment running infra. when we run same deployment multiple times (with same or different params) does each of them create new pods to run new job or are these pods created specific to flows ? and if i want my parallel processing(parallel running tasks or flows) to be distributed into multiple jobs so as to make sure the amount of resources each process is taking is not huge.. how can i achieve that ? is creating separate deployments a way to do it ?
    ✅ 1
    o
    a
    • 3
    • 18
  • c

    Carlo

    10/28/2022, 1:02 PM
    We are on the latest prefect & prefect_aws, and seeing scheduled jobs terminate without being run (intermittently). The UI for the flow run shows no logs, no task runs, and no subflow runs. In cloudwatch logs for the flow run, I found the below. Any thoughts?
    23:02:20.896 | INFO    | prefect.engine - Engine execution of flow run '682567a2-e507-4aa1-b5c7-25bb2cc2573f' aborted by orchestrator: This run has already terminated.
    c
    • 2
    • 28
  • n

    Nic

    10/28/2022, 1:05 PM
    quo https://docs.prefect.io/concepts/deployments/#parameters-in-deployments it's possible to add default parameters in yaml and python deployments... Is it possible or in the works, to also add these via the cli deploy?
    ✅ 1
    c
    • 2
    • 3
  • j

    Jon Young

    10/28/2022, 2:11 PM
    hey there, i'm trying to use a task's result as a directory name. is this possible?
    ✅ 1
    c
    • 2
    • 5
  • p

    Paco Ibañez

    10/28/2022, 2:23 PM
    Hello, I have a flow that I run on my laptop using DockerContainer infrastructure and takes around 4 minutes to complete. When I run that same flow on a kubernetes cluster using KubernetesJob it times out after 30 minutes (limit that I set in my flow). Any tips on how I could troubleshoot what is going on? I am using the default job template.
    ✅ 1
    c
    • 2
    • 19
  • v

    vholmer

    10/28/2022, 2:44 PM
    Hi, I'm running a task that creates a flow run from a deployment ID using create_flow_run_from_deployment. I used to run the following code in prefect 2.5.0:
    await my_task.submit(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)
    However after upgrading to 2.6.5 my_task isn't even being run. My workaround is to just run this:
    await my_task(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)
    But it seems really weird that submit should have stopped working. Has anyone else ran into this issue?
    k
    • 2
    • 5
  • a

    Arun Giridharan

    10/28/2022, 3:36 PM
    I'm running a Scheduled flow that calls another flow using the
    startFlowRun
    method. Initially when I was calling the first flow without using the
    wait
    parameter, the flows worked fine, but after I added the parameter, the parent flow failed with
    TypeError: Object of type SUCCESS is not JSON serializable
    . I was wondering if this is a known issue. Prefect version is 1.4.0
    m
    • 2
    • 2
  • m

    Mansour Zayer

    10/28/2022, 4:01 PM
    Hello. Prefect 1. Is there a way in a graphql query to sort nulls last like:
    order by column_name nulls last
    ? I have this line
    order_by: {start_time: desc}
    in my query. More context in comment
    ✅ 1
    k
    • 2
    • 2
Powered by Linen
Title
m

Mansour Zayer

10/28/2022, 4:01 PM
Hello. Prefect 1. Is there a way in a graphql query to sort nulls last like:
order by column_name nulls last
? I have this line
order_by: {start_time: desc}
in my query. More context in comment
✅ 1
In Prefect 1, I have a graphql query that returns the response in the screenshot. The problem is that the flow_run start_time is
null
(maybe some bug in prefect 1 cloud), and in my request, I'm ordering by start_date (
order_by: {start_time: desc}
), and I always get this null as the first one, which is breaking my code. I want null to be last, not first.
k

Kalise Richmond

10/28/2022, 9:24 PM
Have you tried to filter out the nulls? You could potentially create two qraphql calls, one with the nulls filtered out and one filtering just for the nulls and then concat the data with the nulls at the end before processing.
View count: 1