https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • k

    Kyle McEntush

    12/24/2022, 7:19 PM
    I hope this is a relatively simple question: using prefect 2.0, how can I get a task to return two values? as a function it is something like
    def f():
    ✅ 1
    r
    • 2
    • 2
  • k

    Kyle McEntush

    12/24/2022, 7:20 PM
    i know prefect 1.0 had
    nout
    as a task arg, but I can't find the equivalent in the docs...
    ✅ 1
  • s

    Shubham Wani

    12/24/2022, 7:39 PM
    #needHelp Using Prefect 2.7, I'm trying to connect a MYSQL databse to write data through a prefect task. once I run
    cursor.executemany(query, values),
    I run the
    connection.commit()
    to to transfer all data from buffer to the table in dataset. Error faced in the following command:
    connection.commit()
    Error:
    00:12:33.556 | ERROR   | Task run 'load_records-41eb504c-0' - Encountered exception during execution:
    Traceback (most recent call last):
      File "/home/shubhamwani/nice/edilitics-server/edilitics-env/lib/python3.8/site-packages/prefect/engine.py", line 1346, in orchestrate_task_run
        result = await run_sync(task.fn, *args, **kwargs)
      File "/home/shubhamwani/nice/edilitics-server/edilitics-env/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread
        return await anyio.to_thread.run_sync(call, cancellable=True)
      File "/home/shubhamwani/nice/edilitics-server/edilitics-env/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "/home/shubhamwani/nice/edilitics-server/edilitics-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
        return await future
      File "/home/shubhamwani/nice/edilitics-server/edilitics-env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
        result = context.run(func, *args)
      File "/home/shubhamwani/nice/edilitics-server/prefect_flows/load/prefect2_tasks.py", line 166, in load_records
        (
    TypeError: cannot unpack non-iterable NoneType object
    I've tried using different sql connectors(mysqlclient, mysql-connector, pymysql), but the error persists. is this somehow related to prefect? anyone else facing this issue? Any help is welcomed, Thanks.
    ✅ 1
    s
    • 2
    • 2
  • s

    Shruti Hande

    12/26/2022, 4:40 AM
    prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/2f48cb6f->
    0049-4569-879d-7123d9113e31/workspaces/9da885d9-9d9f-4ee9-9b42-84bfb4a3d9aa/work_queues/name/queue_5'
    Response: {'exception_message': 'Internal Server Error'}
    ✅ 1
    j
    • 2
    • 2
  • a

    Aditya Sharma

    12/26/2022, 8:03 AM
    Hi i am trying to restart the docker agent which is running on linux. i am not able to restart the docker agent
  • a

    Aditya Sharma

    12/26/2022, 8:03 AM
    and the flows are failing as below
  • a

    Aditya Sharma

    12/26/2022, 8:04 AM
    anybody who can add some views on this
  • j

    Jelle Vegter

    12/26/2022, 1:32 PM
    Hi, it looks like the agent isn’t healthy for some reason. Are there logs from the agent?
  • k

    Kelvin DeCosta

    12/26/2022, 1:50 PM
    Never used Azure, but my bet would be on the
    prefect
    blocks not being registered. The hosted agent must run something that looks like this:
    prefect block register -m prefect_azure.container_instance
    j
    • 2
    • 4
  • o

    Ollie Sellers

    12/26/2022, 9:33 PM
    Hello everyone, I have a quick couple of questions around persistent input and result storage: 1) will restarted mapped tasks use results for the partial mapped result or does the task need to finish completely? 2) will the stored results be used even if there are changes to the code in the task if params are the same? Apologies if these are easily answerable through the docs, I couldn't locate the answer in my first search. I am also currently using v1. Thanks, Ollie
  • r

    Rikimaru Yamaguchi

    12/27/2022, 12:58 AM
    Is there any way to check if my flow is a sub flow or not? thank you.
    ✅ 1
    a
    • 2
    • 2
  • d

    David Cheedie

    12/27/2022, 2:54 PM
    Good morning, we received this error on our remote machine which is running prefect agent as a systemd service
  • d

    David Cheedie

    12/27/2022, 3:03 PM
    Good morning, we received this error over the weekend on our remote machine which is running prefect agent as a systemd service
    Dec 24 22:04:58 linux prefect[279492]: 22:04:58.602 | ERROR   | prefect.agent - Server error '500 Internal Server Error' for url
    The service failed, tried to restart a few times, was unsuccessful, and then our scheduled jobs on prefect cloud switched to state=Late I'd like to rule out that this was an error on the prefect cloud side, does anyone know anything about prefect cloud issues around 12/24 22:04 UTC?
    ✅ 1
    b
    • 2
    • 2
  • m

    Michael Levenson

    12/27/2022, 4:41 PM
    Hi all I am getting a weird bug in production for a flow with a kubernetes run config that hangs when I redeploy it
    Prefect Core Version: 1.2.4
    I trigger a flow run from a github actions step and it just hangs (happened yesterday and the run state was still in progress for 2 days) even though the job does not exist I know the flow did not finish because I see that my ml model tracking service did not register a new model I triggered the flow manually and it successfully completed. This flow is also scheduled and completes. What could be causing this issue?
    m
    b
    • 3
    • 2
  • p

    PKay

    12/27/2022, 5:14 PM
    Hi all, anyone found some good documentation on how to build custom docker images. Really confused by the Prefect blocks UI and logging not providing any information. Usually the agent crashes and resulting in the flow run to enter the state of "PENDING" or "CRASHED", provides no information why the image (from registry) wasn't pulled and executed.
    ✅ 1
    b
    • 2
    • 2
  • c

    Chris Whatley

    12/27/2022, 5:54 PM
    Is prefect_slack deprecated? I can’t get it to be recognized as a block with cloud or with a local orion setup that I’m testing.
    ✅ 1
    n
    a
    m
    • 4
    • 8
  • j

    John-Craig Borman

    12/27/2022, 7:35 PM
    Hi all, I'm working on unit testing some flows in Prefect 2 - what are the best practices for populating Blocks via Pytest fixtures?
    👀 1
    ✅ 1
    a
    • 2
    • 1
  • s

    Sebastian Steele

    12/27/2022, 8:57 PM
    Hey everyone is there a way to get a workers poll history through the CLI? i want to see if one failed
    a
    • 2
    • 1
  • m

    Matt Morris

    12/27/2022, 9:06 PM
    Hey everyone, just upgraded from prefect 1 to 2 and enjoying a lot of the new features, especially the
    ConcurrentTaskRunner
    ! However, I am noticing that the speed of using
    prefect_dask.DaskTaskRunner
    is slower than using
    prefect.executors.DaskExecutor
    (prefect 1). In both cases I was using processes rather than threads. Speed depends on the task, but some tasks that read from s3 are taking a little over twice as long with the
    DaskTaskRunner
    . Has anyone faced something or similar, or any pointers as to why I may be experiencing this? Thanks!
    👀 1
    a
    • 2
    • 9
  • r

    Rikimaru Yamaguchi

    12/28/2022, 5:30 AM
    hello. Is there any way to run flow triggered api? Thanks!
    ✅ 1
    k
    b
    • 3
    • 4
  • m

    Mohit Singhal

    12/28/2022, 11:45 AM
    Hi , is there any way to use .fn() and .submit() simultaneously while calling a task function. considering we have to pass parameters as well?
    k
    m
    p
    • 4
    • 11
  • a

    Abosede Oladipo

    12/28/2022, 12:59 PM
    Hello.
  • a

    Abosede Oladipo

    12/28/2022, 1:47 PM
    Hello everyone Is there a way to get a list of all blocks in a workspace using Python code? I know that it can be done via the cli with
    prefect block ls
    , but I can't seem to find anything in the docks using python code.
    ✅ 1
    p
    • 2
    • 3
  • t

    Tadej Svetina

    12/28/2022, 6:00 PM
    Hi, is there a way to use S3 storage (for deployments/blocks) without explicitly providing the secret key/id? I would like to have the authentication handled automatically by AWS SDK, and will be provided using env vars or instance metadata
    j
    • 2
    • 2
  • d

    Dalin Stone

    12/28/2022, 7:25 PM
    Hi Team! 👋
  • k

    Ken Nguyen

    12/28/2022, 7:53 PM
    Hi! I recently had a working config of my Prefect 1 flow by importing the following modules:
    import prefect
    from prefect import task, Flow
    from prefect.storage import GitHub
    from prefect.run_configs import ECSRun
    from prefect.client import Secret
    However, when I run my code now (Prefect 2.7.4), I get a
    ModuleNotFoundError: No module named 'prefect.storage'
    error (same goes for
    prefect.run_configs
    and
    prefect.client
    ). Was there a recent update to the way we should import these modules?
    m
    • 2
    • 7
  • c

    CA Lee

    12/29/2022, 8:39 AM
    Hello, I have a question regarding scheduling (exceptions)
    ✅ 1
    r
    • 2
    • 10
  • p

    Paco Ibañez

    12/29/2022, 3:03 PM
    Hello! I noticed that flow function comments are shown as description in the flow's deployment, is there a way to format the comments? I have tried html and markdown with no luck.
    @flow
    def hello():
       """
       Is there a way to get the text in here to be formatted in the awesome prefect UI?
       """
       pass
    ✅ 1
    j
    • 2
    • 2
  • c

    Chris Gunderson

    12/29/2022, 3:57 PM
    Hi Prefect Team, We have our dev environment up and running for Prefect 2 now. When setting up the Prefect 2 production environment we encountered an issue where we cannot access our database. We are using an ECS Task to run our flows in dev and prod. The dev instance is able to read data and execute the program, but production is not able to. The ECS Task in production is using the same VPC, Subnet, routes and security group as development. Is there another possible reason for this failure?
    ✅ 1
    n
    • 2
    • 8
  • d

    Dalin Stone

    12/29/2022, 4:20 PM
    Hi Team! 👋 Is it possible to have a custom docker image in your local file system be referenced by the docker-container infrastructure block when building a deployment or am I required to have my custom docker image on Docker Hub/Docker Registry and access it through those systems?
    ✅ 1
    a
    • 2
    • 6
Powered by Linen
Title
d

Dalin Stone

12/29/2022, 4:20 PM
Hi Team! 👋 Is it possible to have a custom docker image in your local file system be referenced by the docker-container infrastructure block when building a deployment or am I required to have my custom docker image on Docker Hub/Docker Registry and access it through those systems?
✅ 1
a

Anna Geller

12/29/2022, 4:44 PM
yes it is, you would need to set the image pull policy to Never, then Prefect will only look for images available locally
🙌 2
d

Dalin Stone

12/29/2022, 5:27 PM
You're the best Anna! 🥂
😄 1
@Anna Geller How can I pass the '-p {host_port}:{container_port}' option to the docker infrastructure block to properly map ports? When I try to connect to a external/remote mysql server instance from my code, I receive error '2002, can't connect to mysql server {my_server}, 115' when running the deployment. I think this is because my container doesn't have the properly mapped ports to connect to an external/remote mysql server. None of the documentation mentions how to pass the -p port argument
a

Anna Geller

12/30/2022, 12:58 AM
what's your setup? are you on Prefect Cloud or self-hosting? if you're self-hosting or just exploring things locally with docker, check out this docker compose setup https://discourse.prefect.io/t/running-prefect-2-with-docker-compose/1516
✅ 1
d

Dalin Stone

12/30/2022, 3:12 PM
@Anna Geller We are using Prefect Cloud, is it possible to pass that -p in the docker container infrastructure block or is there another way we could map ports for the container?
This works great, thank you Anna! 🥂
🙌 1
View count: 3