https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • m

    madhav

    02/24/2022, 5:34 PM
    Hi! We are writing out some telemetry and logs to our own postgres instance (stuff that sits outside of the purview of the prefect logs). We have a LogHandler for this, and it calls out to sqlalchemy. Is it recommended to use the PostgresExecute task as opposed to using our own bespoke DB handlers? In that case, we might handle the ORM call out with PostgresExecute.run(..)
    k
    • 2
    • 2
  • j

    Jack Chang

    02/24/2022, 10:36 PM
    When I run
    prefect agent docker start
    is there a way I can specify the image it's pulling from?
    k
    • 2
    • 3
  • s

    Sen

    02/25/2022, 9:29 AM
    Hi Good afternoon, I am trying to use the Mapping Functionality with LocalDaskExecutor to run a set of Tasks parallel, in my case (16 of them at the same time) using a simple flow. I tried both Threads and Processes as the scheduler and it didn't run the tasks parallel. But when in check the mapped function via the Schematic, I only see jobs starting and running one at a time. Please see the images below:

    Pending One Run Image▾

    Running One image▾

    Could you please let me know how to get the parallel task execution using a LocalDaskExecutor. Thanks, Sen
    👋 2
    a
    • 2
    • 10
  • l

    Luuk

    02/25/2022, 12:22 PM
    I'm a bit lost / stuck regarding DockerRun on ACR, combined with Azure storage. So, my Docker agent requires the AZURE_STORAGE_CONNECTION_STRING to pull the flow from the store. But when I pass it as an env in the DockerRun() params, I keep getting the same error that it can't be found.. And no matter what I'm changing, I can't get rid of the error.
    Failed to load and execute flow run: Exception('Azure connection string not provided. Set `AZURE_STORAGE_CONNECTION_STRING` environment variable or save connection string as Prefect secret.')
    First I was working on the Local storage, but that gave me ModuleErrors and an old prefect thread explained that it should be solved through the use of Azure/Github/etc storages.
    with Flow(
        "extract",
        storage=Azure(container="prefect-flows"),
        run_config=DockerRun(image="<http://xxx.azurecr.io/extract:latest|xxx.azurecr.io/extract:latest>", labels=["extract"], env=env_config.dict()),
    ) as flow:
        print(env_config.dict(), file=stderr)
        # Prints: {'AZURE_STORAGE_CONNECTION_STRING': 'DefaultEndpointsProtocol=https;AccountName=xxxxxx==;EndpointSuffix=<http://core.windows.net|core.windows.net>'}
    a
    k
    • 3
    • 28
  • j

    jack

    02/25/2022, 3:00 PM
    Is there a way to allow archived flows to be run? Or to make it so that flows are not automatically archived when a new flow by the same name is registered? Running on ECS, often we will make some changes to the docker image, then run via prefect to test them out. But reverting to the previous flow_id gives the error
    Flow xxx is archived
    .
    k
    • 2
    • 1
  • a

    ash

    02/27/2022, 9:43 PM
    Greetings everyone, I am currently using prefer server deployed on kubernetes with gitlab storage. I want to push some modular code on git and import custom functions in my prefect script but seems like there are issues in the execution. Can anyone guide with an example or something as to how can i write a script that contains a prefect flow and import custom functions from other files in the script.
    a
    • 2
    • 1
  • b

    Brandon Roldan

    02/27/2022, 11:05 PM
    Hi. When i run the command prefect server start --expose, the graphql doesnt listen on all interface. It only listens for localhost. What should i do to fix it? Thanks
    k
    • 2
    • 7
  • j

    jack

    02/28/2022, 7:33 PM
    Are there limits on how many STDOUT lines will be captured in a given timeframe? When using
    print()
    statements in code and using the
    @task(log_stdout=True)
    decorator, running a particular flow manually emits 204 log lines, whereas only 103 lines end up in prefect.
    k
    • 2
    • 7
  • z

    Zhibin Dai

    02/28/2022, 8:14 PM
    Hi all. I have a task that returns a json. However, when I try to convert it into a python dict, I get this error. is there a way to do this conversion?
    TypeError: the JSON object must be str, bytes or bytearray, not FunctionTask
    k
    • 2
    • 2
  • a

    Austin Vecchio

    02/28/2022, 9:29 PM
    Question. If a task were to fail due to insufficient memory/cpu requirements, would it be possible to restart the task or workflow and request more resources?
    k
    • 2
    • 14
  • z

    Zhibin Dai

    02/28/2022, 9:33 PM
    Hi all, im trying to map dbt commands to a task, run_dbt_command, however i keep getting this error, Could not infer an active Flow context while creating edge to <Task: dbt run tag:>. This often means you called a task outside a
    with Flow(...)
    block. If you're trying to run this task outside of a Flow context, you need to call `DbtShellTask(...).run(...)
    . My code is below. Any ideas what's wrong? Thank you!
    k
    • 2
    • 16
  • d

    Daniel Nilsen

    03/01/2022, 9:54 AM
    Hi! How do I persist the logs generated by the server? I would like the logs to stay even if I restart the server
    a
    • 2
    • 1
  • a

    Aniruddha Sengupta

    03/01/2022, 12:33 PM
    Hi everyone I have a Prefect server and agent that is running on a virtual machine. When I add flow runs to a certain project, I add custom labels to make them easy to distinguish in the UI. What is the easiest way to update the already running Agent's labels so it can pick up new ones from the newly added flows? I have been trying to use the GraphQL to do this but I am not sure if there is built in functionality already for this? Thanks, Aniruddha
    a
    • 2
    • 6
  • r

    Ruslan

    03/01/2022, 12:35 PM
    Hi! how to parallel task working in kubernetes agent? I have got flow and 10 tasks inside and I want them to work in 3 threads. LocalDaskExecutor/DaskExecutor not working
    a
    • 2
    • 10
  • h

    Hunter Ellis

    03/01/2022, 11:58 PM
    Good evening everyone. I am trying to install prefect server and then pip install "prefect[dev]". When I do the installation on prefect[dev] I am getting the error below. I have setup a virtual environment in andaconda and I downloaded the prefect server docs from git. I'm new to programming so any help is appreciated. Creating library build\temp.win-amd64-3.10\Release\ast3/Custom\_ast3.cp310-win_amd64.lib and object build\temp.win-amd64-3.10\Release\ast3/Custom\_ast3.cp310-win_amd64.exp ast.obj : error LNK2001: unresolved external symbol _PyUnicode_DecodeUnicodeEscape build\lib.win-amd64-3.10\typed_ast\_ast3.cp310-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120 [end of output]
    a
    • 2
    • 4
  • c

    Christian Nuss

    03/02/2022, 1:19 AM
    hey everyone! thanks for having this community! i recently upgraded from prefect 2021.11.30 to 2022.01.25, and when i run
    prefect get flow-runs -s
    i get:
    prefect.exceptions.ClientError: [{'message': 'unexpected null value for type "String"', 'locations': [{'line': 2, 'column': 5}], 'path': ['flow_run'], 'extensions': {'path': '$.selectionSet.flow_run.args.where._and[0].flow._and[0].project.name._eq', 'code': 'validation-failed', 'exception': {'message': 'unexpected null value for type "String"'}}}]
    any tips on how to get around this? i do believe that
    prefect-server database upgrade
    was run post-upgrade
    k
    m
    • 3
    • 5
  • a

    Aniruddha Sengupta

    03/02/2022, 12:42 PM
    Hi All, I am trying to run a flow with some labels with
    ["aniruddhas", "parallel_example"]
    . I have an Agent whose label's are dynamically updated every time a flow is registered so that any new labels on flows are added to the agent. So now my Agent contains these labels :
    ["aniruddhas", "prefect_spider", "simple_news_spider", "parallel_example"]
    . I have now experimented with it and found that when my flow has just the label
    ["aniruddhas"]
    , the Agent is able to pick them up and run them. But if the flow run has its original labels:
    ["aniruddhas", "parallel_example"]
    , the Agent is not able to pick them up. Why is this the case? Thanks, Aniruddha
    a
    • 2
    • 4
  • m

    Mehdi Nazari

    03/02/2022, 4:01 PM
    Hi All; Are there any thoughts out there on how to make a DbtShellTask dynamic? As in certain parameters would be provided to the flow and based on those the DbtTask would get created and run subsequently? It’d be great if I can see an example.
    🙌 1
    a
    • 2
    • 2
  • b

    Bruno Nunes

    03/02/2022, 4:14 PM
    Hi, I'm doing some tests using the LocalDaskExecutor. Some flows are ending with errors with not much explanation in the pod log. Pod prefect-job-12d6087d-594qj failed. Container 'flow' state: terminated Exit Code:: 139 Reason: Error Do you know what 139 exit code means?
    m
    a
    • 3
    • 4
  • v

    Vishal

    03/02/2022, 4:51 PM
    hi all, I am running
    prefect docker agent start..
    with some labels on my local machine, and am running into this error that I am unable to figure out. I have registered my flow, and it shows up in Prefect UI, but when I run that flow, it goes into Submitted State, and the logs of prefect docker agent show this error:
    docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2/json>: Not Found ("No such container: 4675972bf8cb3506f9edd11788ac666dc61cef7d896c2e5367f38e9a9f3fdea2"
    :discourse: 1
    k
    a
    • 3
    • 19
  • z

    Zhibin Dai

    03/03/2022, 2:31 PM
    Hi. I have a flow that copies data from S3 to Snowflake. It works fine locally, but when I run it in ECS using Fargate, I'm getting a strange error.
    Failed to load and execute flow run: ImportError("cannot import name 'List' from 'pip' (/usr/local/lib/python3.7/site-packages/pip/__init__.py)")
    k
    • 2
    • 13
  • z

    Zhibin Dai

    03/03/2022, 5:08 PM
    Is there a way to pass the value of a Parameter into a PrefectSecret as a string? Is something like this possible?
    run_env = Parameter("run_env", default="DEV")
    sf_user = PrefectSecret(SF_USER + "_" + run_env)
    k
    • 2
    • 10
  • d

    Devin Flake

    03/04/2022, 6:00 PM
    Hi - I'm using Prefect Server and I'm trying to register a flow into a project but I'm getting the following error:
    prefect register --project "Tests" --path hello_world.py
    ... lots of output ...
    prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
    k
    • 2
    • 3
  • l

    Lana Dann

    03/05/2022, 12:51 AM
    hi! where can i find a list of all of the environment variables i.e.
    PREFECT__*
    that i can use to configure prefect?
    k
    • 2
    • 8
  • m

    Michał

    03/07/2022, 7:53 AM
    How to get some kind of alert like slack notification when Agent last query was more than 10 minutes:
    k
    • 2
    • 1
  • f

    Fabrice Toussaint

    03/07/2022, 9:36 AM
    Hi all, Is it possible to cancel a flow after it ran for an X amount of time? Let's say I want to cancel flows that take longer than 1 hour? How would I approach this?
    t
    k
    • 3
    • 5
  • m

    madhav

    03/07/2022, 2:24 PM
    Is there any documentation/examples/recipes for how to deploy custom private modules in a github repo with a flow?
    k
    • 2
    • 8
  • a

    Aniruddha Sengupta

    03/07/2022, 2:27 PM
    Hello all Hope that you are well. I have a flow which runs a series of shell commands but the first command is to make a Singularity image with the command
    make_shell
    . This command does not return any exit codes, so when I tried to run it as a standalone task, my server crashed. What would be the best way to ensure that this command is passed off first before executing other shell tasks? Thanks,
    k
    • 2
    • 9
  • l

    Liam England

    03/07/2022, 4:32 PM
    Hi folks, Is there a way to drop the default label when using
    prefect build
    to generate json for flow registration? There doesn't seem to be an option in the cli as there is for local agents, unless I've missed something.
    k
    • 2
    • 1
  • s

    Shuchita Tripathi

    03/07/2022, 4:59 PM
    Hi all, I am trying to get prefect server started in a VM in Azure. It is giving me this error. The permission seems to be proper. Any idea on what could be missing?
    k
    • 2
    • 6
Powered by Linen
Title
s

Shuchita Tripathi

03/07/2022, 4:59 PM
Hi all, I am trying to get prefect server started in a VM in Azure. It is giving me this error. The permission seems to be proper. Any idea on what could be missing?
I tried running the server on my local machine and everything works fine. I am now trying to run the server on Azure VM, where this issue is coming up.
k

Kevin Kho

03/07/2022, 5:02 PM
Can you try doing
docker run hello-world
on the Azure VM? Without using sudo
s

Shuchita Tripathi

03/07/2022, 5:56 PM
There was some issue with docker. followed this link to resolve https://stackoverflow.com/questions/53344380/errno-13-while-running-docker-compose-up
k

Kevin Kho

03/07/2022, 5:57 PM
Oh ok are you good now?
s

Shuchita Tripathi

03/08/2022, 4:16 PM
yes, thank you!
k

Kevin Kho

03/08/2022, 4:18 PM
No problem!
View count: 1