https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • s

    Simon Macklin

    12/05/2022, 6:50 PM
    hey prefect we had some strange behaviour with our on prem prefect v1 stack, one hosted in aws and another in azure where flows will not run and we also had an unusual amount of Postgres activity, these are completely isolated from each other too would any issues with prefect cloud cause potential problems with the community v1 on prem solution?
    1️⃣ 1
    ✅ 1
    j
    • 2
    • 6
  • m

    Mokshith Voodarla

    12/05/2022, 7:13 PM
    Hi! I was thinking of using Prefect but I'm not sure if it works well for my task. I am running large pools of GPU workers nodes which are orchestrated via some separate set of logic, and latency matters a lot. My thought was to set up Kafka as a messaging queue that all these workers listen too since it also offers the ability to do kSQL joins which are beneficial for me to say match up frames and frame metadata that might've been processed by different models upstream. I was also concerned about somehow having a warm pool of workers so the models don't need to cold start every time. I wouldn't even think about a workflow orchestrator but I want to have the benefits of task observability, logging, etc which it feels I wouldn't get if I used Kafka by itself.
    p
    • 2
    • 6
  • c

    Chanaka De Silva

    12/05/2022, 8:16 PM
    All, I am running a flow using prefect UI and hitting a
    ModleNotFound
    error. I register the code as follows
    prefect register --project CHX -p sparsify.py -n "sparsify" --label chx-2022-2.2
    The folder where
    sparsify.py
    sits has the missing module’s python code; masks.py. And I import the module and relevant functions as follows
    from masks import MaskClient, combine_masks
    Somehow when I run sparsify via UI it can’t find it. The code just runs fine in an
    ipython
    session
    ✅ 1
    k
    • 2
    • 2
  • j

    Jean-Michel Provencher

    12/05/2022, 8:22 PM
    Hello, the doc mentions that the .prefectignore file supports the git ignore patterns. However when testing I see that this does not support wildcard like "*" exclude everything. Am I the first to face this ?
    k
    • 2
    • 9
  • k

    Kristian Andersen Hole

    12/05/2022, 9:49 PM
    Hello, Prefect community! I have a question. I’m considering using ECSTasks for my execution layer setup, but wanted to verify something first. From www.prefect.io/guide...#NewDockerdeploymentexperience
    If your deployment defines
    -ib docker-container/prod
    and doesn’t specify any remote storage block, Prefect will assume that the flow code is baked into the image.
    Does this hold true for
    -ib ecs-task/prod
    as well?
    :aws: 1
    k
    t
    • 3
    • 3
  • n

    nchopra

    12/05/2022, 10:59 PM
    Hi Everyone! I am looking for some advice on Prefect-DBT Core integration. My company uses Prefect version 1. I initially started with cloning DBT repo into Prefect method(referred online resource) which worked fine locally but not inside the Prefect cloud as our internal org repos have strict security policies that don't allow them to be cloned inside 3rd party products. Hence, I decided to try out the git submodule method where the DBT repo resides as a submodule inside Prefect repo and I use DBTShellTask to execute the DAG commands(like below). The flow works fine in my local but in Prefect, it fails due to statement helper_script=f"cd { DBT_DIR}" as it is unable to access the right folder. It evaluates helper_script as (cd: /Users/my-username/Development/prefect-repo/containers/dbt-prefect-container/dbt-repo)
    dbt = DbtShellTask(
            return_all=True,
            profile_name="dbt_repo",
            environment="dev",
            # profiles_dir=".",
            overwrite_profiles=True,
            helper_script=f"cd {DBT_DIR}",
            log_stdout=True,
            log_stderr=True,
            dbt_kwargs={ # removed for readability}
            )(command="dbt docs generate")
    The folder structure is like: prefect-repo |- containers |- dbt-prefect-container |-dbt-repo(submodule) |- flows |-flow.py (from where I want to access the location DBT_DIR) My questions are: 1. Is it okay to use git submodule method inside Prefect? 2. How to pass correct repo location so that Prefect cloud doesn't start the root at local dir: /Users/my-username
    1️⃣ 1
    m
    • 2
    • 1
  • j

    John

    12/06/2022, 12:44 AM
    How to set the time threshold for a flow run to be considered "Late"? (e.g. whether to wait 30 seconds or 30 minutes before marking a flow run as late) Related previous posts (none of those was ever answered): • https://prefect-community.slack.com/archives/CL09KU1K7/p1670260486146599 • https://prefect-community.slack.com/archives/CL09KU1K7/p1670246788470149 • https://prefect-community.slack.com/archives/CL09KU1K7/p1668495473066159
    ✅ 1
    m
    • 2
    • 3
  • m

    Michał Augoff

    12/06/2022, 1:38 AM
    hello! is it possible to build/use custom infrastructure types (beyond what’s available in the
    prefect.infrastructure
    module)? I assume no since this needs to be present in the agent’s env but double checking
    ✅ 1
    m
    • 2
    • 5
  • t

    tsteijvers

    12/06/2022, 8:39 AM
    Hi, In Prefect 1 we often see flows stuck in Running state because the underlying k8s pod might have been terminated. Prefect is for some reason not able to detect this and the flow stays in a running state. In one case the logs show
    Flow run RUNNING: terminal tasks are incomplete.
    as the last log message, then flow stays in running state forever. Is this a known issue?
    1️⃣ 1
    ✅ 1
    m
    • 2
    • 3
  • s

    Sunjay

    12/06/2022, 9:00 AM
    Hi Team, How do i set infra_override in the deplyments yaml to override my ECS task memory for a particular deployment. Can someone please help me with this?
    ✅ 1
    r
    • 2
    • 2
  • j

    James Zhang

    12/06/2022, 11:56 AM
    hey guys, I’m trying to implement an exponential backoff retry in a @task where I use native python http Retry class, however in the execution of the task I don’t see the retry being invoked in the Flow? can someone enlighten me on this one?
    k
    a
    • 3
    • 10
  • d

    Deepanshu Aggarwal

    12/06/2022, 1:52 PM
    Hi ! i was trying to write a flow which runs a deployment (wrapped under a task) parallely for a set of input ( in array) . and i want to perform certain actions only after all these runs have completed.
    from prefect import task, flow
    from prefect.deployments import run_deployment
    
    
    @task()
    def task_A(event):
        output = run_deployment(
            name='some deployment',
            parameters={'event': event}
        )
        if output.state_name != 'Completed':
            raise Exception("flow run not completed")
        return output
    
    
    @flow()
    def flow_B():
        a = [1, 2, 3, 4, 5, 6, 7]
        # method 1
        for i in a:
            task_A.submit(i)
    
        # method 2
        output = task_A.map(a)
        for tasks in output:
            tasks.wait()
    
        #do something when all the parallel executions have completed for task A
    this is what my flow looks like but in method 1 it runs next iteration of task_A only when previous iteration is complete in method2 it doesnt wait for the tasks to complete
    k
    • 2
    • 1
  • p

    Patrick Tan

    12/06/2022, 3:24 PM
    Hi, with Prefect 2.0, how to use command line to run a flow with parameters (prefect deployment run id ......etc)
    ✅ 1
    m
    • 2
    • 8
  • m

    Marc Lipoff

    12/06/2022, 3:57 PM
    I'm looking to try to use a 3rd party secret store (we use Hashicorp Vault, but AWS Secrets Manager, etc would be similar). In our blocks, we want to store a pointer to the secret (path, location, etc), but not the actual value. And then retrieve the value at runtime. Has this been done before?
    ✅ 1
    k
    j
    • 3
    • 5
  • a

    Alexandru Anghel

    12/06/2022, 4:08 PM
    Hi guys! I'm running a flow using the Orion dask executor on Kubernetes but i get an error on very large datasets:
    distributed.scheduler.KilledWorker: Attempted to run task prometheus_to_gcs-02d33e7b-92a6-4fe8-8258-ba9efaa3d609 on 3 different workers, but all those workers died while running it.
    What could be the problem here? Other smaller datasets are running fine. I am running it with adaptive workers (min 4, max 10). I am not sure though if I used correctly to dask configuration for what i am trying to do. Can you please have a look at my code in the thread? I am able to run the same flow in Prefect 1.4. Thanks!
    b
    • 2
    • 9
  • p

    Prem Viswanathan

    12/06/2022, 4:10 PM
    Hey Folks, how can I check the health of a Perfect Agent in 2.0? It is running inside a docker container and I’m trying to figure out a reliable health check locally instead of pinging the Orion server.
    k
    m
    j
    • 4
    • 24
  • b

    Bob Colner

    12/06/2022, 4:54 PM
    Does prefect 2.0 have a way to tell a task to wait for manual intervention? For workflows with humans in the loop.
    ✅ 1
    k
    • 2
    • 2
  • j

    Joseph Loss

    12/06/2022, 5:19 PM
    deploying Azure Container Instances - I keep getting a timeout waiting on container creation. Has anyone else experienced this?
    ✅ 1
    r
    • 2
    • 17
  • x

    Xavier Babu

    12/06/2022, 11:07 PM
    After installed Prefect Orion 2.7, I am getting the following exception and I couldn't see any of my existing flow-runs in Orion UI. TypeError: unsupported operand type(s) for -: 'DateTime' and 'NoneType'
    ✅ 1
    m
    a
    • 3
    • 14
  • z

    Zac Hooper

    12/06/2022, 11:35 PM
    I have a flow that is occasionally failing with the following error: State message: Given states: total=2, not_final=2, 'RUNNING'=1, 'PENDING'=1. Both of the tasks in the flow finish as completed as seen in the photo. The flow's code looks like this:
    @task
    def prod_9e_etl_asx_anns():
        logger = get_run_logger()
        res = invoke_lambda("prod_9e_etl_asx_anns", {}, True)
        <http://logger.info|logger.info>(res)
        return ""
    
    
    @task
    def prod_9e_etl_asx_ann_sentiment():
        logger = get_run_logger()
        res = invoke_lambda("prod_9e_etl_asx_ann_sentiment", {}, True)
        <http://logger.info|logger.info>(res)
        return ""
    
    
    @flow(task_runner=SequentialTaskRunner(), retries=2)
    def perfom_asx_anns_etl():
        x = prod_9e_etl_asx_anns.submit(return_state=True)
        y = prod_9e_etl_asx_ann_sentiment.submit(wait_for=[x])
    I need to make sure the tasks happen in sequential order. Any idea how I prevent this error?
  • j

    Jarvis Stubblefield

    12/07/2022, 12:26 AM
    I’m trying to run
    prefect agent -q tenzinga_django_prod
    on my server … I’m using a
    systemd
    to run. It errors and lets me know that Orion (the server not agent) needs SQLite >= 3.24.0 but Amazon Linux 2 only has 3.7.17 as it’s latest version.
    r
    k
    • 3
    • 36
  • s

    Slackbot

    12/07/2022, 3:14 AM
    This message was deleted.
  • s

    Santhosh Solomon (Fluffy)

    12/07/2022, 6:20 AM
    @Anna Geller Is there a plan to integrate celery for task distribution in roadmap?
  • s

    Santhosh Solomon (Fluffy)

    12/07/2022, 6:21 AM
    @community, anyone used celery with Prefect for task distribution?
  • u

    陳柏翰

    12/07/2022, 7:35 AM
    Dose anyone know the status of the prefect 1.0 API? I have an internal error code 500 while requesting prefect 1.0 api.
    m
    k
    • 3
    • 3
  • n

    Nic

    12/07/2022, 11:03 AM
    Are there any best practices when it comes to setting up a dev and production env in prefect? I was thinking to put it in two different workspaces, so it would also be easier to take advantage of blocks, but I'm not sure if i could run two agents on two different workspaces on the same pc. (Would have to login to two different workspaces simultanously?) Currently i've setup two different docker blocks that injects the secrets i need in different environment into the env vars in the container, but this would lead to exposed secrets if i was to inject secrets into the docker like this. One way would be to use different Azure vaults and just inject these into the docker, but I believe the prettier solution would be to use prefect blocks. Anyone who did something similar and can share ideas or experiences? TL:DR - What is the best way to deal with secerts/connection string and other variables that span across environments? Ex.. The Server1 Var in my Dev environment == DevServer Server1 Var in Prod var == ProdServer
    r
    • 2
    • 8
  • a

    Arnoldas Bankauskas

    12/07/2022, 12:17 PM
    Hi what is minimum hardware requirements (CPU/RAM) for Prefect production installation ? with all components (SQL/UI/Engine/Agent etc.)
  • Retrieving a flow's scheduled start time from context
    l

    Liz McCutcheon

    12/07/2022, 12:34 PM
    Hi all, Is there a way of retrieving a flow's scheduled start time from context in Prefect2 ? As we want to use the scheduled time (not the actual run time) as input for some of its tasks. It was easy in Prefect 1 but can't work out how to do it in Prefect2 now that we've migrated. Thank you !
    r
    • 2
    • 2
  • j

    João Coelho

    12/07/2022, 1:33 PM
    Thanks Anna! Sorry about that, i'll keep that in mind next time :)
    🙌 1
    :gratitude-thank-you: 1
  • z

    Zachary Loertscher

    12/07/2022, 4:22 PM
    Hi all, I need to run a flow which runs a stored procedure on a SQL Server 2017 instance (which is hosted on AWS) using the python package
    pyodbc
    However, I continue to get the error
    Failed to load and execute flow run: ModuleNotFoundError("No module named 'pyodbc'")
    I have.. • added
    pyodbc
    to requirements.txt, which is installed using our
    Dockerfile
    with
    RUN pip install -r requirements.txt
    Info: Prefect version: 1.2.4 Deployment: AWS CDK for AWS Where does your docker container live?: an ECS Fargate deployment. What OS runs on your docker container? Amazon Linux 2 Any ideas? Is Prefect just not finding the package?
    1️⃣ 1
    m
    • 2
    • 3
Powered by Linen
Title
z

Zachary Loertscher

12/07/2022, 4:22 PM
Hi all, I need to run a flow which runs a stored procedure on a SQL Server 2017 instance (which is hosted on AWS) using the python package
pyodbc
However, I continue to get the error
Failed to load and execute flow run: ModuleNotFoundError("No module named 'pyodbc'")
I have.. • added
pyodbc
to requirements.txt, which is installed using our
Dockerfile
with
RUN pip install -r requirements.txt
Info: Prefect version: 1.2.4 Deployment: AWS CDK for AWS Where does your docker container live?: an ECS Fargate deployment. What OS runs on your docker container? Amazon Linux 2 Any ideas? Is Prefect just not finding the package?
1️⃣ 1
m

Mason Menges

12/08/2022, 7:44 PM
Hey @Zachary Loertscher Were you able to sort this out? When you updated the Image did you also re-register the flow? In theory if your just using Latest for the image this may not matter but it's worth mentioning for the sake of argument Ultimately it sounds like the flow may not be running the correct container if you've built the new image and deployed it thus why that error would be displaying. It could also be a Path issue in the Image but that seems unlikely
z

Zachary Loertscher

12/08/2022, 7:51 PM
@Mason Menges Unfortunately I haven't figured it out yet. Yeah, the flow was re-registered. How does the flow interact with the image? Is that something I could configure for a
UniversalRun
? Also, not sure if it matters, but this is running via a Local Agent that gets booted up from the
dockerfile
m

Mason Menges

12/08/2022, 8:08 PM
Hmm Thinking out loud, Essentially that means when the agents submits the flow run it's running as a local process within the agent when you register the flow it's going to assume the "path" to the flow/modules is based on where it was registered from, i.e. the local path for where this was developed, could try setting the path through an environment variable in the docker image prior to installing the requirements. https://docs-v1.prefect.io/core/concepts/configuration.html#environment-variable-interpolation, short version it might just be that wherever the module is getting installed prefect can't find it based on the path it's looking for it in I think. All that said you could also have the agent running separately in a virtual machine and use dockerrun to specify the image the flow needs to run with, if this is through AWS you should be able to push the image to an ECR registry and access it there. Or you could use Local Storage and store that actual flow code within the Docker Image and run it that way https://discourse.prefect.io/t/mounting-volume-to-dockerrun-for-local-storage/722
View count: 6