https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • j

    Jan Nitschke

    04/04/2022, 9:52 AM
    Hi Prefect, I want to run a flow on ECS and use GitHub as storage. My python code imports modules. The flow definition looks something like:
    from tasks import my_task
    from prefect.storage import GitHub
    from prefect import Flow
    from prefect.run_configs import ECSRun
    
    
    storage = GitHub(
        repo="repo",  # name of repo
        path="path/to/myflow.py",  # location of flow file in repo
        access_token_secret="GITHUB_ACCESS_KEY",  # name of personal access token secret
    )
    
    with Flow(name="foobar",
              run_config=ECSRun(),
              storage=storage) as flow:
        my_task()
    The problem seems to be that the GitHub storage only clones the single file and not the entire project which causes my import to fail. (
    ModuleNotFoundError("No module named 'tasks'")
    ) I've seen that there has been some discussion around this issue but it hasn't really helped me to solve the issue.... Is my only option to clone the repo into the custom image that I use for my ECS task? But that would mean that I would have to rebuild that image every time I change something to my underlying modules, right?
    🤪 1
    ✅ 1
    a
    • 2
    • 2
  • a

    Andres

    04/04/2022, 11:54 AM
    Hi Everyone, I'm developing a handler that will be sending slack notification to our team on flow failures with detailed information. I have managed it so far and it runs flawlessly locally. The issue comes when running it on our server, the handler sends a default notification that just says
    Some reference tasks failed.
    I was investigating a bit and seem that the state contains the result of all tasks while running it locally (
    state.result
    ) while on the server this is empty (i printed it using the logger) . Any idea on how to address this?
    :discourse: 1
    ✅ 1
    a
    k
    • 3
    • 11
  • a

    Atul Anand

    04/04/2022, 12:37 PM
    Used Map functionality, But at a given point of time, I only see hike in only one cpi. Used DaskExecutor with 8 worker and 4 thread.Can anyone tell what is the issue.
    k
    • 2
    • 1
  • t

    Tom Klein

    04/04/2022, 1:16 PM
    Hello - I have a Q about ad-hoc env vars when running a flow — is it possible to “save” them somehow so they can be re-used again? or do I have to refill them manually each time i wanna create an ad-hoc run?
    k
    a
    • 3
    • 23
  • s

    Shuchita Tripathi

    04/04/2022, 1:59 PM
    Hi. poetry add prefect[azure] is giving error. And I am guessing this is the reason, my flows are not working. Without azure they were working fine. What can be done here?
    k
    a
    • 3
    • 41
  • r

    Rajan Subramanian

    04/04/2022, 2:08 PM
    Hello all, i had my prefect orion cloud running for last 4 days with no hiccups. today i logged in and i see all the deployments disappeared from the cloud. Is there a time limit for these processes? @Anna Geller
    ✅ 1
    k
    a
    • 3
    • 19
  • j

    Joshua Weber

    04/04/2022, 2:09 PM
    Hey again everyone, is anyone implementing prefect while initializing local agents inside kubernetes pods? When our kubernetes pods die off and restart (for whatever reason) the old agents continue to live in a “dead” state in prefect. Eventually the agents accumlate and slow the whole UI down and deteriorate the performance. Anyone else have this problem or know a way to manage the agents in code?
    ✅ 1
    k
    • 2
    • 9
  • r

    Rajan Subramanian

    04/04/2022, 2:44 PM
    Hello couple questions, 1. After deployment, if i make further changes to my file. How does this new change get incorporated in the cloud? do i need to do a
    prefect deployment create deployment_name
    again for those new changes to take affect? 2) if above is true, then do i need to rerun the tasks again on the UI? 3) sometimes i inadverently press run twice and i have two running processes. Is there anyway to stop a process after it has been started? 4) when i delete the workspace, to start over, i notice when i type,
    ps aux | grep python | wc -l
    the python processes are still running and i have to do a
    pkill python
    to kill all the python processes. Is there any way that once a workspace is killed all the python processes are killed along with it?
    k
    • 2
    • 5
  • s

    Shuchita Tripathi

    04/04/2022, 2:55 PM
    For GitHub storage, can we use it to upload flows as we do with cloud? In the documentation, it asks for 'path' which is location of an already existing flow file. If yes, how can we do that?
    ✅ 1
    a
    k
    • 3
    • 3
  • b

    Bernd Grolig

    04/04/2022, 3:02 PM
    Hi, I'm new to Prefect and installed a local server. Last week everything seemed to work and I started to write my first flows. They were executed as expected, also the UI worked great. However today morning, I tried to continue working on my flows, but for some reason the UI was frozen and I just restarted the containers through docker-compose (down and up again). After that I experienced a curious / weird output in the Postgres logfile (I will explain below). Even the UI was working fine again, the logfile of Postgres was quite confusing. So I thought, that something has messed up with my agent or other communication issues between Prefect server and the database. I decided to make a new clean setup in a new clean virtual environment. I also setup a path to a new database, and deleted the .prefect folder in the home directory. But even with the clean setup the weird behavior of Postgres started right away: ==> More details in thread
    k
    a
    • 3
    • 36
  • d

    Donnchadh McAuliffe

    04/04/2022, 3:52 PM
    Hi there. I'm signed up to Prefect 2.0 cloud. I'm trying to set up a simple agent to run on a docker container. So far this is my
    Dockerfile
    for the agent:
    FROM prefecthq/prefect:2.0b2-python3.9
    
    RUN prefect cloud login --key cloud_api_key
    
    ENTRYPOINT prefect agent start 'queue_id'
    However, the second line requires me to specify the workspace - is there some flag I can add (like
    --workspace {workspace}
    ) ? There probably is a much better way to set up an agent, any other docs would be appreciated! Thank you
    k
    a
    • 3
    • 23
  • j

    joshua mclellan

    04/04/2022, 5:13 PM
    Hi, is there a recommendation for what kind of machine specs we should use to run prefect core? I have a couple instances running on t3.mediums (in separate unconnected environments) and they seem to be failing randomly after several days
    k
    • 2
    • 8
  • m

    Madison Schott

    04/04/2022, 5:21 PM
    Does anyone have a documentation framework for data pipeline/testing best practices? I guess similar to dbt's style guide but for your pipeline?
    k
    a
    • 3
    • 4
  • a

    Atul Anand

    04/04/2022, 7:09 PM
    Task was destroyed but it is pending! task: <Task pending name='Task-36427' ! Sometime I got this error. Note: I have millions of task running parallely.
    k
    • 2
    • 14
  • c

    Chris Reuter

    04/04/2022, 7:36 PM
    A rundown of a variety of 😛refect: content over the next 2 weeks! Look forward to seeing you there, and don't forget to join the #events channel to stay up-to-date. https://prefect-community.slack.com/archives/C036FRC4KMW/p1649100935959699
    :upvote: 2
    :marvin: 3
    j
    a
    • 3
    • 7
  • e

    Ethan Veres

    04/04/2022, 8:56 PM
    Hi all! My company is just starting to use Prefect as our orchestration tool for our data pipeline for end customers and I have a few questions on how best to set it all up. Is there a way to reuse a single flow for every company using our product dynamically? We want to use Prefect to orchestrate each company’s data/ETL pipeline and we have the configuration for the pipeline (which system they’re integrating with etc..) stored in our database. The ideal is to dynamically create flows per company on their own schedule. Is that possible? Am I thinking about this the correct way?
    k
    d
    • 3
    • 12
  • s

    Shiyu Gan

    04/05/2022, 2:24 AM
    In prod, what is the most typical Dask cluster set up? 1. Temporary cluster? 2. Existing cluster? If existing cluster, what is the most popular cluster manager? Yarn, K8s? Thanks!
    k
    • 2
    • 4
  • e

    Eddie Atkinson

    04/05/2022, 3:39 AM
    Is there a way to provision automations as code? Ideally I’d like to check mine into source control and use CI/CD to deploy them
    k
    • 2
    • 3
  • s

    Shiyu Gan

    04/05/2022, 4:10 AM
    Observation: So architecturally, Agent sounds like the "inverse Scheduler" that lives in client's infra. Because Prefect cloud is not allowed to tunnel into client infra, Agent fulfills the role of being an extension of the cloud Scheduler that resides in client infra.
    k
    a
    • 3
    • 3
  • j

    Jonathan Mathews

    04/05/2022, 8:43 AM
    Hello! I’m new to Prefect so please excuse the newbie question. If using Prefect Cloud, do I still need to deploy an agent somewhere? (let’s say I want to trigger a dbt run on dbt cloud). From what I can see from the docs, I always need an agent running to listen for the trigger. I was wondering I could perhaps setup an API endpoint to spin up say a docker container on demand, triggered by Prefect cloud. It looks like I still need an agent running to do that, unless I’m missing the point? Thanks
    a
    k
    • 3
    • 4
  • m

    Muhammad Daniyal

    04/05/2022, 10:36 AM
    Hello Everyone. Hope you all are doing well. I have just started with prefect. I have created a workflow containing consisting of 6 tasks. And they are defined as followed
    def fun1(): <some code here>
    def fun2(): <some code here>
    def fun3(): <some code here>
    def fun4(): <some code here>
    def fun5(): <some code here>
    def fun6(): <some code here>
    
    def execute_flow():
    	
    	@task
    	def t1():
    		fun1()
    	@task
    	def t2():
    		fun2()
    	@task
    	def t3():
    		fun3()
    	@task
    	def t4():
    		fun4()
    	@task
    	def t5():
    		fun5()
    	@task
    	def t6():
    		fun6()
    		
    	
    	with Flow('my flow') as f:
    		
    		a = t1()
    		b = t2()
    		t3(a)
    		t4(b)
    		c = t5()
    		d = t6(c)
    		
    	output = f.run()
    	
    	result = output.result[d]._result.value
    	return result
    the expected behaviour of flow would be t1 -> t2 -> t3 -> t4 -> t5 -> t6 but it is not working this way instead this is what happenning t1 -> t5{fails but shows 'success'} -> t6{fails since result from t5 are not appropiate} -> t2 -> t3{shows success message without even going through half way of function} -> t4{same result as t3} except t5 and t6, every task is time taking
    a
    • 2
    • 13
  • h

    Hawkar Mahmod

    04/05/2022, 10:54 AM
    Hey folks, I have a false positive error occurring when a flow runs. It concerns an SQLAlchemy Result object not being pickable. The flow achieves it’s objective successfully, with data inserted into a database but I still get this error. What is Prefect pickling at runtime that could be the cause of this error. Here’s the specific output from the logs:
    Unexpected error: TypeError("can't pickle sqlalchemy.cprocessors.UnicodeResultProcessor objects")
    ✅ 1
    a
    k
    • 3
    • 8
  • c

    Christian

    04/05/2022, 10:59 AM
    Hi all 👋 Just wondering: are 3rd party tasks (i.e. the Great Expectations validation) compatible with Prefect Orion? Anyone using this? I’d want to validate a dataframe from a upstream task with GE in a flow… Cheers, C
    👋 1
    a
    • 2
    • 4
  • j

    Jonathan Mathews

    04/05/2022, 11:00 AM
    Hi, I’m trying to follow this blog post here and I’m failing at the first hurdle. When I run
    prefect auth login --key <replaced with my key>
    I get the following error:
    Unauthorized. Invalid Prefect Cloud API key.
    The cloud UI says my key has been created
    a
    • 2
    • 9
  • p

    Patrick Koch

    04/05/2022, 2:35 PM
    Hi all, I've a question regarding running a Flow: I've set up Prefect-Server at Rancher, the workloads seem to be stable, UI can be reached, an Agent is registered - everything should be fine. Now I'd like to run a simple Flow on the Prefect-Cluster, using the Python script and using the Prefect Run command - unfortunately I get:
    └── 14:27:40 | INFO    | Entered state <Failed>: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/prefect-flow-example'")
    Flow run failed!
    The file "prefect-flow-example.py" is available at my local host, I've set the proper ENV variables for the connection to the Prefect-Server. Do I need to define some Storage Options before? An Azure subscription would be available. I'm curious about what you would suggest as best practice 🙂 Thank you a lot in advance! Patrick
    k
    • 2
    • 5
  • m

    Matthew Roeschke

    04/05/2022, 3:10 PM
    Hello. Is there a Python API to get the flow version given a flow name? Or is flow version only accessible via the
    prefect get flows
    cli command?
    :discourse: 1
    k
    k
    • 3
    • 4
  • m

    Matt Brown

    04/05/2022, 3:25 PM
    Greetings, is anyone else having issues logging in to prefect.io? Using google auth, i've tried multiple browsers, cleared browser cache. Seems to be spinning
    m
    k
    • 3
    • 5
  • k

    Ken Nguyen

    04/05/2022, 3:43 PM
    Hi, I’m currently pulling a dbt repo with pygit2, then executing dbt using the DbtShellTask. We have a profiles.yml in the dbt repo, but I wanted to ask if there was a way to dynamically make a profiles.yml. The idea is to have a prefect flow where users can input in a schema name (normally defined under
    schema
    within the profiles.yml), and have the dbt task within that flow output to a schema of the user’s choice.
    k
    m
    • 3
    • 6
  • j

    Jonathan Mathews

    04/05/2022, 3:50 PM
    Hi, I’m following the instructions in this great article on using ECS. I’m now setting up a custom docker image to run dbt (I’m assuming I can’t use
    DbtShellTask
    because dbt wouldn’t have been installed). It’s coming back with an error where I need to specify a
    execution_role_arn
    in the
    ECSRun
    which I’ve now done, but can I use the previously defined role, or do I need to define something else? Thanks!
    k
    a
    • 3
    • 4
  • r

    Rhys Mansal

    04/05/2022, 6:24 PM
    Hi. Is there any way to change the automatically generated flow name to include a scheduled run timestamp?
    k
    • 2
    • 3
Powered by Linen
Title
r

Rhys Mansal

04/05/2022, 6:24 PM
Hi. Is there any way to change the automatically generated flow name to include a scheduled run timestamp?
k

Kevin Kho

04/05/2022, 6:31 PM
You can use a combination of the RenameFlowRun task and pulling
scheduled_start_time
from the context
https://discourse.prefect.io/t/how-can-i-set-a-custom-flow-run-name-instead-of-using-the-randomly-generated-adjective-noun-combination/131
r

Rhys Mansal

04/05/2022, 7:02 PM
thank you for your assistance
👍 1
View count: 7