prefect-community #prefect-getting-started

Khuyen Tran

11/10/2022, 7:39 PM

Tip of the day: You can add

wait_for

to subflows so that they can wait for upstream tasks. In the example below,

subflow

didn’t run because the

fails

task failed. This feature is available in Prefect 2.6.6.

🦜 3

gratitude thank you 5

🙌 6

🎉 2

👍 3

🚀 1

Khuyen Tran

11/11/2022, 5:32 PM

Tip of the day: You can now read Prefect documentation in dark mode! 🌙

🔥 4

🕶️ 3

Khuyen Tran

11/11/2022, 11:05 PM

Are you trying to wrap your head around a deployment (i.e, what it does, and how to use it)? In this series of short videos, you will get some basic understanding of deployment: • What is a deployment • Create a deployment • Run a deployment

gratitude thank you 5

🙌 5

👍 5

marvin 4

🎉 2

Kalise Richmond

11/15/2022, 4:46 PM

📺 Join us today at 10 AM PST for Prefect Beginners Demo by @Taylor Curran https://www.youtube.com/c/PrefectIO

blob attention gif 2

🦜 1

Khuyen Tran

11/15/2022, 4:59 PM

Prefect now supports colored log levels in the 2.6.7 version. 🌈 Test code:

Copy code

import asyncio
from prefect import flow, get_run_logger

@flow
async def test_async_flow():
    await asyncio.sleep(3)
    logger = get_run_logger()
    logger.debug("a debug")
    <http://logger.info|logger.info>("my info")
    logger.warning("some warning")
    logger.critical("super critical")
    logger.error("error!")
    print('all done')

@flow
def test_sync_flow():
    test_async_flow()

test_sync_flow()

View more here. Kudos to @Andrew Huang for this cool feature.

terminal 6

🎨 3

🤩 7

Anjaneyulu K

11/16/2022, 1:34 AM

Hi #prefect-getting-started, Can we create dependency flows in prefect like one after other based on action completion?

Colin Bieberstein

11/17/2022, 3:29 AM

Hi everyone. I see lots of examples where people are using Pandas / SQL Alcehemy to do an extract / Load operation. What I haven’t seen are examples of how to handle data sets larger than memory for these operations. Do you advocate to running pyspark, Dask clusters or is there a mechanism to do something with ACI / ECS fargate so that a just in time just big enough worker can be launched?

Anderson Luiz Souza

11/17/2022, 11:44 AM

Hi all! A basic question: I really would like to replace those random suffixes of flows and tasks with more informative names (for example: datetime now or git commit hash). How could I do this? Thanks!

Khuyen Tran

11/17/2022, 4:11 PM

Tips of the day: In Prefect 2.6.7, you can add timeout to tasks 🕐 . If

timeout_seconds=n

and the task takes longer than

seconds to execute, Prefect will raise a TimeOut error. View more here. Kudos to @Peyton Runyan for this feature.

🙌 6

🦜 1

⏲️ 1

gratitude thank you 4

jmill

11/21/2022, 7:27 PM

I want to deploy prefect + UI on a remote machine i own and be able to tunnel into it from my main machine, ideally using docker. is there a good tutorial for this? (e

✅ 1

Boris Tseytlin

11/28/2022, 3:13 PM

Hey guys. I am creating a block from code and running

block.save("name", overwrite=True)

, but I don’t see it in UI. How can I know whatt is wrong?

Boris Tseytlin

11/28/2022, 3:14 PM

Copy code

storage_block = RemoteFileSystem(
        basepath="<s3://prefect-flows>",
        key_type="hash",
        settings=dict(
            use_ssl=False,
            key=config.STORAGE_MINIO_USER,
            secret=config.STORAGE_MINIO_USER,
            client_kwargs=dict(endpoint_url=f"http://{config.STORAGE_MINIO_URL}"),
        ),
    )
    storage_block.save("flow-storage", overwrite=True)
    <http://logging.info|logging.info>("Created storage block")

Kalise Richmond

11/29/2022, 4:41 PM

📺 Come join @Matt Conger at 1 PM EST for a Prefect Beginners Demo live on youtube. https://www.youtube.com/c/PrefectIO

prefect duck 3

P 3

gratitude thank you 1

🙌 1

Sebastian Steele

11/29/2022, 7:10 PM

hey everyone question, let say I am importing a helper file

Copy code

import helper_file.py

'''
flow that uses helper_file
'''

Sebastian Steele

11/29/2022, 7:11 PM

If i make changes to the helper_file do I have to redeploy my flow?

Jordy Wegman

11/29/2022, 8:05 PM

Hey, is there maybe a video on how to get the UI up and running? I can't seem to get 'prefect orion start' to work. My terminal keeps giving this error. I'm probably doing something stupid

✅ 2

11/30/2022, 3:00 AM

asking for some feedback is this simple flow is correct to explain how prefect works to others.

❤️ 3

Khuyen Tran

11/30/2022, 4:30 PM

Tips of the week Prefect logger allows you to leverage Rich to insert their own custom color / styling. Here is an example:

Copy code

from prefect import flow, get_run_logger

@flow
def cow_say():
    logger = get_run_logger()
    <http://logger.info|logger.info>("I'm in [red]danger[/red]!")

if __name__ == "__main__":
    cow_say()

View more styles in Rich’s documentation.

gratitude thank you 4

🎉 4

🙌 5

11/30/2022, 5:02 PM

Is there a page in the documentation showing how to properly implement cicd with github ? I'm kind of confused about how to used deployment yaml files when it comes to automation. Thanks

✅ 1

Uzoma Emuchay

12/01/2022, 2:26 AM

Hey all, is there any documentation beyond the getting started page on best practice for my use case. I'm attempting to deploy data bricks jobs that create tasks within a job dynamically based on certain conditions. If anyone can point me in the right direction, that would be greatly appreciated.

Dragan

12/01/2022, 11:10 AM

Hi all, maybe a basic question, but here goes We have a flow that is triggered to run every hour In random situations we have longer calculations that make the job run for more than an hour Is there a validation that can be added that a flow should not start before previous one finishes, something like

wait until finish

if it is running in same flow, we can check triggers, but can we do it on the flow level?

12/02/2022, 5:02 AM

Hi again, may i know based on https://docs.prefect.io/tutorials/deployments/ how can i leverage the use of config.json file ? is it customable on storage like prefect parameters? tq for your advice

✅ 1

Jordy Wegman

12/02/2022, 8:24 PM

Hey, I'm working my way trough the tutorial and I'm getting some errors on the 'flow and task configuration'. It keeps telling me it 'expected an indented block'. But the code seems fine as it's copied from the Prefect site. I found this link which explains it well, but it doesn't solve my issue since everything looks good. All other script examples do work for me. Anyone got an idea?

✅ 2

Zack

12/03/2022, 7:08 PM

Is there way to pick up cloudwatch events through prefect and communicate the information via email or slack notification?

✅ 1

Anthony

12/07/2022, 3:41 AM

So our DevOps team has implemented Prefect 2.0 on Kubernetes. How do I go about connecting to the Kubernetes cluster to build and deploy prefect code?

✅ 1

Matthew Ferry

12/07/2022, 12:42 PM

I've been struggling to understand how different pieces of Prefect communicate with each other. Is this an accurate representation of the basics? I want to make sure I know where traffic is going to ensure my deployment has the right networking and access. If I have this right, I'd like to expand it to cover the development workflow.

😍 2

Daryl Thomas

12/08/2022, 2:09 PM

I have run prefect server on a single VM, On accessing the ui everything seems to be loading very slowly. In the console i see a lot of Err Connection timed out for the 4200 graphql. I have followed the steps in https://docs-v1.prefect.io/orchestration/server/deploy-local.html#ui-configuration. im running on an aws ec2 instance. When i run the sample flow it is stuck at waiting for flow runs

1️⃣ 2

Delete a custom Prefect Block

Nils

12/08/2022, 2:27 PM

I was playing around with custom blocks. Registered a couple of with the Python command

register_type_and_schema()

. However, now I want to delete those blocks but I can not find a way to do so..

✅ 1

Bryce Morrow

12/08/2022, 6:07 PM

Hey everyone, been playing around with prefect the last few days and really enjoying using it so far, however, I'm running into some issues with my workflow / setup and would appreciate some feedback / help 🙂 My environment is setup like this • I'm working in jupyter-lab environment (installed globally). • I have a python 3.10.8 conda kernel installed in this environment, and I'm using this kernel to run my notebooks. My workflow is like this • Define a number of tasks. Some tasks involve expensive api calls, so for these I set

persist_result=True

, which helps speed up my workflow considerably. For simpler data transformations on the results of these expensive tasks, I don't tend to persist the results. • For the tasks that I am persisting, their implementations often change as I iterate on ideas. So, for example, maybe I change the implementation of the task

one

to return

instead of

. Whenever I change the implementation of one of my tasks with persisted results, I change the cache key. For this example, I'd change

lambda *args: "one"

lambda *args: "one1"

. A simplified example:

Copy code

@task(persist_result=True, cache_key_fn=lambda *args: "one")
async def one(): 
    return 5

@task(persist_result=True, cache_key_fn=lambda *args: "two")
async def two(): 
    return 2

@flow(cache_result_in_memory=False)
async def flow_test():
    a = await one()
    b = await two()
    return a + b 

await flow_test()

This seems to be working at the start, but eventually, I start running into an issue where tasks fail to run due to an error in decrypting their block data. I'll share the exact error log in the replies. I haven't yet determined the exact cause of the problem but I have some suspicions / notes. The first concerning thing is that when I re-run this cell, say after changing a task implementation, I get some warnings that look like this:

Copy code

/opt/miniconda3/envs/flywheel/lib/python3.10/site-packages/prefect/tasks.py:206: UserWarning: A task named 'one' and defined at '/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py:1' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:

 `@task(name='my_unique_name', ...)`
  warnings.warn(
/opt/miniconda3/envs/flywheel/lib/python3.10/site-packages/prefect/tasks.py:206: UserWarning: A task named 'two' and defined at '/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py:5' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:

And this makes sense, since I'm defining a task with the same name again. However, when I look at the path

/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py

, it doesn't exist. However, there are different folders stored at the path

/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/

with names of the form

ipykernel_<some_number>

. If I look at the contents of these directories, they contain small snippets of code from my notebook but they aren't the task definitions themselves usually. So I think the ipykernel is storing this information temporarily but then it changes locations, contents, disappears, etc. I've also tested running the notebook end to end, having everything work fine, then restarting the conda kernel and re-running the notebook. This causes the error I'll share below. This strongly suggests to me that the way the kernel is managing the location of the notebook source code is the culprit here. Another point is that if I clear the orion database using the command

prefect orion database reset -y

then my issues are resolved. However, I would like to determine the issue and fix it in some better way.

Tony

12/09/2022, 3:04 PM

Good morning Everyone. It's been a while but I decided to upgrade to Orion. I see that a lot has changed. I am perplexed by a few steps as i'm trying to get Orion started. I am using VENV into which I have installed prefect 2.7.0. I am having a lot of trouble setting the API addresses, using and setting a profile, and just getting things to work. AT first, the prefect config set PREFECT_API_URL command does not seem to be working. I then export PREFECT_API_URL, while showed correctly as I started prefect Orion, but in either case, once browsing to the dashboard, I get an error saying 127.0.0.1:4200/api is not available. So i seek help in ALL the steps necessary to get this working and available. I have scoured the prefect.io docs, but the section that deals with "Running Orion for the First time" seems to be completely missing. I am anxious to explore the new features that are available in 2.0(Orion). Thanks for any help you can provide.

✅ 1