prefect-community #prefect-community

Idk if this is realistic, but I have my current Git branch shown in terminal. With 2.0 being so terminal based, having a little shortcut to put in .zshrc... that shows the current prefect profile you're using would be HUGE. Would likely reduce errors longterm when accidentally working in the wrong workspace.

✅ 1

Blake Stefansen

09/01/2022, 4:41 PM

Hi Everyone, What is the right format for passing in parameters to my deployments using the python library? I can't seem to get my parameters to show up in orion Version: 2.3.0

✅ 1

Adam Brusselback

09/01/2022, 5:17 PM

Hmm... in some cases prefect works fine with my domain name as the PREFECT_API_URL and in other cases it doesn't work...quite confused.

✅ 1

Venkat Ramakrishnan

09/01/2022, 5:31 PM

Does anyone know of a bug where even though the frequency of the schedule is once in an hour, multiple runs are created for that hour? The flows keep increasing every minute. I now have 5 runs for the same hour! Here is my deployment code:

Copy code

deployment = Deployment.build_from_flow(flow=print_pipeline_hour,\
                                        name="Hourly Pipeline Deployment", version="1", tags=["Iris"],\
                                        schedule={'rrule': 'FREQ=HOURLY;UNTIL=20220912T040000Z', 'timezone': "Asia/Kolkata"},
                                        work_queue_name="hour-work-queue")

✅ 1

Henning Holgersen

09/01/2022, 5:52 PM

Has anyone used Prefect together with Meltano? I am able to get it to work locally with an agent and a shell task, but I have not been able to build a docker image containing both the meltano project and the prefect agent - although Meltano has built-in functionality for containerisation.

👀 1

✅ 1

Krishnan Chandra

09/01/2022, 6:34 PM

Hey folks! I’m wondering what would be a good pattern to set up long-running Prefect agents in Prefect 2.0. It looks like the

DockerContainer

and

KubernetesJob

create new containers/jobs per flow run, but what I’d ideally like to do is run a fleet of long-running workers that process from the queue. What’s the best infrastructure to choose in Prefect 2.0 to achieve this?

Kevin Grismore

09/01/2022, 6:36 PM

is there anything preventing the GitHub block from working with non-GitHub repos, like GitLab?

✅ 1

Parwez Noori

09/01/2022, 6:43 PM

Hi everyone! We are using Prefect 2.0 with Azure. Using the traditional df.to_sql is not fast for ingesting data into our Azure SQL DB. Timetable with 50.000 rows: local dask cluster: to_sql ~ 6 min. local dask cluster: to_sql with fast_executemany ~ 4 min data factory ~ 15 sec. What would you do to increase ingestion speed? Would you recommend using real Dask Executor? Any help or ideas is appreciated!

✅ 1

kiran

09/01/2022, 9:56 PM

Hi all. In Prefect 2, is the default task runner the

Concurrent

Sequential

? In one section, the docs say the default is

Concurrent

but then in another section, they say “Make sure you use
.submit()
to run your task with a task runner. Calling the task directly, without
.submit()
, from within a flow will run the task sequentially instead of using a specified task runner.” which seems to imply that the default is actually

Sequential

✅ 1

Blake Hamm

09/01/2022, 11:23 PM

Prefect 2.3 is exciting for sure! The speed of development is wild and I'm trying to keep up! I'm wondering if

KubernetesJob

blocks can interact with (the new)

DockerContainer

block. Is there a way to deploy with

--ib

as a

DockerContainer

as well as use the

KubernetesJob

? Or is there any roadmap to connect the two together (like how the

DockerContainer

block can access a

DockerRegistry

block? I imagine this isn't feasible, but generally, I really like the ability to pass in a manifest file to a

KubernetesJob

block. Specifically, I'm using EKS on AWS Fargate and really like the ability to define the resources. On the other hand the new

DockerContainer

block seems really handy to manage environments for specific flows. Right now I have one image on ECR with all the dependencies and it's much heavier than it needs to be. From a CI/CD perspective it would be great to have an action creating

DockerContainer

blocks based on the individual flows and another action creating

KubernetesJob

blocks based on different manifest files. In an ideal world, I would love a way to deploy using a

DockerContainer

block as the image inside the

KubernetesJob

block. A current (hacky) solution could be to loop through the flow-level docker files, register them to ECR based on their flow name and create all the necessary

KubernetesJob

blocks for each individual flow. I could use some kind of "resource" tag to pick the necessary manifest file. This would create a distinct

KubernetesJob

block for every flow even though I might only be using 3 distinct manifest files (just lot's of unique containers for each flow). This would also require an --sb (unlike the new standalone

DockerContainer

block.

✅ 1

Venkat Ramakrishnan

09/02/2022, 4:27 AM

Referring to https://docs.prefect.io/concepts/deployments/, and the description for blocks is not clear. There seems to be two types of blocks 'storage block' and 'infrastructure block', but I am not sure. The documentation is referring to 'block', but it is not clear about : 1. What is a storage block and what is it used for? 2. What is infrastructure block and what it is used for ? 3. When blocktype/blockname is specified, 'blockname' is referred to as 'local-file-system' while creating the block. But where is the block created? It is not shown in the document. 4. Is remote storage a substitute for -sb or mutually exclusive? Totally confused. If someone can explain in detail, it would be helpful. Docs are not helping.

✅ 1

Andreas Nord

09/02/2022, 10:04 AM

Hi! I followed the tutorial on https://github.com/anna-geller/prefect-docker-deployment. One step of the docker file is to copy the flows:

Copy code

ADD flows /opt/prefect/flows

Which seems to be consistent with the deployment

Copy code

storage: null
path: /opt/prefect/flows
entrypoint: flows\healthcheck.py:healthcheck

It seems that the deployment worked, but I can't find this path (/opt/prefect/flows) locally. I'm on Windows

Klemen Strojan

09/02/2022, 12:04 PM

Hey all - there has been a lot of discussion and a few great resources for deploying Prefect 2.0 on AKS. https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-an-azure-kubernetes-cluster-and-connect-to-azure-blob-storage/1128 Tutorial mentions another tutorial that will describe connecting Prefect Agent on AKS with Prefect Cloud instance. Is this already available?

✅ 1

Ross Teach

09/02/2022, 1:21 PM

Using Prefect V2 Cloud, noticed a few of the following errors (8AM EST + 9AM EST). Seems to be intermittent. Is there a known issue?

Copy code

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/043b2649-9d07-4c5e-8225-521ba2275e68/workspaces/689b139b-a725-4c2b-b167-86a705b8789d/task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>

✅ 1

José Duarte

09/02/2022, 1:47 PM

Hey yall, is there a complete guide on how to deploy Prefect open source or not really?

👀 1

✅ 1

Seth Goodman

09/02/2022, 2:25 PM

Hi All - I am migrating from Prefect 1.0 to 2.0 and am specifically dealing with a mapped task parallelized using Dask. From the 2.0 docs it sounds like you need to call Task.submit() when using a task runner like Dask, but I am unclear how that is applied when using a map. Any guidance or an example would be appreciated. Thanks!

✅ 1

kwmiebach

09/02/2022, 2:46 PM

Hello 🙂 I believe I ran into a threading problem yesterday. I am converting some data extraction pipelines to prefect 2 flows, which works fine for most of them. I just add the decorators and some logging. But one of the data pipelines uses sqlite for intermediate storage, and this is the error message I receive:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140193261803264 and this is thread id 140192735962880.

I can also paste the part of the code where sqlite is called. I am also trying to guess the reason behind the error. There is an sqlite object created in a function. Within the function I define another function which uses this object. But both python functions are not prefect flows or tasks. They live within a bigger prefect flow. So here comes my first question: Does prefect 2 create a different thread for each nested function inside a flow or a task. Otherwise I cannot explain why the 2 parts of the code would run in different threads.

✅ 1

Sam Garvis

09/02/2022, 2:48 PM

I'm on Prefect 2.3.0

Copy code

prefect deployment build flowname.flowname -n flowname_dev --work-queue=dev-wq-1 -ib kubernetes-job/dev-k8s-job -sb gcs/dev --override image_pull_policy=Always

When I run this, the yaml created has this. Why does it still included the manifest.json? I thought that was deprecated

Copy code

### DO NOT EDIT BELOW THIS LINE
###
flow_name: Colorado Watcher
manifest_path: colorado_watcher-manifest.json

✅ 1

Josh Paulin

09/02/2022, 4:55 PM

I’m trying to understand why my flow keeps crashing partway thorugh, at a seemingly consistent spot. I have a parent flow that runs the first subflow fine, but half way through the second subflow the parent fails with the exception posted in the thread. The UI still shows the second subflow as running. This is on EKS by the way. Locally things look to run until completion.

✅ 1

Saksham Dixit

09/02/2022, 5:02 PM

My Slack webhook on the prefect UI fails but does not give any failure reason. I'm using prefect version `'1.2.2'`:

1️⃣ 1

✅ 1

Ela

09/02/2022, 9:38 PM

hi (: I noticed that my flow runs twice when I import tasks or functions from files placed in the same parent directory as the file which keeps the Flow object. I also get a UserWarning that my flow conflicts with 'another flow' and that the tasks of the flow conflict with 'some other tasks', although each task and flow have a unique name (warning example: "_A task named ... and defined at ... conflicts with another task. Consider specifying a unique

name

parameter in the task definition_"). I tried to see what's happening in the prefect.tasks.py file (which sends the warning) and I made it print out all the names in the PrefectObjectRegistry.get(). I noticed how it printed out the names of the tasks I'm using multiple times All of this disappeared when I imported tasks/functions from a different directory than where the flow is; the flow runs only once and the warnings are gone - I'm using Prefect 2.3.1 (also when I used version 1 these double runs happened until I changed the location of the func/tasks I was importing) I hope this makes sense, if you have any advice on why this happens, please let me know! thanks^^

✅ 1

datamongus

09/02/2022, 9:47 PM

Is there an known issue with prefect cloud, we are noticing several flows showing up as late schedules

✅ 1

Alix Cook

09/02/2022, 10:23 PM

I think I may have seen this asked somewhere before, but can no longer find the discussion, so apologies if this is duplicative, but.... I'm wondering if there's a way to know if a flow run running in a process (i.e. Process infra type) has been killed. I'm currently seeing these flow runs stay in status "running" when this happens, and im writing a cleanup that will reschedule these runs, but I'm having trouble identifying which flow runs have actually had their processes terminated

✅ 1

Matt Fysh

09/02/2022, 11:42 PM

Hi all, is there a way to disable flow/task decorators for testing?

✅ 1

Araik Grigoryan

09/03/2022, 1:15 AM

Hi. I would like to use Prefect 2.0 to schedule/run Java programs. These programs, as well as the shell scripts that invoke them, produce stdout/stderr messages. What is the best practice for capturing stdout/stderr from shell and non-Python programs such that they appear in the Orion UI?

✅ 1

Ghislain Picard

09/03/2022, 5:24 PM

I've a function that leak memory and I can't solve it because this happens in a Fortran library. I would like to run the task (or subflow) in a subprocess. Is there an easy way to do that in prefect ? Or maybe another solution to this problem.

Yaron Levi

09/03/2022, 6:17 PM

Hi 👋 We are getting the: Block document has schema checksum sha256:xxxxx… which does not match the schema checksum for class ‘S3’

✅ 1

Yaron Levi

09/03/2022, 6:17 PM

✅ 1

Yaron Levi

09/03/2022, 6:17 PM