Clint M
09/01/2022, 2:53 PMSam Garvis
09/01/2022, 3:35 PMBlake Stefansen
09/01/2022, 4:41 PMAdam Brusselback
09/01/2022, 5:17 PMVenkat Ramakrishnan
09/01/2022, 5:31 PMdeployment = Deployment.build_from_flow(flow=print_pipeline_hour,\
name="Hourly Pipeline Deployment", version="1", tags=["Iris"],\
schedule={'rrule': 'FREQ=HOURLY;UNTIL=20220912T040000Z', 'timezone': "Asia/Kolkata"},
work_queue_name="hour-work-queue")
Henning Holgersen
09/01/2022, 5:52 PMKrishnan Chandra
09/01/2022, 6:34 PMDockerContainer
and KubernetesJob
create new containers/jobs per flow run, but what I’d ideally like to do is run a fleet of long-running workers that process from the queue. What’s the best infrastructure to choose in Prefect 2.0 to achieve this?Kevin Grismore
09/01/2022, 6:36 PMParwez Noori
09/01/2022, 6:43 PMkiran
09/01/2022, 9:56 PMConcurrent
or Sequential
? In one section, the docs say the default is Concurrent
but then in another section, they say “Make sure you use .submit()
to run your task with a task runner. Calling the task directly, without .submit()
, from within a flow will run the task sequentially instead of using a specified task runner.” which seems to imply that the default is actually Sequential
Blake Hamm
09/01/2022, 11:23 PMKubernetesJob
blocks can interact with (the new) DockerContainer
block. Is there a way to deploy with --ib
as a DockerContainer
as well as use the KubernetesJob
? Or is there any roadmap to connect the two together (like how the DockerContainer
block can access a DockerRegistry
block?
I imagine this isn't feasible, but generally, I really like the ability to pass in a manifest file to a KubernetesJob
block. Specifically, I'm using EKS on AWS Fargate and really like the ability to define the resources. On the other hand the new DockerContainer
block seems really handy to manage environments for specific flows. Right now I have one image on ECR with all the dependencies and it's much heavier than it needs to be.
From a CI/CD perspective it would be great to have an action creating DockerContainer
blocks based on the individual flows and another action creating KubernetesJob
blocks based on different manifest files. In an ideal world, I would love a way to deploy using a DockerContainer
block as the image inside the KubernetesJob
block.
A current (hacky) solution could be to loop through the flow-level docker files, register them to ECR based on their flow name and create all the necessary KubernetesJob
blocks for each individual flow. I could use some kind of "resource" tag to pick the necessary manifest file. This would create a distinct KubernetesJob
block for every flow even though I might only be using 3 distinct manifest files (just lot's of unique containers for each flow). This would also require an --sb (unlike the new standalone DockerContainer
block.Venkat Ramakrishnan
09/02/2022, 4:27 AMAndreas Nord
09/02/2022, 10:04 AMADD flows /opt/prefect/flows
Which seems to be consistent with the deployment
storage: null
path: /opt/prefect/flows
entrypoint: flows\healthcheck.py:healthcheck
It seems that the deployment worked, but I can't find this path (/opt/prefect/flows) locally. I'm on WindowsKlemen Strojan
09/02/2022, 12:04 PMRoss Teach
09/02/2022, 1:21 PMprefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/043b2649-9d07-4c5e-8225-521ba2275e68/workspaces/689b139b-a725-4c2b-b167-86a705b8789d/task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
José Duarte
09/02/2022, 1:47 PMSeth Goodman
09/02/2022, 2:25 PMkwmiebach
09/02/2022, 2:46 PMsqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140193261803264 and this is thread id 140192735962880.
I can also paste the part of the code where sqlite is called.
I am also trying to guess the reason behind the error. There is an sqlite object created in a function. Within the function I define another function which uses this object. But both python functions are not prefect flows or tasks. They live within a bigger prefect flow. So here comes my first question: Does prefect 2 create a different thread for each nested function inside a flow or a task. Otherwise I cannot explain why the 2 parts of the code would run in different threads.Sam Garvis
09/02/2022, 2:48 PMprefect deployment build flowname.flowname -n flowname_dev --work-queue=dev-wq-1 -ib kubernetes-job/dev-k8s-job -sb gcs/dev --override image_pull_policy=Always
When I run this, the yaml created has this. Why does it still included the manifest.json? I thought that was deprecated
### DO NOT EDIT BELOW THIS LINE
###
flow_name: Colorado Watcher
manifest_path: colorado_watcher-manifest.json
Josh Paulin
09/02/2022, 4:55 PMSaksham Dixit
09/02/2022, 5:02 PMEla
09/02/2022, 9:38 PMname
parameter in the task definition_"). I tried to see what's happening in the prefect.tasks.py file (which sends the warning) and I made it print out all the names in the PrefectObjectRegistry.get(). I noticed how it printed out the names of the tasks I'm using multiple times
All of this disappeared when I imported tasks/functions from a different directory than where the flow is; the flow runs only once and the warnings are gone - I'm using Prefect 2.3.1
(also when I used version 1 these double runs happened until I changed the location of the func/tasks I was importing)
I hope this makes sense, if you have any advice on why this happens, please let me know! thanks^^datamongus
09/02/2022, 9:47 PMAlix Cook
09/02/2022, 10:23 PMMatt Fysh
09/02/2022, 11:42 PMAraik Grigoryan
09/03/2022, 1:15 AMGhislain Picard
09/03/2022, 5:24 PMYaron Levi
09/03/2022, 6:17 PMYaron Levi
09/03/2022, 6:17 PMYaron Levi
09/03/2022, 6:17 PM