Ray Bell
09/16/2022, 4:41 PMXavier Babu
09/16/2022, 5:01 PMStephen Herron
09/16/2022, 5:06 PMJosh Paulin
09/16/2022, 6:00 PMNikhil Jain
09/16/2022, 6:47 PMSubmitted for execution: Task arn:aws:ecs:us-west-2:<>:task/prod-prefect-cluster/c01a3ca13b3741a89e58b6c33dda0671
INFO prefect-server.Lazarus.FlowRun Rescheduled by a Lazarus process. This is attempt 1.
Any tips on how to debug this? I can’t find the logs for these tasks in cloudwatch.Philip MacMenamin
09/16/2022, 7:21 PMChris Gunderson
09/16/2022, 7:52 PMLeon Kozlowski
09/16/2022, 8:57 PMclient.create_flow_run
work off of flow group ids?
{
flow(
where: {name: {_eq: "FLOW_NAME"}, archived: {_eq:false}},
order_by: {created: desc}
) {
id
name
project {
name
}
}
}
Sergiy Popovych
09/16/2022, 9:27 PMasaf alina
09/17/2022, 1:21 PMmain
flow, which calls to a subflow go_eat
, each time with different parameters.
something of the sort:
@flow
def go_eat():
...
@flow
def main():
animals = ['dog', 'cat', 'monkey']
for animal in animals:
go_eat(animal)
I want to make those go_eat
flows to run in parallel, and I wonder how to implement this.
it would be best if I could make up to 2 animals to go_eat
in parallel, but no more than that.
I'm not sure if I somehow better use the prefect queue concurrency mechanism, or if I better implement this in pure python.
any suggestions?Yaron Levi
09/18/2022, 4:12 AMAndrei Tulbure
09/18/2022, 10:21 AMJP
09/18/2022, 5:41 PMprefect deployment build flows/redshift_mv_refresh.py:redshift_mv_refresh --name JPMBP-O --queue JPMBP-O --tag "redshift mv-refresh test" --infra process
However I’m running into an issue where the deployment build
command tests the main flow and hence fails as it can not import some of the modules provided by a virtual env.
...
import redshift_connector
ModuleNotFoundError: No module named 'redshift_connector'
...
prefect.exceptions.ScriptError: Script at 'flows/redshift_mv_refresh.py' encountered an exception
An exception occurred.
my project’s directory
❯ tree . -L 2
.
├── flows
│ ├── __pycache__
│ ├── lib
│ └── redshift_mv_refresh.py
├── requirements.txt
└── venv
├── bin
├── include
├── lib
└── pyvenv.cfg
How can I tell deployment build
to look for python in
venv/bin/python
Thank you a bunch!
edit: I’m using Prefect 2 with Cloud.Vadym Dytyniak
09/19/2022, 8:25 AMDimosthenis Schizas
09/19/2022, 10:28 AMJP
09/19/2022, 11:50 AM['analytics.mv_1', 'analytics.mv_2']
in the Parameters form, the parameter is treated as a string 😞
...
13:31:26.728 | INFO | Flow run 'lush-bettong' - Scheduling materialized views {mat_view_names}
13:31:26.729 | INFO | Flow run 'lush-bettong' - Refreshing materialized view '['
13:31:26.735 | INFO | Flow run 'lush-bettong' - Refreshing materialized view '''
13:31:26.737 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'a'
13:31:26.739 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'n'
13:31:26.741 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'a'
13:31:26.745 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'l'
...
Vlad Tudor
09/19/2022, 12:22 PMECSAgent
? I don't quite manage to get it right. Thank you!
The Agent registration is successful, I start the Flow, in AWS the task is created but the TaskDefinition is INACTIVE and the Task itself is PENDINGGiorgio Basile
09/19/2022, 12:43 PMBal Raj
09/19/2022, 1:18 PMCarlos Cueto
09/19/2022, 1:29 PMWebHookAction
Automation set up that keeps generating false failure notifications on Cloud UI. The WebHook sends an HTTP POST to a specific endpoint and returns an HTTP 202 status code upon success, but Prefect seems to think it's an error. Every time the action triggers, I get a new failure notification. This literally spams our Cloud Notifications with false failures.
FYI this issue seems to be the exact same issue, but I was told it wasn't, and to open a new ticket (I still think it's the same issue): https://github.com/PrefectHQ/prefect/issues/5955
Screenshot of false failure notification:JP
09/19/2022, 1:39 PMAdrien Besnard
09/19/2022, 1:51 PMJustin Trautmann
09/19/2022, 2:02 PMAlix Cook
09/19/2022, 3:09 PMJosé Duarte
09/19/2022, 4:03 PMNick Coy
09/19/2022, 4:12 PMNic
09/19/2022, 4:30 PMTaylor Babin
09/19/2022, 4:49 PMSam Garvis
09/19/2022, 5:20 PMStream closed EOF for prod-prefect/analytic-shrimp5rkr9-j82lj (prefect-job)
for all of them.
Are they supposed to auto delete?JP
09/19/2022, 6:38 PMJP
09/19/2022, 6:38 PMSam Garvis
09/19/2022, 6:51 PMJP
09/19/2022, 6:55 PMMason Menges
09/19/2022, 7:00 PMJP
09/19/2022, 7:01 PMMason Menges
09/19/2022, 7:02 PMJP
09/19/2022, 7:09 PMexpected_start_time
(A) from flow context and compare it with current time - 5 minutes
(B), if A < B, then have flow complete without running any tasks.
There are couple of issues with my idea and it feels fairly hacky 😞
What would be the recommended approach to achieve something like this?Mason Menges
09/20/2022, 3:41 PMJP
09/20/2022, 3:53 PMfrom prefect import flow, context
@flow(name="Hello Flow")
def hello_world():
ctx = context.get_run_context()
print('================')
print('----CONTEXT----')
print(ctx)
print('----CONTEXT, start_time ----')
print(ctx.get().start_time)
print('----CONTEXT, expected_start_time ----')
print(ctx.get().expected_start_time)
print('================')
if __name__ == '__main__':
hello_world()
I found this out solely by experimentation.
I suspect I’m querying the flow’s context incorrectly with ctx.get().expected_start_time
as it produces following error (full run in attachment)
...
AttributeError: 'FlowRunContext' object has no attribute 'expected_start_time'
...
If you have any docs on mechanics of Context object for prefect 2, please do point me in the correct direction 🙏Mason Menges
09/20/2022, 4:10 PMfrom prefect import context
test_context = context.FlowRunContext.get()
start_time = test_context.flow_run.expected_start_time
JP
09/20/2022, 4:14 PMflow_run=FlowRun(..., estimated_run_time=foo)
.
Thank you, thank you, thank you!Mason Menges
09/20/2022, 4:32 PM