Yaron Levi
09/18/2022, 4:12 AMAndrei Tulbure
09/18/2022, 10:21 AMjpuris
09/18/2022, 5:41 PMprefect deployment build flows/redshift_mv_refresh.py:redshift_mv_refresh --name JPMBP-O --queue JPMBP-O --tag "redshift mv-refresh test" --infra process
However I’m running into an issue where the deployment build
command tests the main flow and hence fails as it can not import some of the modules provided by a virtual env.
...
import redshift_connector
ModuleNotFoundError: No module named 'redshift_connector'
...
prefect.exceptions.ScriptError: Script at 'flows/redshift_mv_refresh.py' encountered an exception
An exception occurred.
my project’s directory
❯ tree . -L 2
.
├── flows
│ ├── __pycache__
│ ├── lib
│ └── redshift_mv_refresh.py
├── requirements.txt
└── venv
├── bin
├── include
├── lib
└── pyvenv.cfg
How can I tell deployment build
to look for python in
venv/bin/python
Thank you a bunch!
edit: I’m using Prefect 2 with Cloud.Vadym Dytyniak
09/19/2022, 8:25 AMDimosthenis Schizas
09/19/2022, 10:28 AMjpuris
09/19/2022, 11:50 AM['analytics.mv_1', 'analytics.mv_2']
in the Parameters form, the parameter is treated as a string 😞
...
13:31:26.728 | INFO | Flow run 'lush-bettong' - Scheduling materialized views {mat_view_names}
13:31:26.729 | INFO | Flow run 'lush-bettong' - Refreshing materialized view '['
13:31:26.735 | INFO | Flow run 'lush-bettong' - Refreshing materialized view '''
13:31:26.737 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'a'
13:31:26.739 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'n'
13:31:26.741 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'a'
13:31:26.745 | INFO | Flow run 'lush-bettong' - Refreshing materialized view 'l'
...
Vlad Tudor
09/19/2022, 12:22 PMECSAgent
? I don't quite manage to get it right. Thank you!
The Agent registration is successful, I start the Flow, in AWS the task is created but the TaskDefinition is INACTIVE and the Task itself is PENDINGGiorgio Basile
09/19/2022, 12:43 PMBal Raj
09/19/2022, 1:18 PMCarlos Cueto
09/19/2022, 1:29 PMWebHookAction
Automation set up that keeps generating false failure notifications on Cloud UI. The WebHook sends an HTTP POST to a specific endpoint and returns an HTTP 202 status code upon success, but Prefect seems to think it's an error. Every time the action triggers, I get a new failure notification. This literally spams our Cloud Notifications with false failures.
FYI this issue seems to be the exact same issue, but I was told it wasn't, and to open a new ticket (I still think it's the same issue): https://github.com/PrefectHQ/prefect/issues/5955
Screenshot of false failure notification:jpuris
09/19/2022, 1:39 PMAdrien Besnard
09/19/2022, 1:51 PMJustin Trautmann
09/19/2022, 2:02 PMAlix Cook
09/19/2022, 3:09 PMJosé Duarte
09/19/2022, 4:03 PMNick Coy
09/19/2022, 4:12 PMNic
09/19/2022, 4:30 PMTaylor Babin
09/19/2022, 4:49 PMSam Garvis
09/19/2022, 5:20 PMStream closed EOF for prod-prefect/analytic-shrimp5rkr9-j82lj (prefect-job)
for all of them.
Are they supposed to auto delete?jpuris
09/19/2022, 6:38 PMLuis Henrique
09/19/2022, 10:07 PMMichael Ng
09/19/2022, 10:17 PMKarim Soliman
09/19/2022, 11:45 PM16 September 2022,01:33:41 prefect.CloudTaskRunner INFO Task 'create_ingestion_params': Starting task run...
16 September 2022,01:33:41 prefect.CloudTaskRunner INFO Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
16 September 2022,01:33:41 prefect.CloudTaskRunner INFO Task 'create_flow_run': Starting task run...
16 September 2022,01:33:41 prefect.CloudTaskRunner INFO Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
16 September 2022,01:33:42 prefect.CloudTaskRunner INFO Task 'create_flow_run[0]': Starting task run...
16 September 2022,01:33:42 prefect.create_flow_run[0] INFO Creating flow run 'fierce-flounder-Tome Ingestion' for flow 'Tome Ingestion'...
16 September 2022,01:33:42 prefect.create_flow_run[0] INFO Created flow run 'fierce-flounder-Tome Ingestion': <https://cloud.prefect.io/tome/flow-run/8d5af013-baa8-4830-be75-7d212a4f3e74>
16 September 2022,01:33:42 prefect.CloudTaskRunner INFO Task 'create_flow_run[0]': Finished task run for task with final state: 'Success'
16 September 2022,01:33:43 prefect.CloudTaskRunner INFO Task 'create_flow_run[1]': Starting task run...
16 September 2022,01:33:43 prefect.create_flow_run[1] INFO Creating flow run 'fierce-flounder-Tome Ingestion' for flow 'Tome Ingestion'...
16 September 2022,01:33:43 prefect.create_flow_run[1] INFO Created flow run 'fierce-flounder-Tome Ingestion': <https://cloud.prefect.io/tome/flow-run/7729a5c3-5fb3-474f-bfbc-7d2fbce841d8>
16 September 2022,01:33:44 prefect.CloudTaskRunner INFO Task 'create_flow_run[1]': Finished task run for task with final state: 'Success'
log from my local python execution
[2022-09-19 23:44:08+0000] INFO - prefect.CloudTaskRunner | Task 'create_ingestion_params': Starting task run...
INFO [task_runner.py:241] Task 'create_ingestion_params': Starting task run...
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
INFO [task_runner.py:335] Task 'create_ingestion_params': Finished task run for task with final state: 'Success'
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_flow_run': Starting task run...
INFO [task_runner.py:241] Task 'create_flow_run': Starting task run...
[2022-09-19 23:44:14+0000] INFO - prefect.CloudTaskRunner | Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
INFO [task_runner.py:335] Task 'create_flow_run': Finished task run for task with final state: 'Mapped'
[2022-09-19 23:44:15+0000] INFO - prefect.CloudTaskRunner | Task 'get_task_run_result': Starting task run...
INFO [task_runner.py:241] Task 'get_task_run_result': Starting task run...
Stephen Lloyd
09/20/2022, 3:17 AMStephen Lloyd
09/20/2022, 3:18 AMChing
09/20/2022, 7:27 AMjpuris
09/20/2022, 8:26 AM10
but are plenty, when it is left to Unlimited?
When there are no visible scheduled jobs, my agent polling that queue would not retrieve any jobs either 😕 As soon as I change it to Unlimited
, it comes to life and starts running all the late jobs like nothing had happened.
I’m using Prefect Cloud v2Carlos Paiva
09/20/2022, 8:29 AMAndrei Tulbure
09/20/2022, 8:42 AMAndreas Nord
09/20/2022, 8:55 AM