Stephen Lloyd
04/06/2022, 7:44 AMThomas Mignon
04/06/2022, 8:22 AMNicolaas Negron
04/06/2022, 12:00 PMEwerton Henrique Marschalk
04/06/2022, 12:42 PMАндрій Демиденко
04/06/2022, 1:18 PMChristoph Mayer
04/06/2022, 1:56 PMHaseeb Ahmad
04/06/2022, 3:01 PM{
"flow_name": "",
"flow_id": "",
"task_name": "",
"task_id": "",
"message": "",
"customer_id": ""
}
The questions we have are:
1. Is using PREFECT__LOGGING__FORMAT
the correct place to specify the JSON format?
2. How can we handle this additional attribute customer_id
. Our intent here is for most tasks that are related to our customers to do something like <http://logger.info|logger.info>('Something', customer_id)
3. To the previous point, how can we extend this further such that we can add arbitrary parameters or objects to the log, <http://logger.info|logger.info>('Something', my_object,my_second_object)
and have them show up nsted in the json?Noosymer
04/06/2022, 3:01 PMChris Reuter
04/06/2022, 3:35 PMFina Silva-Santisteban
04/06/2022, 4:14 PMs3_list_task = S3List()
flow.set_dependencies(
task=s3_list_task,
keyword_tasks=dict(
bucket='my-bucket-name'
)
)
And this is the error message:
TypeError: run() missing 1 required positional argument: 'prefix'
Does this mean I can’t use buckets without prefixes together with the S3 prefect tasks?Nicolaas Negron
04/06/2022, 5:10 PMMatthew Seligson
04/06/2022, 5:19 PMEwerton Henrique Marschalk
04/06/2022, 6:02 PMFuETL
04/06/2022, 6:48 PMChris Reuter
04/06/2022, 7:00 PMMatt Yamkowy
04/06/2022, 8:06 PMRajan Subramanian
04/06/2022, 8:09 PMprefect deployment ls
but i want to get all my created deployments...1000 of them and run them all. so im assuming i have to programatically execute prefect deployment run deploymenet_name for each deployment in my deploymentsHenrietta Salonen
04/06/2022, 9:24 PMprod
project suddenly fail with Failed to load and execute flow run: ModuleNotFoundError("No module named 'project_name'")
however, all the flows with exactly the same source code in Prefect dev
project run just fine. When I pull these images and start containers locally the project structure and packages are the same for both dev and prod. What could be the issue here?Shiyu Gan
04/06/2022, 9:42 PMShiyu Gan
04/06/2022, 9:49 PMparameter_defaults
for Clocks, but they seem to be static w.r.t each clock.
For example, I want the parameter be "day of the week",i.e. Mon, Tue....Brad
04/06/2022, 10:18 PMShiyu Gan
04/06/2022, 10:25 PMJason Motley
04/06/2022, 11:12 PMmain
section of a standard python script?Shiyu Gan
04/07/2022, 2:11 AMAlan Ning
04/07/2022, 2:28 AMMoss Ebeling
04/07/2022, 6:08 AMRicardo Gaspar
04/07/2022, 12:11 PM2.0.b2
.
do you know where the resource_manager
is located now? Before I was importing from prefect import resource_manager
https://docs.prefect.io/core/idioms/resource-manager.html
CC @Anna Geller @Kevin KhoRobert Holmes
04/07/2022, 1:38 PMdef prefect_flow():
with Flow('cloud_reporting_etl') as flow:
for column, value in jobs_df.iterrows():
job_name = value['Job Name']
query = value['Query']
filename = value['Filename']
extracted = extract_data(f"""{query}""")
load_data_to_s3(extracted, filename)
return flow
Bruno Nunes
04/07/2022, 2:39 PMRicardo Gaspar
04/07/2022, 3:45 PM2.0b2
and starting the server (on an EC2 machine) with an IP on a different port. (self-hosted).
I’m able to connect to it via browser (I’ve set all the security groups and routes to allow me to connect to it on a browser).
However it seems that internally Prefect Orion is still interacting with the default host and port. So the UI doesn’t render all the info (see screenshot in the thread)
Is there something extra I need to change?