Geoffrey Keating
04/29/2022, 8:55 PMWalter Cavinaw
04/29/2022, 9:34 PMTim Wright
04/30/2022, 12:24 AM__init__
methodRaviraja Ganta
04/30/2022, 11:10 AMprefect.context.get("logger")
is coming in the UI. How to surface my custom logger method logs also into the UI?Raviraja Ganta
04/30/2022, 8:09 PMmain-project
|
src
|
| __init__.py
| module1.py
| module2.py
|
configs
|
| config.yaml
|
flows
|
| sample_flow.py
module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2.
• What is the best way to package the code?
• What is the way to run agent that uses GPU on EC2?Zach Schumacher
05/01/2022, 3:26 PMCarlos Paiva
05/01/2022, 5:49 PMCole Murray
05/01/2022, 9:52 PM• the centralized nature of the Airflow scheduler provides a single point of failure for the systemIn a typical master/worker deployment, you have a single-point-of-failure schedule (yes you can run it highly available with locking / other mechanisms) responsible for reading from a schedule DB, to invoke tasks into a queue to be processed by workers. Can someone clarify how Prefect server solves this issue? From the docs, it seems Prefect server is also a SPOF in this architecture. Based on code here: https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/scheduler.py#L21, we would not be able to run several instances of the server simultaneously, as there is no locking taking place against the DB, and would cause double execution
Eddie Atkinson
05/02/2022, 5:58 AMDaskExecutor
and then use a temporary cluster as described here: https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-local-cluster
Just want to make sure I’m not missing a more obvious solution before I move to a cluster.Efraim Globus
05/02/2022, 8:44 AMDockerRun
, only LocalRun
is supported ”
what does it mean?
with Flow("iris_batch", storage=Local(path="/app/batch/flow.py",stored_as_script=True),
run_config=DockerRun(image="test:latest")
) as batch_prediction_flow:
What i am missing?
Amy help will be appreciated:)
Thank you!Alexander van Eck
05/02/2022, 11:21 AMHoratiu Bota
05/02/2022, 11:35 AM<address>:4200/graphql
in the browser - i've followed the troubleshooting guide, but no successNaga Sravika Bodapati
05/02/2022, 11:36 AMRon Levi
05/02/2022, 11:52 AMBob Colner
05/02/2022, 12:24 PMstate_handlers
functionality in Orion/Prefect2.0. Is is on the roadmap? (or I’m I missing something..) thanksMatthew Seligson
05/02/2022, 12:41 PMMars
05/02/2022, 3:16 PMPREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN
variable in the k8s manifest created by prefect agent kubernetes install
? Do I have to use the --env
switch or can I provide the value by editing the generated deployment container’s env:
list directly?Daniel Tavares
05/02/2022, 4:28 PMSlackbot
05/02/2022, 4:42 PMChris Reuter
05/02/2022, 4:54 PMLeon Kozlowski
05/02/2022, 5:53 PMPoosh Adhikary
05/02/2022, 7:08 PMnum_workers
for LocalDaskExecutor based on the number of cores available on the container?Christian Nuss
05/02/2022, 7:39 PMChris Reuter
05/02/2022, 8:10 PMZach Schumacher
05/02/2022, 9:18 PMMatt Alhonte
05/02/2022, 9:56 PMECSRun
would I put Dask security args? Like tls_ca_file
and stuff?Anthony L
05/02/2022, 10:02 PMrandom
generates the same values between runs.
I'm using Prefect Cloud 1.0 and running a local agent (with Prefect 1.2.1 library installed).
Here's my flow:
import prefect
from prefect import task, Flow
from random import random
@task
def task1():
logger = prefect.context.get("logger")
random_val = random()
<http://logger.info|logger.info>(f"{random_val=}")
if random_val < 0.5:
raise Exception()
with Flow("random-bug-flow") as flow:
task1()
flow.register(project_name="ant-flows")
I manually scheduled a run of this flow in Prefect Cloud and I see the random value in the logs. However, when I schedule another run (and any run thereafter), it generates the same "random" value. Is this expected?Gaurav
05/02/2022, 10:52 PMArtem Vysotsky
05/03/2022, 2:12 AMCole Murray
05/03/2022, 5:19 AM