Faisal k k
02/24/2022, 10:24 AMEdvard Kristiansen
02/24/2022, 10:50 AM[unix_http_server]
file=/tmp/supervisor.sock
[supervisord]
loglevel=debug
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
[program:local-agent]
command=prefect agent local start -l local
[program:docker-agent]
command=prefect agent docker start -l docker
The main issue I am trying to solve is keeping the agent up and running at all times. My VM does a scheduled restart once a day which means any agents I spin up manually will be shut down. Is there any way of setting the agent itself up as a docker container which starts on startup like the prefect server itself? It seems like the docker agent itself isnt actually running in docker but rather just issues the flow to its own docker container?Donnchadh McAuliffe
02/24/2022, 10:58 AM2.0a9
to 2.0a12
. When I try run the server prefect orion start
it throws this error:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table flow already exists
how do I get around this?Tomer Cagan
02/24/2022, 1:18 PMjspeis
02/24/2022, 1:36 PMwith Flow("FlowB") as flow:
task_c = do_task_c()
# ...
with Flow("FlowA") as flow:
task_a = do_task_a()
task_b = do_task_b(upstream_tasks=[task_a])
# ...
create_flow_run(flow_name="FlowB")
though just want to confirm whether if I get create_flow_run to wait for task b to complete? (what would be the best way to arrange this?)Florian Kühnlenz
02/24/2022, 2:45 PMScott Aefsky
02/24/2022, 2:53 PMshijas km
02/24/2022, 3:11 PMZach Schumacher
02/24/2022, 4:27 PMwith flow:
start_date, end_date = Parameter("start-date", default=None), Parameter("end-date", default=None)
start_date, end_date = set_date_params(start_date, end_date)
matches = get_completed_matches_by_date_range(start_date, end_date)
there_are_matches = are_there_any_matches(matches)
with case(there_are_matches, True):
some_data = some_task.run(matches)
Gabriel Milan
02/24/2022, 5:02 PMFlow
or State
objects? I'm trying to get it using the on_failure
callback for my flowDonnchadh McAuliffe
02/24/2022, 5:12 PMaaron
02/24/2022, 6:00 PMWilliam Grim
02/24/2022, 6:29 PMDocekrStorage
, and my Dockerfile
looks like the below. It gets through all the steps except the final step with an error. Does anyone know how to resolve this?Arcondo Dasilva
02/24/2022, 6:43 PMArcondo Dasilva
02/24/2022, 6:45 PM(from prefect.tasks.notifications import email)
seems not working in orion. am I wrong ?Kevin Kho
Milton
02/24/2022, 7:19 PMcustom.py
like this
import logging
import sys
def run():
logger = logging.getLogger("custom")
logger.setLevel('INFO')
log_stream = logging.StreamHandler(sys.stdout)
log_stream.setFormatter(LOG_FORMAT)
logger.addHandler(log_stream)
and also have
[logging]
# Extra loggers for Prefect log configuration
extra_loggers = "['custom']"
in the toml config, but logs from the custom logger don’t seem to be captured by Prefect and show up in the cloud console.William Grim
02/24/2022, 8:45 PMDockerStorage
and then realized I can't use a local
agent to run flows that use that. No problem... since my local agent is setup in a docker container, I figured I would just use a DockerRun
config. This seems to go alright at the very first, but then it can't pull the image from our ECR. 🧵David Wang
02/24/2022, 9:28 PMRUN_CONFIG = ECSRun(
labels=["dev"],
task_role_arn=f"arn:aws:iam::xxx:role/prefectTaskRole", # a role with S3 permissions
execution_role_arn="arn:aws:iam::xxx:role/prefectECSAgentTaskExecutionRole",
run_task_kwargs=dict(cluster="prefectEcsClusterDev"),
image="<http://xxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:docker-test-v1|xxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:docker-test-v1>"
)
Gabriel Milan
02/24/2022, 9:41 PMprefect
namespace and work fine. I needed to add one more agent to the cluster, but this one should be on another namespace, say prefect-agent-xxxx
. When I do this, I can successfully submit runs to it, they do get deployed, but it doesn't seem to actually run and no logs are shown. I've tried configuring the apollo URL to <http://prefect-apollo.prefect.svc.cluster.local:4200>
and also setting an ExternalName
to it in the prefect-agent-xxxx
namespace and using it, but none of them works. Any ideas on how I could debug this?Heeje Cho
02/25/2022, 12:34 AMR Zo
02/25/2022, 2:46 AMR Zo
02/25/2022, 2:47 AMR Zo
02/25/2022, 2:48 AMBen Muller
02/25/2022, 3:52 AMcreate_flow_run.run(
flow_name="advanced_cleaning",
project_name="modelling",
run_name=f"data_update={data_update}",
parameters=dict(data_update=data_update),
)
When I run this it is called multiple times, but only one sometimes two flows are triggered in the cloud, is this a possible bug?
Do run names need to be unique?
This is being called from within a task fyiTomer Cagan
02/25/2022, 8:57 AMAditi Tambi
02/25/2022, 10:22 AMdatamongus
02/25/2022, 12:47 PMDonnchadh McAuliffe
02/25/2022, 1:14 PMMatthias
02/25/2022, 1:34 PMdocker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/f30a3e9241fab0d272c3f36eae16867487fb187b964c8b3f22bc8fd05d2aa4d0/json>: Not Found ("No such container: f30a3e9241fab0d272c3f36eae16867487fb187b964c8b3f22bc8fd05d2aa4d0")
and the flow is stuck in submitted state. Does anyone knows how to fix it? Just to make sure, the only difference between the successful run and the run stuck in submitted state is the fact that I added a non-root user to the image.
https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows_no_build/docker_script_docker_run_local_image.py