https://prefect.io
Join Slack
is there anyway to set the `max_retries` on a task from the UI (or without having to upload new code...
k

Kathryn Klarich

about 4 years ago
is there anyway to set the
max_retries
on a task from the UI (or without having to upload new code)? I was hoping I could template the value and pass it to the @task decorator, but it doesn't look like you can use templates with the
max_retries
argument
k
k
  • 2
  • 8
  • 145
Hey all, I'm having some issues where flow runs will hang indefinitely. There seem to be two issues:...
b

Brian Oldak

9 months ago
Hey all, I'm having some issues where flow runs will hang indefinitely. There seem to be two issues: 1. Root cause of why sometimes a flow will just run indefinitely and stop logging after some amount of time 2. Flow not timing out even though I've configured the flow with
@flow(timeout_seconds=16200)
for a 4.5 hour timeout. I'm mostly trying to solve the second one here. I see that some flows do actually hit this timeout and others will run for 9 hours and still not hit it. In the documentation I see a note "Flow execution may continue until the next task is called", and I'm wondering if maybe this could be causing my issue? I'm not actually using tasks per se, but I do notice that in the flows that do not time out there are no logs for hours vs the ones that do time out, there are logs right up until the timeout. Ironically that's the opposite of the behavior I want, which would be timing out iff there is no activity for a long time, but that's neither here nor there. Any help would be much appreciated!
b
b
  • 2
  • 12
  • 144
What’s the best practice for Data Retention Policy on Prefect deployment runs? Just as a reference,...
o

Ofir

over 2 years ago
What’s the best practice for Data Retention Policy on Prefect deployment runs? Just as a reference, here is how it is implemented for Apache Airflow, as yet another garbage collector DAG: https://stackoverflow.com/questions/66580751/configure-logging-retention-policy-for-apache-airflow I’m sure that Prefect has either a built-in mechanism for that, or encourages a common idiom for rotating / archiving / deleting artifacts from old runs. Context: We have a persistent storage on Azure Blob Storage (the S3 equivalent) where we store artifacts (e.g. output files and images) from a Machine Learning (Kedro) run. The space can pile up pretty quickly across runs and we would run out of storage, rendering our Prefect deployments not operational. What kind of policies are recommended to evict data from old runs? I don’t want to run out of space and I want the Prefect pipelines to remain operational. I know that some of you would say: “_It depends_”, so for the sake of this example let’s imagine that I have a dedicated 256GB of storage. Should I set a threshold (e.g. 70% of full) that will be as a trigger for evicting (removing) artifacts from old runs? Also, when should this run? as the first (prerequisite) subflow in my bigger flow, or as yet another deployment in Prefect on a recurring schedule? Thanks!
✅ 1
o
n
  • 2
  • 3
  • 144
Something I am struggling to get my head around is security in regards to `FargateCluster`. It seems...
e

Eddie Atkinson

over 3 years ago
Something I am struggling to get my head around is security in regards to
FargateCluster
. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass
"fargate_private_ip": True
to
cluster_kwargs
my cluster fails to start with the error:
Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s
That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic through
e
k
a
  • 3
  • 15
  • 144
Hello people, anybody know how to display progress bars from a running task on the prefect URL UI in...
j

Jeremiah Lethoba

almost 4 years ago
Hello people, anybody know how to display progress bars from a running task on the prefect URL UI in logs
j
a
  • 2
  • 2
  • 144
Hey folks! Does anyone have an example repo they could share that's handling the deployment of a Pre...
c

ciaran

over 4 years ago
Hey folks! Does anyone have an example repo they could share that's handling the deployment of a Prefect Agent with AKS? Currently starting off with spinning up a cluster with Terraform, but my k8s skills are sub-par so some useful starters would be handy. For reference i've CDK'd up a Prefect Agent & respective cluster in ECS on AWS before, but that wasn't involving k8s.
c
t
k
  • 3
  • 248
  • 144
<@ULVA73B9P> how do I set the number of concurrent flows for a deployment? if I don't specify one, i...
c

chris

10 months ago
@Marvin how do I set the number of concurrent flows for a deployment? if I don't specify one, is the default 1?
c
m
  • 2
  • 1
  • 143
Hi <@ULVA73B9P> I want to run a flow from another flow, I am able to do it But the catch is, my m...
k

Karthikeya

12 months ago
Hi @Marvin I want to run a flow from another flow, I am able to do it But the catch is, my main flow should not wait until the child flow finishes, it needs to show the status as completed/ success/fail . After it triggers the child flow. I can't use automations option as I need to trigger child only in few conditions of the parameters passed from deployment. How can this be achieved any guidance please
k
m
  • 2
  • 5
  • 143
Hi all, just getting started with Prefect and I'm stuck on error: ValueError: No Prefect API URL pro...
c

CJ

12 months ago
Hi all, just getting started with Prefect and I'm stuck on error: ValueError: No Prefect API URL provided. Please set PREFECT_API_URL to the address of a running Prefect server. prefect version = 3.1.1 I've confirmed the server is running at http://127.0.0.1:4200 I've updated pyproject.toml with
#  pyproject.tom

[tool.prefect]
logging.level = "DEBUG"
api.url = "<http://127.0.0.1:4200/api>"
Any guidance is appreciated
c
n
  • 2
  • 15
  • 143
<@ULVA73B9P> Hello Can you help on how we can connect to MS SQL Server Database..
v

Venkat Reddy

about 1 year ago
@Marvin Hello Can you help on how we can connect to MS SQL Server Database..
v
m
  • 2
  • 6
  • 143
Previous414243Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by