Job "late" for long time what can be the cause of:

No heartbeat detected from the remote task; retrying the run.This will be retry 1 of 2

I also have a flow that I started manually, but it does not start, even though I have no other flows running. in general, if I have a flow that I need to ensure is running at an exact time (no more than few seconds off) , is it good to use Prefect for this, or is it better to use cron job ?

Hey @YD, will answer this tomorrow

thanks

Hey, on the second issue, whenever a flow is scheduled and not running, it is because of labels 99% of the time. Check these docs . So flows can only be picked up with Agent with the same labels. The Agent abels need to be a superset of the Flow labels. What were the labels for your Flow and Agent when you tried this? Yes schedules will fire on time if there is an agent to pick up the Flow.

Hi @YD, regarding your question about heartbeats above, Prefect will start a heartbeat process along with a flow to tell the server that a flow is still running. If this process stops sending heartbeats, the server will assume that something has gone wrong (like an infrastructure failure) and will retry the run--this is intentional to prevent the flow from appearing as "running" in the UI forever. Sometimes though, the heartbeat process can be prematurely terminated by the system (for example, in the event of a memory issue), and if this happens consistently, please give us more information and we can see if something more is going on and try other solutions, such as disabling heartbeats completely.

the systems were AWS c6g.2xlarge (run some ML code) AWS small postgres I did noticed that the postgres was very high on resources usage. I installed postgres on the EC2, an it resources where not very highly utilized I will check the labels tonight The agent on the EC2 was communicating with cloud prefect

There will be no logs because nothing started. Check the flow labels here in the UI, and then check the labels of your agents and see if there is an agent capable of picking up that flow

those flows run before, without any labels

Prefect adds default labels for local storage and local agent. What storage and agent are you using?

I ran the agent without giving it a specific label, so it shows the IP address of my AWS machine

You can turn it off by doing

prefect agent local start --no-hostname-label

what I did was

prefect agent local start

Yep the local agent has a default label. Where are you storing the flow?

flows are AWS EC2 machine (this is where the code and where I run the

flow.register

from) on that machine I added the cloud Key to the .prefect/config.toml file is this what you are asking?

Ah flow I’m asking about the Storage class like this . The default if you don’t specify is to store it on the Local machine.

I just used the defaults, I did not specify and storage location

from prefect import task, Flow
from prefect.executors import LocalDaskExecutor
from prefect.schedules import Schedule
from prefect.schedules.clocks import CronClock
import pendulum

@task(...)
def my_task()

def main():
    start_date = pendulum.datetime(2019, 1, 1, tz="America/New_York")
    schedule = Schedule(clocks=[CronClock('2 9 * * mon-fri', start_date=start_date)])

    with Flow("my flow name", schedule=schedule, executor=LocalDaskExecutor()) as flow:
        my_task()

    flow.register(project_name='my project name')

if __name__ == "__main__":
    main()

Did you register this on the same VM did the agent?

yes.. I use a single EC2 for this process

The labels should have been the same by default. Do you see you agent in Prefect Cloud and do both the flow and agent have the same label?

both have the same label but I do not see in the agent, when I click on "more" any acknowledgement that this flow was submitted even though I do see other flows, that were submitted from the same machine in the same way

A bit confused. You don’t see your agent in the Agents screen, but other flows have been running successfully?

I do see the agent in the agent screen and both the agent and the flow have the same label some of the flows run without an issue while some just hangs, and do not run at all

And what are the labels of the ones that don’t hang? does the agent show logs that it picked it i[?

All have the same labels

So confused because we actually have people complaining that too many flows run on the agent causing it to crash sometimes so as long as the agent can pick up the flow it will without worrying about memory/cpu available

I have 4 flows

The LocalAgent is the one responsible for executing the Flow. It executes the Flow as a local process. What happens when you click Quick Run? Do you just see a new yellow bar?

yes... just the the yellow

can you click into the yellow and take a screen shot of the dashboard?

I am running on the EC2 the command

nohup prefect agent local start --no-hostname-label > ~/tmp/prefect_agent.log &

then deleting all flow and re-registering them

wait if you register them, they will get the default label (looking at your code above). so with the agent, i think you don’t want to do

--no-hostname-label

. Either way just note the labels for the flows

Ok I'll do

nohup prefect agent local start -l aws > ~/tmp/prefect_agent.log &

and in the code use

flow.register(project_name='my flow name', labels=['aws'])

yeah that looks good. just make sure also the labels show up the same in the UI

I have the same issue again with a production pipeline

This seems to show that the flows are not even being picked up by an agent. I think it would be a label issue where no agent is capable to picking up these flows?

Isn’t crontab

@reboot

to run something when the machine reboots? I guess this won’t restart the process if it just dies but the machine doesn’t start? We have this section in our docs about using

supervisor

here to always run the agent. Did that agent config work previously for you?

the machine was not rebooted I am not sure why it died, and also why I did not get an alert email

I wouldn’t know, but maybe you can try spinning up something with the config and then turning it down and seeing if you get an email?

I’ll try today to generate an alert by killing the agent… to make sure that my set up is ok