e

    ep

    4 months ago
    Hello everyone. I have a setup with aws fargate that was working great until 2 days ago. For some reason since yesterday we are getting a weird error message and we haven't changed anything. In prefect the error says "A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed" In the agent we can see the following error: "Error: no such option: --token (Possible options --env --key) Last night, process on prefect tried to run 3 times and failed. He gets stuck on "submitted" So, today, I changed the schedule and after stuck for 19 minutes at "submitted" it was able to run. I have no idea what's going on, does someone understands what's going on?
    Anna Geller

    Anna Geller

    4 months ago
    Are you using
    latest
    docker image tag? it looks like your docker container is trying to pull the latest version using API keys, but your Server is running with an older Prefect version < 1.0 Can you share the
    prefect diagnostics
    ?
    does someone understands what's going on?
    I may have some ideas but not yet enough information to be sure about the root cause
    Are you using ECSAgent on Fargate or running Server on Fargate? Can you send an example flow code of the flow that caused some issues (I'm interested in storage and run config)?
    e

    ep

    4 months ago
    yes using the latest tag. Using ECSAgent on fargate, image: prefecthq/prefect:latest-python3.8
    Anna Geller

    Anna Geller

    4 months ago
    thanks, can you answer all questions?
    Are you on Prefect Cloud or Server?
    e

    ep

    4 months ago
    not sure about prefect diagnostics, how can I run that? Using Prefect cloud
    Anna Geller

    Anna Geller

    4 months ago
    you run it in your terminal where you installed prefect python package
    e

    ep

    4 months ago
    here