Daniel Black

    Daniel Black

    1 year ago
    Hello. I am getting a weird error when my task runs on schedule. All the task is doing is executing a query in a postgres db. It is scheduled to run daily but fails because "the table does not exist". I know it exists and if I restart the same task from the Prefect Cloud UI it works just fine. Our deployment is using a fargate task to run a docker container. Any ideas as to why this error would occur?
    Jenny

    Jenny

    1 year ago
    Hi @Daniel Black - Can you give a bit more information about what your task and any other tasks that are running before it? Might also be helpful to know if you're doing anything special to spin up your postgres db?
    Daniel Black

    Daniel Black

    1 year ago
    there aren't any tasks running before it. It is just sending a query to execute in the db. I don't know if we are doing anything special to spin up postgres db but I have other tasks on a schedule with the same framework that execute just fine
    the thing that is really throwing me off is why would it fail when scheduled but run successfully when manually restarted? what is the difference between the runs?
    Jenny

    Jenny

    1 year ago
    I wouldn't expect there to be a difference between the runs (except that the restart is run later or after some other task has completed?) Does it consistently fail when scheduled and then consistently work when restarted? If you can share any logs or screenshot of the error message that might help too.
    Daniel Black

    Daniel Black

    1 year ago
    it behaves like this consistently. I can try scheduling it for a different time to see if that makes a difference (not sure why it would)
    here is the error I end up with
    Jenny

    Jenny

    1 year ago
    I don't see any obvious answer here. Looks like a psycopg programming error? but I can't see why it would work on restart. A few Qs to help debug: Are you running this on Cloud or Server? Does the same code work in a local run (e.g. using
    flow.run
    ) and does it work for a non-scheduled run (e.g. by using quick start or the UI run page)?
    Daniel Black

    Daniel Black

    1 year ago
    this is being run on Cloud. It works using local run and quick start. It's very bizzare
    Jenny

    Jenny

    1 year ago
    Thanks - I'm asking the wider team to see if anyone has any ideas for you!
    Daniel Black

    Daniel Black

    1 year ago
    thank you! I am just as perplexed as you are
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    This is odd. Is any part of your flow parametrized, or relying on values set via
    prefect.context
    ? I'm wondering if the flow run environments accidentally differ in some way.
    When it does run successfully, is it running on the same agent as when the run failed, or on a different agent?
    Daniel Black

    Daniel Black

    1 year ago
    it is not parameterized
    aha! I think you figured it out! I think the wrong agent is picking up the scheduled task but when its rerun the correct agent is running it successfully!
    thank you for your help @Jenny @Jim Crist-Harif
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Ah, glad to hear it!