k

    Kieran

    1 year ago
    I am getting a health check error when registering a flow with Docker storage:
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 151, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
    ModuleNotFoundError: No module named 'flows.tasks'
    Registering the same flow locally is passing my check.
    from prefect.utilities.debug import is_serializable
    
    if is_serializable(flow): ...
    My PYTHONPATH is the root of the directory and when SSH'ing into the container I am able to import the "missing" module... Any pointers?
    Michael Adkins

    Michael Adkins

    1 year ago
    Can you try adding a
    if __name__ == "__main__":
    block around your flow register/storage building? Sometimes cloud pickle can be finicky about that.
    You could also try disabling healthchecks and testing your flow just to narrow down the issue but it's very likely to fail at runtime with the same error.
    k

    Kieran

    1 year ago
    Thanks @Michael Adkins, I will give that a go. I reverted some steps and the pickle problem was coming from me creating a task, like so:
    @task
    def some_function():
      return ...
    And importing a helper function (which wasn't prefaced with
    @task
    ). Moving my helpers inside this @task ovecame the pickle problem. But ideally, I would be able to share helper functions across tasks without having to make them their own respective task?
    Michael Adkins

    Michael Adkins

    1 year ago
    If you're unable to use helper functions then generally your module isn't being added to your image correctly. I'd recommend using
    pip install
    rather than extending your
    PYTHONPATH
    it's generally more reliable.