Ian Andres Etnyre Mercader

    Ian Andres Etnyre Mercader

    8 months ago
    Hi, I'am getting an unpickling error when I use Logstash, details in thread
    Below you can find my code when I use python logging handler to send some logs to my logstash server.
    def tcp_logger():
        file_logger = logging.getLogger()
        if not any(
            [
                isinstance(handler, TCPLogstashHandler)
                for handler in file_logger.handlers
            ]
        ):
            host = os.environ.get('LOGSTASH_IP')
            port = 5959
            handle = TCPLogstashHandler(host, port)
            file_logger.setLevel(<http://logging.INFO|logging.INFO>)
            file_logger.addHandler(handle)
        return file_logger
    Task code:
    from src.pubmed.logger import tcp_logger as get_logger
    
    @task
    def prepare_destination():
        get_logger().info("starting prepare_destination")
        files_path = prefect.context.get("pubmed_jsonl_local_path")
        os.makedirs(files_path, exist_ok=True)
        files_path = prefect.context.get("pubmed_xmls_local_path")
        os.makedirs(files_path, exist_ok=True)
        get_logger().info("ending prepare_destination")
        return True
    When I test my flow on a LocalExcecutor on my laptop and it logs successfully with no error But when I deploy my flow on my docker environment in the production server I get this error:
    Failed to load and execute Flow's environment: FlowStorageError("An error occurred while unpickling the flow:\n  TypeError('expected str, bytes or os.PathLike object, not NoneType')")
    Kevin Kho

    Kevin Kho

    8 months ago
    What storage did you use for the Flow to put it in Docker?
    Ian Andres Etnyre Mercader

    Ian Andres Etnyre Mercader

    8 months ago
    in my docker I use:
    flow.storage = prefect.storage.Local()
    Kevin Kho

    Kevin Kho

    8 months ago
    Do you use a RunConfig?
    Ian Andres Etnyre Mercader

    Ian Andres Etnyre Mercader

    8 months ago
    yes
    flow.run_config = LocalRun(labels=['scicarta'])
    Kevin Kho

    Kevin Kho

    8 months ago
    So did you register the Flow in machine A and then try to run it in a different environment (Docker container)?
    Ian Andres Etnyre Mercader

    Ian Andres Etnyre Mercader

    8 months ago
    we run a docker that starts an agent and register the flow.
    I haves tested commenting
    from src.pubmed.logger import tcp_logger as get_logger
    and the lines where i use the
    get_logger
    and my flow runs normal
    Kevin Kho

    Kevin Kho

    8 months ago
    Ah I see. When you register, the default behavior is to serialize the Flow and put it in storage. So there is something happening with the serialization of that logger. I think it would just be easier if you stored the Flow as a script to avoid serialization. There is more info about it here
    Daniel Kornhauser

    Daniel Kornhauser

    8 months ago
    Thanks @Kevin Kho
    Ian Andres Etnyre Mercader

    Ian Andres Etnyre Mercader

    8 months ago
    Thanks @Kevin Kho