Hi, I'am getting an unpickling error when I use Lo...
# ask-community
i
Hi, I'am getting an unpickling error when I use Logstash, details in thread
Below you can find my code when I use python logging handler to send some logs to my logstash server.
Copy code
def tcp_logger():
    file_logger = logging.getLogger()
    if not any(
        [
            isinstance(handler, TCPLogstashHandler)
            for handler in file_logger.handlers
        ]
    ):
        host = os.environ.get('LOGSTASH_IP')
        port = 5959
        handle = TCPLogstashHandler(host, port)
        file_logger.setLevel(<http://logging.INFO|logging.INFO>)
        file_logger.addHandler(handle)
    return file_logger
Task code:
Copy code
from src.pubmed.logger import tcp_logger as get_logger

@task
def prepare_destination():
    get_logger().info("starting prepare_destination")
    files_path = prefect.context.get("pubmed_jsonl_local_path")
    os.makedirs(files_path, exist_ok=True)
    files_path = prefect.context.get("pubmed_xmls_local_path")
    os.makedirs(files_path, exist_ok=True)
    get_logger().info("ending prepare_destination")
    return True
When I test my flow on a LocalExcecutor on my laptop and it logs successfully with no error But when I deploy my flow on my docker environment in the production server I get this error:
Failed to load and execute Flow's environment: FlowStorageError("An error occurred while unpickling the flow:\n  TypeError('expected str, bytes or os.PathLike object, not NoneType')")
k
What storage did you use for the Flow to put it in Docker?
i
in my docker I use:
Copy code
flow.storage = prefect.storage.Local()
k
Do you use a RunConfig?
i
yes
Copy code
flow.run_config = LocalRun(labels=['scicarta'])
k
So did you register the Flow in machine A and then try to run it in a different environment (Docker container)?
i
we run a docker that starts an agent and register the flow.
I haves tested commenting
Copy code
from src.pubmed.logger import tcp_logger as get_logger
and the lines where i use the
get_logger
and my flow runs normal
k
Ah I see. When you register, the default behavior is to serialize the Flow and put it in storage. So there is something happening with the serialization of that logger. I think it would just be easier if you stored the Flow as a script to avoid serialization. There is more info about it here
👍 1
d
Thanks @Kevin Kho
👍 1
i
Thanks @Kevin Kho