https://prefect.io logo
Title
s

Skip Breidbach

08/06/2020, 11:47 PM
Getting a surprising-to-me error running a super-simple flow using the "Non-Docker Storage for Containerized Environments" recipe - something that was working in 12.x. If I start my agent with
--show-flow-logs
and try to run the registered flow via the UI. Agent picks up the job fine but reports:
[2020-08-06 23:43:16,058] ERROR - agent | Error while deploying flow: AttributeError("Can't pickle local object 'NpipeHTTPAdapter.__init__.<locals>.<lambda>'")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python38\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Python38\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
I stripped anything out of the flow that was unique to the container just in case that was the issue. It runs fine under the standard agent, but fails with the docker agent.
n

nicholas

08/07/2020, 1:39 PM
Hi @Skip Breidbach - not sure what might be going on here, can you provide a min reproducible example for us to take a look at?
s

Skip Breidbach

08/10/2020, 9:57 PM
@nicholas sure thing! Sorry for the delay. Here's a bare-bones example.
from prefect import Flow, task
from prefect.environments import LocalEnvironment
from prefect.environments.storage import S3, Docker


@task
def do_it():
    return 1

storage = Docker(
    image_name="simple-container",
    image_tag='latest',
    base_image='prefecthq/prefect:all_extras-0.13.1',
)

storage.build()

with Flow("simple",
          storage=S3(bucket="skip-sandbox", key="flows/simple"),
          environment=LocalEnvironment(metadata={"image": "simple-container"})) as flow:
    x = do_it()

flow.register(project_name="Skip's Sandbox")
And I start the agent with
start prefect agent start docker -l s3-flow-storage -l <machine name tag> --show-flow-logs
Edit* in my example above I was initially pulling
prefect:all_extras-0.13.0
- I ran the same sample against 13.1 just in case there was a change but got the same results.
n

nicholas

08/11/2020, 2:07 AM
Thanks @Skip Breidbach, I'm going to open an issue for the Core team to take a look at this behavior.
@Marvin open "Can't pickle local object when using Non-Docker Storage for Containerized Environments"
j

josh

08/11/2020, 11:30 AM
Hey @Skip Breidbach I posted a comment on that issue 🙂
💥 1