Hey guys, I know in the past you’ve mentioned that generators don’t necessarily play well with prefect since cloudpickle can’t serialize a generator. Have you taken a look at this project? I don’t know much of C or cloudpickle’s internals, so not sure if this could be used to solve a problem, but would leave ot hear your thoughts on it. https://github.com/llllllllll/cloudpickle-generators. Edit: There’s also an open issue for the cloudpickle project about integrating it, but they seem hesitant as it might break pypy and jython implementations (https://github.com/cloudpipe/cloudpickle/issues/146)

Hey Alex! I’ve seen this project before, and generally trust that it does what it claims to do. Given that it relies on somewhat subtle requirements for how Python is compiled / running, I’m very hesitant to recommend it except to anyone other than our most advanced users. We’ve already dealt with a lot of people who get confused by things like Python version mismatches between their local machine and their Flow’s docker image, or dependencies not matching on dask workers, so this feels like it could introduce more problems than it ultimately solves for our user base at this point in time.

Definitely agree that it would be more than an update to the requirements to get right. My guess based on the amount of support the repo has would involve you guys probably forking it and maintaining it separately to ensure it plays well with your product… which is definitely a nice chunk amount of work. Plus, C 😅

haha yea

Thanks for taking a look! Would eventually love to have generators integrated into the product!

yea for sure!