I'm currently working on using the prefect API through an iPython environment (Jupyter Notebook atm). Because my environment is heavily containerized, I'm looking at using the Webhook Storage, since that'll let me reuse a bunch of existing infrastructure. Am I correct in my reading of the source that if I want to store the flow as a script (i.e. not have it pickled), I would need to write a file out temporarily to local storage, if the flow is some python functions defined in a cell?
f
flavienbwk
04/29/2021, 6:12 PM
Hi Joseph, you might want to use an S3 storage to save your flow code. You can get an example here.
👍 1
j
Joe
04/29/2021, 6:17 PM
Unfortunately I don't have any S3 storage since I'm running on a physical host in my datacenter, and not a cloud environment
I probably could, but I'd like to avoid adding even more different stuff to the overall system. using the existing stuff is a lot better.
Joe
04/29/2021, 6:40 PM
Yeah, there's a setting for it, but, here's an excerpt from webhook:
Copy code
for flow_name, flow in self._flows.items():
<http://self.logger.info|self.logger.info>("Uploading flow '{}'".format(flow_name))
data = flow_to_bytes_pickle(flow)
if self.stored_as_script:
# these checks are here in build() instead of the constructor
# so that serialization and deserialization of flows doesnot fail
if not self.flow_script_path:
msg = "flow_script_path must be provided if stored_as_script=True"
self.logger.critical(msg)
raise RuntimeError(msg)
if not os.path.isfile(self.flow_script_path):
msg = "file '{}' passed to flow_script_path does not exist".format(
self.flow_script_path
)
self.logger.critical(msg)
raise RuntimeError(msg)
with open(self.flow_script_path, "r") as f:
data = f.read().encode("utf-8")
req_function = self._method_to_function[self.build_request_http_method]
Joe
04/29/2021, 6:40 PM
which looks like I need a preexisting local file to use to do that.
k
Kevin Kho
04/29/2021, 6:42 PM
I see what you mean. Let me look into it a bit more.
Kevin Kho
04/29/2021, 6:44 PM
Yeah I’m honestly not seeing other way. I will double check with the team though.
Kevin Kho
04/29/2021, 7:07 PM
Confirmed that you need to create a file.
j
Joe
04/29/2021, 7:23 PM
Thanks, now to figure out if there's cell magic for this.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.