With Orion, do you think it would be feasible to have a task that returns a pre-configured fsspec filesystem that is set up from Orion Storage?
We are using different storage types depending on the environment. I would like to be able to remove a lot of conditional code we have today with prefect 1.0, and it look like Orion with the Storage may be really helpful for that.
๐ 1
j
Jeremiah
03/22/2022, 12:49 AM
hey @davzucky, yes, Orion storage will be completely usable within your flow (including fsspec support). Give us a couple releases to hammer out the UX beyond automatic task result storage, but I think this is representative of a user story weโre very excited about. Cc-ing @Zanie for visibility.
๐ 1
d
davzucky
03/22/2022, 1:15 AM
@Jeremiah Thank you for the update. I will keep an eye on all future releases. I like the idea that you will expose that at the API/Task level
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.