Hi all, I'm new to Prefect and trying to set up a flow with Azure Blob Storage. How can I convert the downloaded file to a pandas dataframe? I don't think I'm properly understanding and using the Class. Thanks!
k
Kevin Kho
07/26/2021, 3:45 PM
Hey @Jelle Vegter, I assume you’re calling
.run()
after line 12? I think this returns the contents as a String. You seem to be doing the right thing. Maybe you can log the content of
download
so we understand the format?
Do you get an error message?
j
Jelle Vegter
07/26/2021, 3:57 PM
Your comment made me realise I needed to specify a .run() on the class in the function. I was counting on the flow.run() to do that. This works now, thanks!
k
Kevin Kho
07/26/2021, 3:57 PM
Oh yeah. In order you use a task in other task, you just need to call the
.run()
method. The Flow block does this for you.
j
Jelle Vegter
07/26/2021, 6:11 PM
If I can bother you with another question, is it best practice to make DAGs explicit and more elaborate or keep the DAG simple and put multiple steps within the same task? Would this be fine for example?
k
Kevin Kho
07/26/2021, 6:16 PM
The level of task granularity is very flexible in Prefect and we support as much or as little observability as you need. In short, we don’t really prescribe something here, but do recommend you choose based the right level of observability needed.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.