Hi folks, how you doing? I hope you're doing good :) I'm experimenting with both Prefect and Airflow. My POC is pretty simple: 1. Query production database (MySQL) and store results; 2. Create a

.csv

with query results; 3. GZIP the

.csv

and upload it to S3; 4. Dump the uploaded file content into a Redshift table (using

COPY

with the correct settings) So I have a few questions: • A noticed that there is a 😛refect: S3 task for uploading, but I guess I can't pass gzipped content to it, right? So I built a custom Task that handles the gzipping + uploading for me. Am I using the S3Upload Task wrong or for my purposes this is what I should be doing? • Is there a way to generate dynamic tasks/flows? Example: with Airflow I made both a DAG and a Task factory using a

.yml

file as reference. My tasks are pretty repetitive and I only need to change some fields (source table name, destination schema/table, schedule time and incremental key). How could I achieve this with Prefect? Could you point me some directions or a documentation link? Thank you very much and congratulations for the awesome work.

Hi @Gabriel Montañola, and Welcome! Gzipping content before uploading/downloading to/from S3 sounds like a great new feature for the S3 Tasks! I’ll open an issue for this 👍. In fact, if you were interested in contributing to Prefect, adding a

gzip

or

compress

parameter to the S3 tasks would be an excellent opportunity to get your hands dirty! In the meantime, that’s exactly what you should be doing (you found a task to use as a template but tweaked it to suit your needs 💯 )

Nice! I'm totally swamped right now, but I would really enjoy contributing. I'll try as soon as I can! And thanks for for helping me. I'm a one-man team right know and Airflow overburdens me with many responsibilities. I started this adventure Monday, before this I have never heard of Prefect. But the time I spent to run my first custom Flow was WAAAY faster than to run my first custom DAG. Once again, congrats for the pretty piece of engineering you folks put together.

Of course! Let us know how we can help =]

No problem, I'll experiment meanwhile.! And about the demo, I would be interested! But I won't lie and say that I can schedule this this week or the next. So I'll reach you (if you're OK with it).

Of course!

sales at prefect dot io

is the best email for demos, or jump into intercom on https://www.prefect.io whenever you’re ready

just an update: I managed to work on something using the

.map

on the tasks. Sorry for the blurring, I'm just testing stuff and did not use Secrets ):

Awesome!