https://prefect.io logo
#prefect-community
Title
# prefect-community
g

Gabriel Montañola

09/03/2020, 7:08 PM
Hi folks, how you doing? I hope you're doing good :) I'm experimenting with both Prefect and Airflow. My POC is pretty simple: 1. Query production database (MySQL) and store results; 2. Create a
.csv
with query results; 3. GZIP the
.csv
and upload it to S3; 4. Dump the uploaded file content into a Redshift table (using
COPY
with the correct settings) So I have a few questions: • A noticed that there is a P S3 task for uploading, but I guess I can't pass gzipped content to it, right? So I built a custom Task that handles the gzipping + uploading for me. Am I using the S3Upload Task wrong or for my purposes this is what I should be doing?Is there a way to generate dynamic tasks/flows? Example: with Airflow I made both a DAG and a Task factory using a
.yml
file as reference. My tasks are pretty repetitive and I only need to change some fields (source table name, destination schema/table, schedule time and incremental key). How could I achieve this with Prefect? Could you point me some directions or a documentation link? Thank you very much and congratulations for the awesome work.
d

Dylan

09/03/2020, 8:26 PM
Hi @Gabriel Montañola, and Welcome! Gzipping content before uploading/downloading to/from S3 sounds like a great new feature for the S3 Tasks! I’ll open an issue for this 👍. In fact, if you were interested in contributing to Prefect, adding a
gzip
or
compress
parameter to the S3 tasks would be an excellent opportunity to get your hands dirty! In the meantime, that’s exactly what you should be doing (you found a task to use as a template but tweaked it to suit your needs 💯 )
I know some of our users generate flows using yaml and the like
I’m looking for an example for you
Here’s the issue I opened for your S3 question: https://github.com/PrefectHQ/prefect/issues/3259
1
g

Gabriel Montañola

09/03/2020, 8:39 PM
Nice! I'm totally swamped right now, but I would really enjoy contributing. I'll try as soon as I can! And thanks for for helping me. I'm a one-man team right know and Airflow overburdens me with many responsibilities. I started this adventure Monday, before this I have never heard of Prefect. But the time I spent to run my first custom Flow was WAAAY faster than to run my first custom DAG. Once again, congrats for the pretty piece of engineering you folks put together.
🚀 1
d

Dylan

09/03/2020, 8:39 PM
Of course! Let us know how we can help =]
Also happy to help schedule a demo for Prefect Cloud if you are interested!
I can’t seem to find any specific docs on generating tasks/flows from yaml
g

Gabriel Montañola

09/03/2020, 8:53 PM
No problem, I'll experiment meanwhile.! And about the demo, I would be interested! But I won't lie and say that I can schedule this this week or the next. So I'll reach you (if you're OK with it).
d

Dylan

09/03/2020, 9:02 PM
Of course!
sales at prefect dot io
is the best email for demos, or jump into intercom on https://www.prefect.io whenever you’re ready
1
g

Gabriel Montañola

09/03/2020, 10:04 PM
just an update: I managed to work on something using the
.map
on the tasks. Sorry for the blurring, I'm just testing stuff and did not use Secrets ):
And it worked alright.
d

Dylan

09/03/2020, 10:06 PM
Awesome!
💎 1