Gabriel Montañola

    Gabriel Montañola

    2 years ago
    Hi folks, how you doing? I hope you're doing good 😃 I'm experimenting with both Prefect and Airflow. My POC is pretty simple: 1. Query production database (MySQL) and store results; 2. Create a
    .csv
    with query results; 3. GZIP the
    .csv
    and upload it to S3; 4. Dump the uploaded file content into a Redshift table (using
    COPY
    with the correct settings) So I have a few questions: • A noticed that there is a 😛refect: S3 task for uploading, but I guess I can't pass gzipped content to it, right? So I built a custom Task that handles the gzipping + uploading for me. Am I using the S3Upload Task wrong or for my purposes this is what I should be doing? Is there a way to generate dynamic tasks/flows? Example: with Airflow I made both a DAG and a Task factory using a
    .yml
    file as reference. My tasks are pretty repetitive and I only need to change some fields (source table name, destination schema/table, schedule time and incremental key). How could I achieve this with Prefect? Could you point me some directions or a documentation link? Thank you very much and congratulations for the awesome work.
    Dylan

    Dylan

    2 years ago
    Hi @Gabriel Montañola, and Welcome! Gzipping content before uploading/downloading to/from S3 sounds like a great new feature for the S3 Tasks! I’ll open an issue for this 👍. In fact, if you were interested in contributing to Prefect, adding a
    gzip
    or
    compress
    parameter to the S3 tasks would be an excellent opportunity to get your hands dirty! In the meantime, that’s exactly what you should be doing (you found a task to use as a template but tweaked it to suit your needs 💯 )
    I know some of our users generate flows using yaml and the like
    I’m looking for an example for you
    Here’s the issue I opened for your S3 question: https://github.com/PrefectHQ/prefect/issues/3259
    Gabriel Montañola

    Gabriel Montañola

    2 years ago
    Nice! I'm totally swamped right now, but I would really enjoy contributing. I'll try as soon as I can! And thanks for for helping me. I'm a one-man team right know and Airflow overburdens me with many responsibilities. I started this adventure Monday, before this I have never heard of Prefect. But the time I spent to run my first custom Flow was WAAAY faster than to run my first custom DAG. Once again, congrats for the pretty piece of engineering you folks put together.
    Dylan

    Dylan

    2 years ago
    Of course! Let us know how we can help =]
    Also happy to help schedule a demo for Prefect Cloud if you are interested!
    I can’t seem to find any specific docs on generating tasks/flows from yaml
    Gabriel Montañola

    Gabriel Montañola

    2 years ago
    No problem, I'll experiment meanwhile.! And about the demo, I would be interested! But I won't lie and say that I can schedule this this week or the next. So I'll reach you (if you're OK with it).
    Dylan

    Dylan

    2 years ago
    Of course!
    sales at prefect dot io
    is the best email for demos, or jump into intercom on https://www.prefect.io whenever you’re ready
    Gabriel Montañola

    Gabriel Montañola

    2 years ago
    just an update: I managed to work on something using the
    .map
    on the tasks. Sorry for the blurring, I'm just testing stuff and did not use Secrets ):
    And it worked alright.
    Dylan

    Dylan

    2 years ago
    Awesome!