https://prefect.io
Join Slack
Hi! Curious to see if anyone has been able to use Singer and Prefect together?
# prefect-server
f
Hi! Curious to see if anyone has been able to use Singer and Prefect together?
w
I just started a project that is using singer and prefect
@Frances Sun I have some working examples of using Singer and Prefect if you're interested in chatting about it
f
Thanks Will! Love to chat more about your usage in a few weeks. We're trying to catch some deadlines this month due to year end, so holding off on any new integrations until 2021. Until then, what taps & targets have you used? I've only tested a klaviyo tap with a jsonl & csv target
w
currently we're using the Salesforce tap and a custom tap we developed, both going to a snowflake target
i've also used a redshift target in the past as well
f
Curious, I'm almost sure the taps return json data (guessing), what mappings were necessary to push that data into redshift/snowflake, if any?
w
nothing, the singer spec is able to handle the json output. The targets handle the json output and convert it as necessary
s
@Will Milner Hi Will. Would you be so kind as to pass along the aforementioned examples of Singer+Prefect? I have written several over past 2 years, and our pipeline has grown well past the point where CRON suffices for orchestration
Did you have any luck with this combination, @Frances Sun? Is the thought to make a generic 
tap task
? Or something that is specific to a given tap…or to just use either the python or bash tasks?
w
sorry for the late reply. Since all singer taps/targets can be invoked via bash, our approach is to install them into the flow as a docker image and invoke them via a Shell task
only odd part is we set up a virtual environment within the docker image which is a bit unorthodox in terms of using docker but is best practice with using singer. An example task invocation looks a bit like this
Copy code
task_shell = ShellTask()    
salesforce = task_shell(command="/salesforce_tap_env/bin/tap-salesforce --config /config/salesforce_config.json --properties /config/salesforce_props.json --state /config/salesforce_state.json |"
                                    " /snowflake_target_env/bin/target-snowflake --config /config/snowflake_salesforce_config.json",
                    task_args={"name": "salesforce"})
you could also do them as python tasks are you suggested, but I find it easier to call them via shell. All of our taps and targets are their own stand alone package that we deploy to our own internal pypi server that can then be installed into our docker images
I have ran into some dependency conflicts between prefect and some singer taps, so sometimes I need to install a specific dependency package, here is a snippet from our docker file for our flows, this is installing https://github.com/singer-io/tap-salesforce
Copy code
FROM python:3.7


COPY flows/config /config



RUN pip install requests===2.23.0. # prefect tries to install a different version of requests that is not compatible with tap-salesforce

# install singer tap-salesforce
RUN python -m venv salesforce_tap_env
RUN /bin/bash -c "source salesforce_tap_env/bin/activate && pip install tap-salesforce && deactivate"
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by