Apologies if this is a basic question -- we are considering using Prefect Cloud for our ETL tasks.
I cannot find specifically what data is sent from Prefect Server to Prefect Cloud -- is there a document outlining the data being sent and security standards / policies more generally?
I'd appreciate anyone who can point me in the right direction here!
n
nicholas
12/04/2020, 7:24 PM
Hi @Daniel Nussbaum - to clarify, are you specifically talking about Telemetry data sent from Prefect Server or what data is sent to Cloud when you register a flow from Prefect Core?
d
Daniel Nussbaum
12/04/2020, 7:26 PM
Both!
n
nicholas
12/04/2020, 7:27 PM
I can say definitively that no data that isn't about the structure of your flow is sent to Prefect that you don't explicitly log. When you register a flow with Prefect, the dependency structure and peripheries (storage, environment, schedules etc) are serialized, so that Prefect knows what you want to happen and where. However, the internals of task and data that you pass between tasks are never sent to Prefect.
d
Daniel Nussbaum
12/04/2020, 7:29 PM
Got it - so no actual data ever makes it to Prefect Cloud except for information that there is a flow scheduled (and metadata about that flow which you choose to send).
👍 1
n
nicholas
12/04/2020, 7:32 PM
That's correct! In addition, if you're running Prefect Server, even metadata is never sent to Prefect, those are completely separate applications. Telemetry (which can be turned off!) in Prefect Server consists of an anonymous ID and heartbeats that we use to understand how often Prefect Server is being deployed.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.