Hi, I am migrating to Prefect 2, and I am facing some troubles there. What the job does is:
1- Downloads a jar from somewhere
2- Uploads it to databricks file system (DBFS) through Databricks Rest API 2.0
3- Use it on databricks job (trough
/job/update
API)
4- Run a job
The problem I am facing is on step 2, when I call
Add Block
Endpoint (through async method
execute_endpoint
), when I got a Time Out (no matter which timeout I set). Can you show me an sample of how can this get done with Prefect 2??? thanks in advance
s
Sahil Rangwala
04/06/2023, 5:30 PM
What type of block are you trying to add? The rest api allows you to create your own block type and block documents + schemas
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.