I am new to Prefect but experienced with airflow I wrote a s

Question

I am new to Prefect but experienced with airflow I wrote a simple script but found my flow run multiple times automatically sometimes 3 other times 6 Can t find anything in the community please help from prefect import Parameter Flow task print task def get bbp data return bbp data task def get sem data return sem data task def align bbp sem data bbp sem return align data task def generate train data align return train data with Flow I5 ETL as flow bbp data = get bbp data sem data = get sem data align data = align bbp sem data bbp data sem data train data = generate train data align data flow visualize flow run

Dylan · Answer

Hi < John Song> We re here to help smile Can you send some of the logs that show your flow running multiple times Is this flow running with Prefect Cloud Server by any chance or just locally

John Song · Answer

itWorkspace gt python on 3 7 5 tags v3 7 5 5c02a39a0b Oct 15 2019 00 11 34 MSC v 1916 64 bit AMD64 on win32 help copyright credits or license for more information from i5 data workflow import pipeline 0 09 10 20 45 23 INFO prefect FlowRunner | Beginning Flow run for I5 ETL prefect FlowRunner Beginning Flow run for I5 ETL 0 09 10 20 45 23 INFO prefect FlowRunner | Starting flow run prefect FlowRunner Starting flow run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get bbp data Starting task run prefect TaskRunner Task get bbp data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get bbp data finished task run for task with final state Success prefect TaskRunner Task get bbp data finished task run for task with final state Success 0 09 10 20 45 23 INFO prefect TaskRunner | Task get sem data Starting task run prefect TaskRunner Task get sem data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get sem data finished task run for task with final state Success prefect TaskRunner Task get sem data finished task run for task with final state Success 0 09 10 20 45 23 INFO prefect TaskRunner | Task align bbp sem data Starting task run prefect TaskRunner Task align bbp sem data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task align bbp sem data finished task run for task with final state Success prefect TaskRunner Task align bbp sem data finished task run for task with final state Success 0 09 10 20 45 23 INFO prefect TaskRunner | Task generate train data Starting task run prefect TaskRunner Task generate train data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task generate train data finished task run for task with final state Success prefect TaskRunner Task generate train data finished task run for task with final state Success 0 09 10 20 45 23 INFO prefect FlowRunner | Flow run SUCCESS all reference tasks succeeded prefect FlowRunner Flow run SUCCESS all reference tasks succeeded 0 09 10 20 45 23 INFO prefect FlowRunner | Beginning Flow run for I5 ETL prefect FlowRunner Beginning Flow run for I5 ETL 0 09 10 20 45 23 INFO prefect FlowRunner | Starting flow run prefect FlowRunner Starting flow run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get bbp data Starting task run prefect TaskRunner Task get bbp data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get bbp data finished task run for task with final state Success prefect TaskRunner Task get bbp data finished task run for task with final state Success 0 09 10 20 45 23 INFO prefect TaskRunner | Task get sem data Starting task run prefect TaskRunner Task get sem data Starting task run 0 09 10 20 45 23 INFO prefect TaskRunner | Task get sem data finished task run for task with final state Success prefect TaskRunner Task get sem data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect TaskRunner | Task align bbp sem data Starting task run prefect TaskRunner Task align bbp sem data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task align bbp sem data finished task run for task with final state Success prefect TaskRunner Task align bbp sem data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect TaskRunner | Task generate train data Starting task run prefect TaskRunner Task generate train data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task generate train data finished task run for task with final state Success prefect TaskRunner Task generate train data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect FlowRunner | Flow run SUCCESS all reference tasks succeeded prefect FlowRunner Flow run SUCCESS all reference tasks succeeded 0 09 10 20 45 24 INFO prefect FlowRunner | Beginning Flow run for I5 ETL prefect FlowRunner Beginning Flow run for I5 ETL 0 09 10 20 45 24 INFO prefect FlowRunner | Starting flow run prefect FlowRunner Starting flow run 0 09 10 20 45 24 INFO prefect TaskRunner | Task get bbp data Starting task run prefect TaskRunner Task get bbp data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task get bbp data finished task run for task with final state Success prefect TaskRunner Task get bbp data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect TaskRunner | Task get sem data Starting task run prefect TaskRunner Task get sem data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task get sem data finished task run for task with final state Success prefect TaskRunner Task get sem data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect TaskRunner | Task align bbp sem data Starting task run prefect TaskRunner Task align bbp sem data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task align bbp sem data finished task run for task with final state Success prefect TaskRunner Task align bbp sem data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect TaskRunner | Task generate train data Starting task run prefect TaskRunner Task generate train data Starting task run 0 09 10 20 45 24 INFO prefect TaskRunner | Task generate train data finished task run for task with final state Success prefect TaskRunner Task generate train data finished task run for task with final state Success 0 09 10 20 45 24 INFO prefect FlowRunner | Flow run SUCCESS all reference tasks succeeded prefect FlowRunner Flow run SUCCESS all reference tasks succeeded

John Song · Answer

Here is the log it ran 3 times

Dylan · Answer

Are multiple visualizations produced as well

Dylan · Answer

can you run `prefect diagnostics` please

Dylan · Answer

I am unable to replicate the multiple runs problem

John Song · Answer

Yes there are 3 plots as well

John Song · Answer

let me run this on linux server to see if it s the same

John Song · Answer

same on linux server

Dylan · Answer

Would you mind running `prefect diagnositcs`

Dylan · Answer

This outputs some info that may help me assist you further

John Song · Answer

INFO prefect FlowRunner Flow run SUCCESS all reference tasks succeeded config overrides env vars flow information environment executor true labels false logger true metadata false on exit false on start false type LocalEnvironment result schedule storage task count 4 system information platform Windows 10 10 0 17134 SP0 prefect version 0 12 6 python version 3 7 5

John Song · Answer

print diagnostic info flow

John Song · Answer

added above line it printed out three times

John Song · Answer

def run flow with Flow I5 ETL as flow bbp data = get bbp data sem data = get sem data align data = align bbp sem data bbp data sem data train data = generate train data align data flow visualize flow run print diagnostic info flow

John Song · Answer

Now I put flow run logic inside one method then call it from outside

John Song · Answer

from i5 data workflow import pipeline pipeline run flow

John Song · Answer

the flow only ran once now but still printed 3 times

John Song · Answer

somehow this file is being parsed 3 times

John Song · Answer

for now I am good thanks for jumping in

Dylan · Answer

Anytime