Milou Van Cau
12/10/2022, 8:16 PM<http://DATA_FULL.to|DATA_FULL.to>_csv("csv_files/DATA_FULL.csv")
But when i run it without prefect it works fine.
does anyone has an idea of what i have to change ?Peyton Runyan
12/10/2022, 11:16 PMMilou Van Cau
12/11/2022, 12:44 AMAndrew Huang
12/11/2022, 2:20 AMfrom pathlib import Path
from prefect import flow
import pandas as pd
@flow
def test_flow():
df = pd.DataFrame({"a": [1, 2, 3]})
path = Path("csv_files/DATA_FULL.csv")
path.parent.mkdir(exist_ok=True)
df.to_csv(path)
test()
this too:
from pathlib import Path
from prefect import flow
import pandas as pd
@flow
def test():
df = pd.DataFrame({"a": [1, 2, 3]})
df.to_csv("DATA_FULL.csv")
test()
Jeff Hale
12/11/2022, 3:28 AMprefect version
from the command line, @Milou Van Cau?Milou Van Cau
12/11/2022, 10:40 AMVersion: 2.7.0
API version: 0.8.3
Python version: 3.10.6
Git commit: c2833339
Built: Thu, Dec 1, 2022 4:03 PM
OS/Arch: darwin/x86_64
Profile: default
Server type: hostedJeff Hale
12/11/2022, 1:52 PM<http://df.to|df.to>_csv("DATA_FULL.csv")
- so not in a subfolder - does that work?Milou Van Cau
12/11/2022, 1:53 PMJeff Hale
12/11/2022, 2:09 PMMilou Van Cau
12/11/2022, 2:10 PMfrom prefect import flow, get_run_logger
from main import update_funda
import numpy as np
@flow
def main_flow():
results = update_funda()
logger = get_run_logger()
<http://logger.info|logger.info>(f'THE RESULTS >> Fit time = {np.sum(results["fit_time"])} Test score = {np.mean(results["test_score"])}')
if __name__ == "__main__":
main_flow()
this function calls this one
def update_funda():
df = start_scraping_funda()
print(f"scraper {df.shape}")
df = indexer_funda(df)
print(f"indexer {df.shape} ✅ Saved NL_DATA_FULL")
df = cleaner_funda(df)
print(f"cleaner {df.shape} ✅ Saved NL_DATA_FULL_cleaned")
df = encoder_funda(df)
print(f"encoder {df.shape}")
print("Running model")
results = model_funda(df)
print("run checker")
df = checker_funda()
df.to_csv("csv_files/funda_csv/DATA_FULL.csv")
return results
the whole code works fine but at the end, the to_csv function is not executed and nothing is savedJeff Hale
12/11/2022, 2:48 PMWhen the main_flow is runned from VScode, everything is working fine.Are you running it from VScode like this:
python main_flow
?
And by “from the orion interface” - do you mean running a deployment from the UI?
I can’t reproduce with python my_file.py.
My attempt:
from prefect import flow
import pandas as pd
def do_it():
df = pd.DataFrame({"a": [1, 2, 3]})
return df
def make_csv():
df = do_it()
df.to_csv("sub/DATA_FULL.csv")
@flow
def test():
make_csv()
if __name__ == "__main__":
test()
Milou Van Cau
12/11/2022, 3:23 PMfrom pathlib import Path
from prefect import flow
import pandas as pd
@flow
def test():
df = pd.DataFrame({"a": [1, 2, 3]})
df.to_csv("DATA_FULL.csv")
if __name__ == "__main__":
test()
and the same result happened.Jeff Hale
12/11/2022, 5:21 PMMilou Van Cau
12/11/2022, 5:23 PMJeff Hale
12/11/2022, 5:40 PMMilou Van Cau
12/11/2022, 5:42 PMJeff Hale
12/11/2022, 5:43 PMMilou Van Cau
12/11/2022, 5:48 PMJeff Hale
12/11/2022, 6:06 PMdeployment run
from the cli it doesn't work, does it?Milou Van Cau
12/11/2022, 6:17 PM###
### A complete description of a Prefect Deployment for flow 'main-flow'
###
name: Funda
description: null
version: a7811d689d2cc02c955ccce13dad9b22
# The work queue that will handle this deployment's runs
work_queue_name: default
tags:
- emile
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
type: process
env: {}
labels: {}
name: null
command: null
stream_output: true
working_dir: null
block_type_slug: process
_block_type_slug: process
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: main-flow
manifest_path: null
storage: null
path: /home/jeroen/code/Jstach22/Seren-IT
entrypoint: flow.py:main_flow
parameter_openapi_schema:
title: Parameters
type: object
properties: {}
required: null
definitions: null
prefect deployment run
the script runs but the csv is still not saved indeedJeff Hale
12/11/2022, 6:31 PMMilou Van Cau
12/11/2022, 6:33 PMJeff Hale
12/12/2022, 3:35 PMMilou Van Cau
12/12/2022, 3:35 PMRyan Peden
12/12/2022, 3:48 PMfrom pathlib import Path
from prefect import flow
import pandas as pd
@flow
def test():
df = pd.DataFrame({"a": [1, 2, 3]})
df.to_csv("/home/jeroen/code/Jstach22/Seren-IT/DATA_FULL.csv")
if __name__ == "__main__":
test()
Milou Van Cau
12/12/2022, 5:00 PM