Ben Collier
07/05/2021, 10:08 AMEvgenii
07/05/2021, 3:55 PMBen Muller
07/05/2021, 10:37 PMwiretrack
07/05/2021, 11:04 PMkubernetes agent
keep getting [Errno -2] Name or service not known
. Tried every variation for apollo: `prefect-apollo-service`(my svc name), <http://prefect-apollo-service:4200/graphql>
, the https
address, can’t make it to work. any ideas on what I should be doing to get the agent to talk to apollo? If I remove the http
I get a different error: 1 No connection adapters were found for 'prefect-apollo-service:4200/graphql'
Laura Vaida
07/06/2021, 7:48 AM@task(log_stdout=True)
def write_order_data (dataframe):
current_date = dt.today().strftime("%Y_%m_%d")
#GCS_Result = GCSResult(bucket='uwg-mail', location = 'orders_import_sf' + '_' + current_date + '.csv')
dataframe.to_csv('<gs://uwg-mail/orders_import_sf.csv>', dataframe, header=True)
Gabriel Santos
07/06/2021, 4:20 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/app/'
Did i do something wrong?Joseph Loss
07/06/2021, 5:34 PMMadison Schott
07/06/2021, 7:53 PMfatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file
wiretrack
07/06/2021, 8:25 PMSuchindra
07/06/2021, 8:26 PMZach Schumacher
07/06/2021, 8:27 PMDaskLocalExecutor
isn’t the default executor, if dask is a dep of prefect anyways.Samuel Kohlleffel
07/06/2021, 9:07 PMBlobStorageUpload
is there anyway to set overwrite=True
so the files in the container are overwritten with the new uploaded files when my flow runs?Peyton Murray
07/06/2021, 11:45 PMimport prefect as pf
from prefect.engine.results import LocalResult
@pf.task(checkpoint=True, result=LocalResult(dir=path_to_result)
def my_task(a, b, c):
return do_stuff(a, b, c)
with pf.Flow('my flow') as flow:
my_task(1, 2, 'foo') # <--- I want to be able to specify path_to_result here
flow.run()
What's the right way to structure this to specify path_to_result
at the indicated location?matta
07/07/2021, 12:54 AMFileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.prefect/auth.toml'
Brad I
07/07/2021, 1:12 AMfailed to authenticate, missing token
. It works if I set the key in both variables, is this expected?
env:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
value: XXXXXXXX
- name: PREFECT__CLOUD__API
value: <https://api.prefect.io>
- name: PREFECT__BACKEND
value: cloud
- name: PREFECT__CLOUD__API_KEY
value: XXXXXXXX
- name: PREFECT__CLOUD__TENANT_ID
value: TTTTTTTT
image: prefecthq/prefect:0.15.0-python3.7
Ben Muller
07/07/2021, 2:58 AMAustralia/Sydney
time?Sean True
07/07/2021, 8:31 AMKrzysztof Nawara
07/07/2021, 10:52 AMBen Collier
07/07/2021, 10:58 AMnib
07/07/2021, 12:14 PMnew_state.result
) and error details. But it’s empty and I can get only “Some reference tasks failed.” from new_state
. Is it possible to extract this kind of details?Madison Schott
07/07/2021, 2:30 PMFiveTranSyncTask()?
Also is it ok if I just have the dbt_task
defined before this with the parameters needed?
user_profile_w_campaign = Flow("User Profile with Campaign")
user_profile_w_campaign.set_dependencies(
task=dbt_task,
upstream_tasks=[FivetranSyncTask()]
)
user_profile_w_campaign.run()
Mike Wochner
07/07/2021, 3:36 PMwith Flow('Example') as flow:
today_date = datetime.date.today().strftime("%Y-%m-%d")
data = extract_data(security_list, today_date)
load_data(data)
...
more_data = extract_more_data(security_list)
load_more_data(more_data, today_date)
Amit
07/07/2021, 5:12 PMAmit
07/07/2021, 5:17 PMale
07/07/2021, 5:33 PMMadison Schott
07/07/2021, 6:09 PMwiretrack
07/07/2021, 8:40 PMflow
), I was wondering if the large amount of rows on other tables will start to get in the way of the frontend performance (and hasura’s, and apollo’s) . Putting state
in mongodb or something should completely solve the challenge (not really sure if it’s really a challenge), but it seems that this would be a huge change, since the code is really well tied together. I was wondering how do you guys see scalability on the server, and curious on what approaches the cloud version uses to overcome potencial scalability issues in the long term.Aric Huang
07/07/2021, 9:55 PMFailed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
error running the flow, when using the job_template_path
option for KubernetesRun. I can successfully register the flow, and when running the flow it seems to be respecting the template.yml
I passed in (I see my Kubernetes cluster running an appropriate pod based on my template) - but after pulling the image I get the FileNotFoundError
. Thoughts on what is going on? My flow basically looks like this:
with Flow("Test") as test_flow:
...
test_flow.run_config = KubernetesRun(
job_template_path="template.yml"
)
test_flow.storage = GitHub(
repo="<path>",
path="flows/test_flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
Joseph Loss
07/07/2021, 10:02 PMFlow run 478adfa1-5f4f-4dec-a121-a2443bc0a253 has a `run_config` of type `LocalRun`, only `DockerRun` is supported
I had registered the flow and previously used the flow on LocalRun, now all of a sudden it's failing but it was working a few hours ago?