Daniel Ross
03/05/2022, 8:50 PMrequests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1911c36d90>: Failed to establish a new connection: [Errno 111] Connection refused'))
If I look at the task itself, I can see that the environment variable for PREFECT__CLOUD__API is set to http://127.0.0.1:4200. So this seems like the problem.
The host it's trying to connect to is clearly wrong (since the server itself is running on an EC2 instance). So I've adjusted my ~/.prefect/config.toml to look like this:
host_ip = "my.ip.goes.here"
host_port = "4200"
host = "http://${server.host_ip}"
port = "4200"
endpoint = "${server.host}:${server.port}"
[server.ui]
apollo_url = "<http://my.ip.goes.here:4200/graphql>"
[cloud]
api = "${${backend}.endpoint}"
endpoint = "<https://api.prefect.io>"
graphql = "${cloud.api}/graphql"
No luck.
So I added the PREFECT__CLOUD__API definition to my environment variables in the container definition. Still no luck. However, when I look at the task definition, I can see the correct (or at least intended) PREFECT__CLOUD__API environment variable there. But the variable in the task is still set to http://127.0.0.1:4200, and the problem persists!
I am pretty stuck on this, and hoping that someone here has a line of sight to the solution. (This all worked without much configuration previously ... which now seems weird.)
All help appreciated!Jacqueline Riley Garrahan
03/06/2022, 6:06 AMapiVersion: batch/v1
kind: Job
metadata:
name: prefect-job
labels: {}
spec:
template:
metadata:
labels: {}
spec:
containers:
- name: flow
image: prefecthq/prefect:latest
imagePullPolicy: IfNotPresent
...
volumeMounts:
- name: filesystem-dir
mountPath: /job-files
volumes:
- name: filesystem-dir
hostPath:
# directory location on host
path: /path/to/my/dir
# this field is optional
type: Directory
I'm unable to mount the path which I believe is a function of the volume not being available on the agent. Is there a straightforward way for me to mount a volume to the agent in the helm chart deployment?Khen Price
03/06/2022, 10:43 AMDo mapped tasks run concurrently? Or does concurrency only happen when a dask executor is used?
Thanks!Dekel R
03/06/2022, 8:43 PMcomparable_items_df['tag_rank'] = comparable_items_df.groupby(['id', 'tag_name']).cumcount()
And I get the following error (when running outside of a task context it works)
TypeError: unhashable type: 'ResultSet'
For now I’m running this locally on my mac but it will run eventually using Vertex AI.
I’ll appreciate any help since I’m currently stuck.
Thanks.Bihag Kashikar
03/07/2022, 12:55 AMSen
03/07/2022, 12:39 PMEmma Rizzi
03/07/2022, 1:08 PMJustin Martin
03/07/2022, 2:34 PM@task
decorator? Here is an extremely simplified version of what i'm doing (See thread).
Am I able to define the run_load_proc function as just a FUnctionTask within the flow without an explicit function? Also, not sure if defining the global SqlServerExecute is a total anti-pattern, let me know. Thanks for all of your help.Haseeb Ahmad
03/07/2022, 2:35 PMInspecting flow runs
from prefect.backend import FlowRunView
flow_run = FlowRunView.from_flow_run_id("4c0101af-c6bb-4b96-8661-63a5bbfb5596")
Can you please guide me on the best approach to achieve the above use case. Really appreciate your time and help.Sarah Floris
03/07/2022, 2:42 PMKevin Kho
03/07/2022, 3:36 PMDaniel Nilsen
03/07/2022, 4:26 PMflow.register(«myProject»)
. To connect to the server I change the server.endpoint
in config.toml to the correct url. This works. But I don’t want to change the config manualy like this. Is there a way to temporarily change it when registering? I have tried set_temporary_config({"server.endpoint": "123"})
but it does not seem to work 🤔Zhibin Dai
03/07/2022, 4:34 PMwith case
statements, or can i use just one that does something if a condition is true?Jason Motley
03/07/2022, 4:59 PM: No heartbeat detected from the remote task;
Paul Butler
03/07/2022, 5:01 PMPaul Butler
03/07/2022, 5:33 PMQuestion for the Community - I am trying to use Prefect to schedule and monitor a Dbt project/pipeline. I follow example and pass in dbt_kwargs for connecting to snowflake. My dbt project runs fine with dbt run or dbt compile command, but the DbtShellTask fails - yet does not provide any reason for error. Anywhere more detailed logging is recorded??
Sample tasks like the hello one included here, run OK. I'm using Studio Code to debug this code, but also try to run it in Python IDLE and get same error output I'm running this:
from prefect import task, Flow, Parameter
from prefect.tasks.shell import ShellTask
from prefect.tasks.dbt import DbtShellTask
@task(log_stdout=True)
def say_hi(name):
print("hello {}!".format(name))
with Flow(name="dbt_flow") as f:
name = Parameter('name')
say_hi(name)
task = DbtShellTask(
profile_name='default',
environment='dev',
dbt_kwargs={
'type': 'snowflake',
'threads': 4,
'account': 'mysnowflake.account',
'user': '<mailto:myemal@myco.com|myemal@myco.com>',
'authenticator': 'externalbrowser',
'role': 'ROLENAME',
'database': 'DBNAME',
'warehouse': 'ENGINEERING_XS',
'schema': 'DV_PROTO'
},
overwrite_profiles=False,
profiles_dir='C:\\Users\myDBTuser\.dbt'
)(command='dbt compile')
out = f.run(name='Paul')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2022-03-07 17:18:07+0000] ERROR - prefect.DbtShellTask | Command failed with exit code 1
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Starting task run...
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | hello Paul!
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Finished task run for task with final state: 'Success'
Adam Roderick
03/07/2022, 6:06 PM...venv/lib/python3.8/site-packages/prefect/client/client.py:848: UserWarning: A flow with the same name is already contained in storage; if you changed your Flow since the last build, you might experience unexpected issues and should re-create your storage object.
I don't understand why we see that, because, we create a new Docker storage object on every deployment
flow.storage = Docker(....)
Can anyone explain what this error means, or how to address it?kevin
03/07/2022, 7:40 PMflow_run.duration
but I can’t seem to find that queryable value in the flow_run
fields on the interactive API in Prefect Cloud. Could someone help me resolve this differential?Pedro Machado
03/07/2022, 8:51 PMJean-Michel Provencher
03/07/2022, 9:20 PMprefecthq/prefect:1.0.0
and when scanning the docker image with Snyk I’m getting up to 114 security issues, as well as 8 critical vulnerabilities.
Are you guys planning on fixing them in the base docker image?Dylan
03/07/2022, 9:40 PMDylan
03/07/2022, 9:41 PMKelly Huang
03/07/2022, 11:27 PMin get_flow
from github import UnknownObjectException
ModuleNotFoundError: No module named 'github'
I'm reading differing things about whether or not github storage can access package dependencies? Correct me if I'm wrong, but shouldn't it definitely be able to access my Pipfile and therefore dependencies? Otherwise what would the point of github storage be?Andrew Moist
03/08/2022, 9:37 AMTomer Cagan
03/08/2022, 9:38 AMVadym Dytyniak
03/08/2022, 9:51 AMcheckpointing
works in prefect 1.0.0? In documentation I see that it is enabled by default, but what is the default Result
?Bruno Murino
03/08/2022, 12:48 PMTypeError: StartFlowRun.run() missing 1 required positional argument: 'self'
. I don’t know if I’m doing something wrong, but it’s a bit unclear what to doTom
03/08/2022, 1:06 PMTom
03/08/2022, 1:57 PMflow.run()
? (https://stackoverflow.com/questions/66660927/prefect-workflow-how-to-persist-data-of-previous-every-schedule-run)Thomas Opsomer
03/08/2022, 2:55 PMmanual_only
trigger), after being approved using the UI on prefect cloud, it takes a very long time before the flow is resumed (like several hours :/).
It's not the first time it happens, but currently it's happening on every flows.
Anyone having the same issue ? Is it an issue on prefect backend side or should we do something ?Thomas Opsomer
03/08/2022, 2:55 PMmanual_only
trigger), after being approved using the UI on prefect cloud, it takes a very long time before the flow is resumed (like several hours :/).
It's not the first time it happens, but currently it's happening on every flows.
Anyone having the same issue ? Is it an issue on prefect backend side or should we do something ?Kevin Kho
03/08/2022, 2:57 PMThomas Opsomer
03/08/2022, 3:06 PMKevin Kho
03/08/2022, 3:06 PMThomas Opsomer
03/08/2022, 3:06 PMKevin Kho
03/08/2022, 5:05 PMThomas Opsomer
03/08/2022, 5:09 PMKevin Kho
03/08/2022, 5:10 PMThomas Opsomer
03/11/2022, 4:49 PMKevin Kho
03/11/2022, 4:55 PMThomas Opsomer
03/11/2022, 4:59 PMKevin Kho
03/11/2022, 5:01 PMThomas Opsomer
03/11/2022, 5:16 PMKevin Kho
03/14/2022, 5:08 PMThomas Opsomer
03/14/2022, 5:32 PMKevin Kho
03/14/2022, 5:36 PM