Eden
11/14/2022, 7:30 AMpy any_flow.py
successfully and insert data into BQ]
The problem I encounter is that failed to run a flow after the deployment.
Make it easier. I, then, follow the deployment, HERE_FROM_NATE, to create a very simple job.
After clicking running a flow for the deployment foo
, it shows error about connection issue.
07:17:34.363 | DEBUG | prefect.client - Connecting to API at │
│ <http://127.0.0.1:4200/api/> │
│ 07:17:35.873 | ERROR | prefect.engine - Engine execution of flow run │
│ 'd533fb3e-d96f-4774-ad53-bac47071f3c5' exited with unexpected exception
What I can think about is that the agent runs but it should not be <http://127.0.0.1:4200/api/>
for the agents to do the connection. But,
1. how could I set this? FQDN? public endpoint?
2. In where? Deployment?
Appreciate any help here, thx 🙇♂️🏻Steven Wilber
11/14/2022, 10:10 AMhttpx.HTTPStatusError: Client error '401 Unauthorized' for url '<http://localhost:8000/api/v1/health/>'
But I can check that url and it works fine and returns:
{"available":true}
Any help is much appreciated. Thanks.Stéphan Taljaard
11/14/2022, 1:28 PMMiremad Aghili
11/14/2022, 4:15 PMChristian Juhl
11/14/2022, 4:31 PMfrom prefect import task, flow
numbers = {
'a': 1,
'b': 2,
'c': 3,
'd': 4
}
@task
def square_number(number):
return number ** 2
@flow
def my_flow():
squared_numbers = square_number.with_options(tags=numbers.keys()).map(number=numbers.values())
return squared_numbers
if __name__ == '__main__':
output = my_flow()
Vishnu Duggirala
11/14/2022, 4:36 PMBlake Stefansen
11/14/2022, 5:22 PMflow_run
names are more than 63 characters
, our agent fails to create a deployment run using k8 jobs because it can't create a job label
Will post agent log in threadAmey Desai
11/14/2022, 6:09 PMalex
11/14/2022, 7:34 PMlogger = logging.getLogger("mylogs")
<http://logger.info|logger.info>("msg")
+
PREFECT__LOGGING__EXTRA_LOGGERS="['mylogs']"
Logs from my main thread are logged but not from any additional threads I am using.Madison Schott
11/14/2022, 9:51 PMSTORAGE = Docker(registry_url='<http://ecr.us-west-2.amazonaws.com/|ecr.us-west-2.amazonaws.com/>',
image_name='prefect-flows',
dockerfile='dbt_snowflake/DockerFile')
RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'prefect-prod'},
env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
execution_role_arn='xx',
labels=['ecs-agent', 'prod', 'winc'])
Heather DeHaven
11/14/2022, 10:07 PMDeepanshu Aggarwal
11/15/2022, 6:12 AMShruti Hande
11/15/2022, 6:55 AM11:13:27.904 | ERROR | Flow run 'pink-pogona' - Crash detected! Request to <prefect cloud url >/task_runs/ failed: Traceback (most recent call last):
File "/home/<my_username>/<my_venv_name>/lib/python3.8/site-packages/anyio/streams/tls.py", line 108, in _call_sslobject_method
result = func(*args)
File "/usr/lib/python3.8/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
ssl.SSLSyscallError: Some I/O error occurred (_ssl.c:1131)
#prefect-community #prefect-cloudAndreas Nigg
11/15/2022, 6:57 AMAndreas Nigg
11/15/2022, 6:59 AMVadym Dytyniak
11/15/2022, 8:53 AMmax
11/15/2022, 10:32 AMprefect work-queue set-concurrency-limit
?Simon Macklin
11/15/2022, 11:25 AMFuETL
11/15/2022, 3:17 PMBlake Hamm
11/15/2022, 3:26 PMredsquare
11/15/2022, 3:48 PMKalise Richmond
11/15/2022, 4:47 PMvholmer
11/15/2022, 5:00 PMCrash detected! Request to <https://api.prefect.cloud/api/accounts/CENSORED/workspaces/CENSORED/flow_runs/CENSORED/set_state> failed: Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 33, in read
return await self._stream.receive(max_bytes=max_bytes)
File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
data = await self.transport_stream.receive()
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
await self._protocol.read_event.wait()
File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
await fut
asyncio.exceptions.CancelledError
The URL points to the flow's own id. What's going on here? Did anyone else experience this?Luca Schneider
11/15/2022, 5:21 PMSowmiya Anand
11/15/2022, 5:48 PMMadison Schott
11/15/2022, 5:55 PMfrom prefect_fivetran.connectors import fivetran_sync_flow
ImportError: cannot import name 'fivetran_sync_flow' from 'prefect_fivetran.connectors' (/usr/local/lib/python3.9/site-packages/prefect_fivetran/connectors.py)
Amey Desai
11/15/2022, 6:40 PMSam Garvis
11/15/2022, 6:50 PMwhile true:
in the flow, but that has always shut down in the past after a few hours on 2.0Nathaniel Russell
11/15/2022, 8:04 PMIf this isn't clear:
flow run A is running
flow run A has list of info on flow runs B -> G
flow run A picks out flow run D
I want flow run A to wait until flow run D is done before continuing, how do I do this step?
jrose
11/15/2022, 8:33 PMSegmentation fault (core dumped)
. I have run this using different version of pandas (1.5.1 and 1.3.5) and still get the same error. I am able to import pandas and run a command such as pd.show_versions()
without issues but whenever I try to run anything that would create a dataframe I get the segfault error.
Including my currently installed versions and the flow I am trying to run below. Thanks in advance!jrose
11/15/2022, 8:33 PMSegmentation fault (core dumped)
. I have run this using different version of pandas (1.5.1 and 1.3.5) and still get the same error. I am able to import pandas and run a command such as pd.show_versions()
without issues but whenever I try to run anything that would create a dataframe I get the segfault error.
Including my currently installed versions and the flow I am trying to run below. Thanks in advance!INSTALLED VERSIONS
------------------
commit : 66e3805b8cabe977f40c05259cc3fcf7ead5687d
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.14.294-220.533.amzn2.x86_64
Version : #1 SMP Thu Sep 29 01:01:23 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.3.5
numpy : 1.23.4
pytz : 2022.6
dateutil : 2.8.2
pip : 20.0.2
setuptools : 45.2.0
Cython : None
pytest : 7.2.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : 2.9.5 (dt dec pq3 ext lo64)
jinja2 : 3.1.2
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2022.11.0
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.0
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlwt : None
numba : None
import os
import prefect
import sys
import pandas as pd
from prefect import task, Flow
from prefect.storage import S3
from prefect.run_configs import ECSRun
from prefect.run_configs import LocalRun
# create logger
logger = prefect.utilities.logging.get_logger()
@task
def say_hello():
print(pd.show_versions())
data = {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']}
df = pd.DataFrame.from_dict(data)
<http://logger.info|logger.info>(df.head())
<http://logger.info|logger.info>("Hello, Cloud!")
flow = Flow("jrose_flow", tasks=[say_hello])
kwargs = {}
kwargs["cluster"] = f"arn:aws:ecs:us-west-2:XXXXXXXXXXX:cluster/prefect-agent-dev"
flow.run_config = ECSRun(task_role_arn="arn:aws:iam::XXXXXXXXXXX:role/prefect-dev-rpt-services-role",
execution_role_arn="arn:aws:iam::XXXXXXXXXXX:role/prefect-dev-rpt-services-role",
task_definition_arn="arn:aws:ecs:us-west-2:XXXXXXXXXXX:task-definition/rn-rpt-services-dev",
run_task_kwargs = kwargs)
flow.storage = S3(bucket="bucket_name", key="prefecttest/rpt/jrose_flow.py",
stored_as_script=False)
Bianca Hoch
11/17/2022, 10:08 PMSegmentation fault
error arises when your system is trying to access memory that it does not have access to, or memory that doesn't exist.jrose
11/17/2022, 10:16 PMBianca Hoch
11/17/2022, 10:18 PMjrose
11/18/2022, 2:09 PM