Arthur Jacquemart
06/01/2022, 11:47 AMCamilo Fernandez
06/01/2022, 1:02 PMMarcin Grzybowski
06/01/2022, 2:00 PMMatthew Seligson
06/01/2022, 2:29 PMRobert Esteves
06/01/2022, 3:03 PMPatrick Tan
06/01/2022, 3:48 PMDekel R
06/01/2022, 4:07 PMUnexpected error: TypeError("cannot pickle '_thread.lock' object")
I used is_serializable before registering the flow and got True.
Any ideas on how to tackle this?
ThxFrederick Thomas
06/01/2022, 5:32 PMfrom prefect import Flow
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
with Flow("parent-flow") as flow:
# Create the child flow run, look up the flow by name
child_run_id = create_flow_run(flow_name="gotta-run-flow")
and got this:
ImportError: cannot import name 'create_flow_run' from 'prefect.tasks.prefect' (/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/__init__.py)
Could someone please assist? Thanks!Jai P
06/01/2022, 6:01 PMprefect 2.0
and pydantic
where it looks like extra attributes are being ignored? details in threadSushma Adari
06/01/2022, 6:39 PMNo heartbeat detected from the remote task; marking the run as failed.
Chris Reuter
06/01/2022, 6:50 PMPatrick Tan
06/01/2022, 8:05 PMwith Flow("LiveLots-ETL-Parent", storage=S3(bucket=config_bucket,
client_options={
"aws_access_key_id":aws_access_key_id,
"aws_secret_access_key":aws_secret_access_key,
"aws_session_token":aws_session_token},)) as f:
Local agent is running on container as ECS task. The ECS task is attached with permission to access all S3 bucket.
{
"Statement": [
{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::*"
},
{
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::wp-livelots-pt/*",
"arn:aws:s3:::wp-live-lots/*"
]
}
],
"Version": "2012-10-17"
}
Initially flow is running without issue. After one hour, I am getting
Failed to load and execute flow run: ClientError('An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.')
Looks like ECS task is using the credentials when I registered flow. Please helpFuETL
06/01/2022, 8:45 PMDylan
06/01/2022, 9:33 PMDK
06/01/2022, 9:41 PMRainer Schülke
06/02/2022, 9:59 AMEinar Ellingsen
06/02/2022, 12:23 PMmail_settings = {
"mail_title": "Title",
"mail_recipient": "<mailto:recipient.1@enova.no|recipient.1@enova.no>,recipient.2@enova.no",
"mail_sender": "<mailto:sender@enova.no|sender@enova.no>",
"mail_smtp_server": "<http://smtp-mail.outlook.com|smtp-mail.outlook.com>",
"mail_smtp_port": 587,
"mail_smtp_type": "STARTTLS",
}
with case(new_file, True):
email_sent = email(
subject=mail_settings["mail_title"],
email_to=mail_settings["mail_recipient"],
email_from=mail_settings["mail_sender"],
smtp_server=mail_settings["mail_smtp_server"],
smtp_port=mail_settings["mail_smtp_port"],
smtp_type=mail_settings["mail_smtp_type"],
msg_plain=email_body,
attachments=[most_recent_file["local_path"]],
)
Florian Guily
06/02/2022, 12:38 PMRio McMahon
06/02/2022, 3:47 PMConnor Skennerton
06/02/2022, 4:14 PMDharit Sura
06/02/2022, 4:22 PMJohn Mizerany
06/02/2022, 4:23 PMS3Upload
module in prefect is it possible to specify the content-type?Connor Skennerton
06/02/2022, 4:31 PMFailed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1479, in get_task_run_info
result = self.graphql(mutation) # type: Any
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]
Marcin Grzybowski
06/02/2022, 4:35 PMroot@3b776c4684ea:/prefect/jobs# python3 simple.py
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 17, in wait
await self._event.wait()
File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Jessica Smith
06/02/2022, 4:53 PMOctopus
06/02/2022, 6:22 PMJessica Smith
06/02/2022, 6:32 PM{
"errors": [
{
"path": [
"flow_run",
0,
"id"
],
"message": "Cannot return null for non-nullable field flow_run.id.",
"extensions": {
"code": "INTERNAL_SERVER_ERROR"
}
}
],
"data": null
}
I don't know how these values can be null if they...can't be null.Patrick Tan
06/02/2022, 6:33 PMMatt Alhonte
06/02/2022, 11:54 PMroot
?)wonsun
06/03/2022, 1:31 AMpath
must be provided to show where flow .py
file is stored.`
My flow code like this!
with Flow('flow_step1', storage=Local()) as flow:
engine = create_engine('<mysql+pymysql://wonsun.jeong>:~~~~~~~~~~~~~~~', echo=True)
connection = engine.connect()
before = check_db()
after = check_storage()
answer = coin_inspection(after, before)
with case(answer, False):
new_datas = action_if_false()
raw_extract = each_file_info.map(new_datas)
final = each_ecg_meta.map(raw_extract)
with case(answer, True):
action_if_true()
flow.storage = Local(stored_as_script=True)
flow.register(project_name='try')
flow.run()
# flow.visualize()
This python file name is 'flow_step1.py'. What's the problem??
Also, most weird thing is window(local) already run 'flow_step1' and create result files but prefect web UI can't detect this flow!
I'm so confused because i don't understand exactly Flow, Register, Agent, Server, Local, Path.. in Prefect. Help me please.. 🤯wonsun
06/03/2022, 1:31 AMpath
must be provided to show where flow .py
file is stored.`
My flow code like this!
with Flow('flow_step1', storage=Local()) as flow:
engine = create_engine('<mysql+pymysql://wonsun.jeong>:~~~~~~~~~~~~~~~', echo=True)
connection = engine.connect()
before = check_db()
after = check_storage()
answer = coin_inspection(after, before)
with case(answer, False):
new_datas = action_if_false()
raw_extract = each_file_info.map(new_datas)
final = each_ecg_meta.map(raw_extract)
with case(answer, True):
action_if_true()
flow.storage = Local(stored_as_script=True)
flow.register(project_name='try')
flow.run()
# flow.visualize()
This python file name is 'flow_step1.py'. What's the problem??
Also, most weird thing is window(local) already run 'flow_step1' and create result files but prefect web UI can't detect this flow!
I'm so confused because i don't understand exactly Flow, Register, Agent, Server, Local, Path.. in Prefect. Help me please.. 🤯Kevin Kho
06/03/2022, 1:50 AMstored_as_script=True
, you need to point to where the file lives with path
.
flow.storage=Local(path=...,stored_as_script=True
wonsun
06/03/2022, 1:59 AMflow.storage=Local(path='C:\\Users\\user\\.prefect',stored_as_script=True)
Right?Kevin Kho
06/03/2022, 2:11 AMwonsun
06/03/2022, 5:37 AM