Tarek
06/14/2022, 10:39 AMSang Young Noh
06/14/2022, 1:06 PMToby Rahloff
06/14/2022, 1:17 PMAyush Bairagi
06/14/2022, 1:33 PMIlhom Hayot o'g'li
06/14/2022, 1:44 PMJoseph Mathes
06/14/2022, 2:25 PMPaco Ibañez
06/14/2022, 2:34 PM@flow
def records_analysis_flow():
...
future = ingest_csv(config)
future.wait()
future = analyze_records(ra_config)
future.wait()
I noticed somewhat inconsistent performance results. If I run the code in docker without prefect it takes around 154 seconds but when I run it in prefect it takes almost 900 seconds. However, if I explicitly set the task runner to sequential, then it takes ~ 180 seconds. Is my flow doing something wrong? Why does the concurrent task runner take so long? Thanks!John-Craig Borman
06/14/2022, 2:48 PMJason Damiani
06/14/2022, 4:40 PMAndreas
06/14/2022, 4:41 PMJehan Abduljabbar
06/14/2022, 5:26 PMRio McMahon
06/14/2022, 5:42 PMsklearn
or similar into it) then run prefect register --project "Project Name" -l label -p flow.py
from within the pipeline. Realistically I could probably use the container that I am running the flow from within but am curious if there is an easier way. Thanks.Tony Yun
06/14/2022, 6:26 PMTask 'RunNamespacedJob - DBT Run': Finished task run for task with final state: 'Queued'
Paco Ibañez
06/14/2022, 8:24 PMaaron
06/14/2022, 9:21 PMstored_as_script=True
). I’m attempting to use the ExecuteNotebook task and I’m getting the following only with debug logs on the Docker agent:
[2022-06-14 20:54:12+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'Test Docker Notebook Flow'
[2022-06-14 20:54:13+0000] INFO - prefect.CloudTaskRunner | Task 'notebook0': Starting task run...
Executing: 0%| | 0/2 [00:00<?, ?cell/s]Operation not permitted (src/thread.cpp:309)
This is erroring out in the container when trying to access the notebook but not making it back to the Prefect API to change task state (the task run just hangs until it times out with a No heartbeat detected from the remote task
error). Any ideas?Ahmed Ezzat
06/14/2022, 10:51 PM2.0b6
Jovan Sakovic
06/14/2022, 11:12 PMFaheem Khan
06/15/2022, 12:02 AMWalter Cavinaw
06/15/2022, 2:35 AMwonsun
06/15/2022, 7:03 AMconfig.toml
file be in the path where prefect is installed? There is only backend.toml
file in the path where my prefect is installed, are those two the same?
Also i can't read backend.toml
. I got the error message like below:
TomlDecodeError: Found invalid character in key name: ':'. Try quoting the key name. (line 1 column 2 char 1)
What's the backend.toml file?Jeff Kehler
06/15/2022, 7:38 AMapollo_url
under [server.ui]
in the .prefect/config.toml
file does not appear to be working. How do I change the host that the UI will call to access the API?Andreas
06/15/2022, 10:17 AMDaskTaskRunner
tasks can stay in a pending state forever if they have custom dependencies (using _`wait_for`)_ on tasks that have failed. The failed state of the upstream tasks do not cause the downstream tasks to fail.. This isn't the case when using the default ConcurrentTaskRunner
where the failed state correctly propagates downwards and causes the dependent tasks to fail by defaultSheila
06/15/2022, 10:31 AMMatt Delacour
06/15/2022, 10:45 AM[2022-06-15 11:08:52+0200] ERROR - prefect.S3 | Error uploading Flow to S3 bucket prefect-flows-proto: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.Error the error + snippet of code
frojo
06/15/2022, 10:52 AMwith Flow('slack-test',state_handlers=[post_to_slack]) as flow_slack:
ini_message="Some Init Message"
#
post_slack_init(ini_message)
#
count()
We get the initial flow message start (post_to_slack
notifies us in start/end/failure), but we don't get
the "ini_message" until the flow ends. Even more if count
task (wich also carries a state handler) fails we get the fail of the task in first place and then the post_slack_init
(just a post request no state handler on this task) message. Any Clues? Thanks in advance!Toby Rahloff
06/15/2022, 10:59 AMApostolos Papafragkakis
06/15/2022, 12:20 PMGintautas Jankus
06/15/2022, 1:05 PMAn error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation: User: is not authorized to perform: ecs:RegisterTaskDefinition on resource: * because no identity-based policy allows the ecs:RegisterTaskDefinition action
I have tried also to pass flow_a task_definition_arn to parent_flow, then it runs flow, but gets this error:`in create_flow_run ValueError: Received both flow_id
and flow_name
. Only one flow identifier can be passed.` Maybe someone can share the best practices how I could create flow that would run in order these two already registered flows ? Big thanks in advance ! 🙂Adam
06/15/2022, 1:54 PMflow.register
and getting an error from the requests
module, presumably when it’s trying to register the flow to prefect cloud. It’s getting a webpage in the response, with a lot of obfuscated js. Any ideas why this is happening? We’re using prefect 0.14 on python 3.10.4Matteo Fiorillo
06/15/2022, 2:33 PMMatteo Fiorillo
06/15/2022, 2:33 PMKevin Kho
06/15/2022, 2:34 PMMatteo Fiorillo
06/15/2022, 2:38 PMKevin Kho
06/15/2022, 2:39 PMMatteo Fiorillo
06/15/2022, 2:40 PMKevin Kho
06/15/2022, 2:41 PMMatteo Fiorillo
06/15/2022, 2:42 PMfrom prefect import Flow
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
airbyte_conn = AirbyteConnectionTask(
airbyte_server_host="XX.XX.XX.XX",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id="<uuid_of_the_connection>"
)
with Flow("material-scraper-data") as flow:
flow.add_task(airbyte_conn)
# Register the flow under the "airbyte" project
flow.register(project_name="laserhub")
Kevin Kho
06/15/2022, 2:44 PMMatteo Fiorillo
06/15/2022, 2:46 PMKevin Kho
06/15/2022, 2:46 PMMatteo Fiorillo
06/15/2022, 2:47 PMKevin Kho
06/15/2022, 2:57 PMMatteo Fiorillo
06/15/2022, 2:58 PMKevin Kho
06/21/2022, 1:35 PMMatteo Fiorillo
06/21/2022, 1:36 PMKevin Kho
06/21/2022, 1:37 PMMatteo Fiorillo
06/21/2022, 1:37 PMKevin Kho
06/21/2022, 1:51 PMMatteo Fiorillo
06/21/2022, 1:52 PMKevin Kho
06/21/2022, 1:56 PMMatteo Fiorillo
06/21/2022, 1:57 PMKevin Kho
06/22/2022, 1:52 PMBoggdan Barrientos
07/05/2022, 9:06 PMKevin Kho
07/05/2022, 9:30 PMMatteo Fiorillo
07/06/2022, 7:46 AMTask 'AirbyteConnectionTask': Exception encountered during task execution!
Traceback (most recent call last):
File "/home/m_fiorillo/miniconda3/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/home/m_fiorillo/miniconda3/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/home/m_fiorillo/miniconda3/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
return run_method(self, *args, **kwargs)
File "/home/m_fiorillo/miniconda3/lib/python3.9/site-packages/prefect/tasks/airbyte/airbyte.py", line 340, in run
connection_status = self._get_connection_status(
File "/home/m_fiorillo/miniconda3/lib/python3.9/site-packages/prefect/tasks/airbyte/airbyte.py", line 176, in _get_connection_status
self.logger.log(level=self.stream_output, msg=response.json())
AttributeError: 'AirbyteConnectionTask' object has no attribute 'stream_output'
Kevin Kho
07/07/2022, 4:34 PM