Hugo Kitano
07/20/2021, 11:14 PMprefect agent ecs start
with the parameter --run-task-kwargs config.yaml
where config.yaml contains the following:
networkConfiguration:
awsvpcConfiguration:
subnets:
- 'subnet-something'
assignPublicIp: 'ENABLED'
I would instead like to be able to input this as a command-line argument. Will there be any fixes down the road?Blake List
07/21/2021, 12:40 AMKarlygash Mukhitova
07/21/2021, 7:29 AMGoh Rui Zhi
07/21/2021, 8:57 AMmain()
of each job into a task. This runs fine, but the problem is that the logs for the functions outside of each the main()
method will not be logged using prefect.context
and hence not appear on the prefect UI. Is there a way for Prefect to capture all logs without explicitly changing the logger for each of the sub functions to use that of prefect.context
? I know that making each job into a subprocess and then redirecting the stdout into Prefect's logger will work, but would like to avoid using subprocesses in general. Besides, this method also seems very hackish haha.Dragan
07/21/2021, 10:03 AMDotan Asselmann
07/21/2021, 11:17 AMRanu Goldan
07/21/2021, 11:30 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 4b5a98d6-b0c0-4dc5-b540-7f00b17cea2d: provided a running state but associated flow run fc3bdb79-c506-4710-87c1-c039f29d727a is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 95, in call_runner_target_handlers
cache_for=self.task.cache_for,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1526, in set_task_run_state
version=version,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 4b5a98d6-b0c0-4dc5-b540-7f00b17cea2d: provided a running state but associated flow run fc3bdb79-c506-4710-87c1-c039f29d727a is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
It happens when task raised FAIL()
and retries. and the the task will be marked ClientFailed
state, then the flowrun is Cancelled
What I don't understand is why suddenly the error says flowrun not in a running state? How to resolve this?Brett Naul
07/21/2021, 1:11 PM<http://lr-ingest.io|lr-ingest.io>
, not sure if that's a symptom or a cause. any suggestions for how to further debug? I'm staving off an anti-Prefect mutiny from my team because of how painful it is for anyone to access the site 🏴☠️Oliver Götze
07/21/2021, 1:34 PMapollo_1 | PayloadTooLargeError: request entity too large
(full traceback in the thread)
When this error is thrown, no further log entries will be written for that particular flow run.
Logging continues however for other flow runs.
I've been able to mitigate this problem by decreasing the number of log messages, i.e. merging similar log messages into a single entry before sending them to logging.
Nevertheless I was wondering if there's a way to make the logging more resilient? Maybe Prefect Cloud performs better in such a scenario?Bruno Murino
07/21/2021, 1:34 PMPeter Peter
07/21/2021, 1:54 PM@task(log_stdout=True)
def print_msg():
print("Will it work?")
Sample task will print message to the log:
@task(log_stdout=True)
def print_msg():
print("Will it work? Yup!")
time.sleep(5)
Any help appreciated,
PRinze
07/21/2021, 2:56 PMconfig.toml
. I've tried many things already so I'd your input since I might be overlooking something.
To keep everything together, I prefer to have the config.toml
in the same git folder as my flows ( (base/src/config.toml
). In base/.env
there are a bunch of environment variables, among others PREFECT__USER_CONFIG_PATH
. Initially this worked on my local machine and the things defined in the config file were used in the flow, both on Server and when I switched to Cloud. But now I cloned the repo to a different machine and it fails to import the config.toml. Strangely enough, when I output the value of PREFECT__USER_CONFIG_PATH
to the log, it is correct. More info in the thread.
Is there a better way to do this, or am I forgetting something?Elliot Oram
07/21/2021, 3:53 PMmain()
in the attached code snippet.
When I execute this in a prefect flow I get the logging from main, but not from the thread. Any ideas? Thanks in advance 🙏Krzysztof Nawara
07/21/2021, 4:49 PMJustin Liu
07/21/2021, 6:39 PMSean Talia
07/21/2021, 7:10 PMCreate
access on Flow > Run
and Flow > Logs
, but the person whom I gave this role was still unable to actually create a flow run from the prefect UIDarshan
07/21/2021, 7:26 PMFrank Fineis
07/21/2021, 7:38 PMkiran
07/21/2021, 9:23 PMwith Flow(...):
context manager basically replace using a main()
function? i.e., instead of something like this:
from prefect import task, Flow
@task
def say_hello():
print("Hello, world!")
@task
def say_goodbye():
print("Goodbye, world!")
with Flow("My First Flow") as flow:
say_hello()
say_goodbye()
Would main()
be used with the context manager (and is it weird/redundant to do so)? i.e., would you decorate each function with @task
, order them in main()
(but not decorate main()
) and then call main()
from the context manager like so:
from prefect import task, Flow
@task
def say_hello():
print("Hello, world!")
@task
def say_goodbye():
print("Goodbye, world!")
def main():
say_hello()
say_goodbye()
if __name__ == "__main__":
with Flow("My First Flow") as flow:
main()
Thanks for any advice/insights on best practices around this!DK
07/21/2021, 9:58 PMwith Flow('Test 1') as flow:
print(f"""
flow name: {context.flow_name}
flow id: {context.flow_id}
flow run name: {context.flow_run_name}
flow_run_id: {context.flow_run_id}
""")
results in :
flow name: Test 1
flow id: Test 1
flow run name: bc5e286e-6268-43a8-aaaf-a5d03275917d
flow_run_id: c867f4ea-1599-4201-a8e0-886cadbe0e1e
It looks like the context variables are not entirely correct? Flow Name and Flow ID are both pulling the same field and I would expect Flow Run Name to pull something like 'violet-marmoset'.
Is there an actual issue here or am I using these incorrectly?Maikel Penz
07/22/2021, 4:49 AMpython_dependencies
property.
However, I’m doing it dynamically and reading from a flow configured with Poetry. I noticed that the format in Poetry doesn’t match what is expected in the python_dependencies
E.g:
In Poetry: colorama = "^0.4.4"
What I tried: `python_dependencies=["colorama^0.4.4"]`<< doesn’t work
How it should be: python_dependencies=["colorama>=0.4.4"]
Before I write something ugly to convert Poetry to Requirements.txt style, is there something available on Prefect to convert it or Poetry support ?Tom Blake
07/22/2021, 6:38 AMRob Fowler
07/22/2021, 7:20 AMdipsy wong
07/22/2021, 8:46 AMMohammad GH
07/22/2021, 9:08 AMMohammad GH
07/22/2021, 9:08 AM09:32:17 INFO CloudFlowRunner Flow run SUCCESS: all reference
tasks succeeded 09:32:17 ERROR CloudFlowRunner Unexpected error:
AttributeError("'Flow' object has no attribute
'terminal_state_handler'") Traceback (most recent call last): File
"/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line
48, in inner new_state = method(self, state, *args, **kwargs) File
"/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py",
line 669, in get_flow_run_state state = self.determine_final_state(
File
"/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py",
line 720, in determine_final_state if
self.flow.terminal_state_handler: AttributeError: 'Flow' object has no
attribute 'terminal_state_handler'
Mohammad GH
07/22/2021, 9:09 AMterminal_state_handler
issueBrett Jurman
07/22/2021, 10:47 AMMichal Zawadzki
07/22/2021, 11:25 AMAlexander
07/22/2021, 12:01 PM22 July 2021,11:22:45 MSK prefect.sometask ERROR Our job is failed
22 July 2021,11:22:45 MSK prefect.CloudTaskRunner DEBUG Task 'facebook_insights': Execution process closed, collecting result...
22 July 2021,11:22:45 MSK prefect.CloudTaskRunner DEBUG Task 'facebook_insights': Handling state change from Running to TimedOut
The "Execution process closed, collecting result..." log is coming from prefect.utilities.executors.run_with_multiprocess_timeout
. But i dont see any other logs from this functions (according to source code they exist there). There is indeed a timeout set for a task, like a 1 hour, but according to logs it finished in 20 minutes.
How can i debug this?
We are running latest prefect version with ECS agent