Blake List
07/12/2021, 10:40 PMdf = df.apply(my_function, axis=1)
with prefect.
Thanks!Pedro Machado
07/12/2021, 11:21 PM<http://logger.info|logger.info>()
? Thanks!Ben Muller
07/12/2021, 11:58 PMflow.run(run_on_schedule=False)
.
I can't find this option for the cli when running prefect run -p my_flow.py
What am I missing?Ben Muller
07/13/2021, 6:14 AMError uploading to S3: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
This is from my S3Result object trying to persist the results to the bucket.
Not sure why this is happening as my task-role
has full S3 access and I have an identical set up in staging, yet I dont get these errors.
Am I right in assuming that the S3Result just assumes the task-role
? Why would boto3 not be picking this up?Ben Muller
07/13/2021, 6:19 AMBen Muller
07/13/2021, 6:37 AMMarko Herkaliuk
07/13/2021, 10:16 AMLaura Vaida
07/13/2021, 10:23 AMHilary Roberts
07/13/2021, 1:35 PMstorage = Docker(registry_url="https://<our account id>.<http://dkr.ecr.eu-west-1.amazonaws.com/data/prefect-application/%22|dkr.ecr.eu-west-1.amazonaws.com/data/prefect-application/">)
storage.build()
I get this error
[2021-07-13 14:32:00+0100] INFO - prefect.Docker | Building the flow's Docker storage...
invalid reference format
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/hilaryroberts/.pyenv/versions/3.8.11/lib/python3.8/site-packages/prefect/storage/docker.py", line 303, in build
self._build_image(push=push)
File "/Users/hilaryroberts/.pyenv/versions/3.8.11/lib/python3.8/site-packages/prefect/storage/docker.py", line 369, in _build_image
raise ValueError(
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
It should be using the default image from the prefecthq dockerhub, so I wonder why it's failing to build. Anyone know what I might be doing wrong?Ali Abbas Zaidi
07/13/2021, 2:05 PMDmitry
07/13/2021, 2:13 PMfrom prefect.storage.github import GitHub
my_string = GitHub(access_token_secret="github", repo="my_repo", path="etl/query.sql")
but I get only <class 'prefect.storage.github.GitHub'>
I need to get this file like a string.
How I can do it?Zach Schumacher
07/13/2021, 2:33 PM@task(name="set-date-params", nout=2)
def set_date_params(
start_date: typing.Optional[str],
end_date: typing.Optional[str]
) -> typing.Tuple[datetime.date, datetime.date]:
"""
If start date and end date are explicitly set on the Flow, we simply construct date objects and return them.
If start_date is not set, we default it to 7 days from the local context date of the flow.
If end_date is not set, we default it to the local context date of the flow.
"""
context_run_datetime = context.get("date")
aware_utc_run_datetime = context_run_datetime.replace(tzinfo=utc)
aware_et_run_datetime = aware_utc_run_datetime.astimezone(et)
if not start_date:
seven_days_ago = aware_et_run_datetime - datetime.timedelta(days=7)
start_date = seven_days_ago.date().isoformat()
if not end_date:
end_date = aware_et_run_datetime.date().isoformat()
<http://logger.info|logger.info>(f"start_date={start_date}, end_date={end_date}")
return datetime.date.fromisoformat(start_date), datetime.date.fromisoformat(end_date)
with flow:
start_date_, end_date_ = Parameter("start-date", default=None), Parameter("end-date", default=None)
start_date_, end_date_ = set_date_params(start_date_, end_date_)
Thomas Opsomer
07/13/2021, 2:41 PMPod prefect-job-9f127532-nnd2f failed.
Container 'flow' state: terminated
Exit Code:: 1
Reason: Error
No heartbeat detected from the remote task; marking the run as failed.
But when I look at the pod logs, I can see the real error.
In addition to that, we have no slack notification when the flow fails like that.
Any idea how to overcome this ?domagoj jercic
07/13/2021, 2:50 PMMarie
07/13/2021, 2:55 PMPending
and the run shows as In progress
until I manually cancel it.
Did anyone already run into this issue?
No heartbeat detected from the remote task; marking the run as failed.
Bruno Murino
07/13/2021, 2:58 PMSarita Patel
07/13/2021, 4:04 PMprefect create project 'prod'
to create it."Joseph Loss
07/13/2021, 5:05 PMHugo Kitano
07/13/2021, 5:14 PMquery = """mutation {
create_flow_run(input: {
flow_id: "5ac80adc-442a-4e53-bc01-d9d65bffd6aa"}) {
id
}
}"""
Joseph Loss
07/13/2021, 5:50 PMBruno Murino
07/13/2021, 8:08 PMENTRYPOINT ["make", "deploy"]
and when I ran the flow no logs came to the UI at all (as expected, I think), because the container threw an error with “make” (probably because directory config and etc so no worries about it)
so the flow is still in the “submitted” state, even though the actual ECS task has already failed.
Is there any way to have better visibility of when that happens? in case a dev forgets to remove the entrypoint from the dockerfile..Kathryn Klarich
07/13/2021, 8:46 PMVincent
07/13/2021, 9:08 PMEnda Peng
07/14/2021, 2:43 AMprefect backend server
Traceback (most recent call last):
xxxxx
File "pyev/lib/python3.6/site-packages/prefect/utilities/notifications/__init__.py", line 1, in <module>
from prefect.utilities.notifications.notifications import callback_factory
File "pyev/lib/python3.6/site-packages/prefect/utilities/notifications/notifications.py", line 13, in <module>
from toolz import curry
ModuleNotFoundError: No module named 'toolz'
Blake List
07/14/2021, 3:10 AMBruno Murino
07/14/2021, 9:42 AMFailed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')
Slackbot
07/14/2021, 2:45 PMLeon Kozlowski
07/14/2021, 3:10 PM--build-arg
for a Dockerfile to the prefect register CLI command?Nishtha Varshney
07/14/2021, 4:39 PMNelson Griffiths
07/14/2021, 4:46 PM