Nathaniel Russell
09/26/2022, 8:12 PMBradley Hurley
09/26/2022, 8:14 PM"No heartbeat detected from the remote task"
issue a few times and its well documented here to consult https://discourse.prefect.io/t/flow-is-failing-with-an-error-message-no-heartbeat-detected-from-the-remote-task/79.badasstronaut
09/26/2022, 9:13 PMDaskTaskRunner
class has a Dask client under self._client
, but it does not look like that’s intended to be used by the flow or tasks. Any tips are appricated!badasstronaut
09/27/2022, 1:07 AMZac Hooper
09/27/2022, 2:13 AMBen Ayers-Glassey
09/27/2022, 2:48 AMsecret = prefect.client.secrets.Secret(SFTP_PASSWORD_SECRET_NAME)
return secret.get()
...and it fails with a prefect.exceptions.ClientError
from the GraphQL API.Ben Ayers-Glassey
09/27/2022, 3:37 AMGuillaume G
09/27/2022, 7:25 AMKubernetesJob
? I want to pull a image from a private registry.
reception_deployment : Deployment = Deployment.build_from_flow(
name="private-flow",
flow=main_flow,
output="deployment-private-flow.yaml",
description="private-flow",
version="snapshot",
work_queue_name="kubernetes",
infrastructure=KubernetesJob(),
infra_overrides=dict(
image="myprivate-image:latest",
env={},
finished_job_ttl=300)
)
I can not pull "myprivate-image:latest"
Do I have to use service_account
? 🤨
Thank youMalavika S Menon
09/27/2022, 7:26 AM@flow
def random_flow():
id = Flow.flow_id
return 1
How do I access any member say flow_id inside the flow, when it is being run?Hongbo Miao
09/27/2022, 8:04 AMawait
. Posted the answer. Hi Prefect team, worth updating that demo. Thanks!flapili
09/27/2022, 8:17 AMflapili
09/27/2022, 8:18 AMDeepanshu Aggarwal
09/27/2022, 9:29 AMDavid
09/27/2022, 9:52 AMMichal Luščon
09/27/2022, 11:48 AMHa Pham
09/27/2022, 12:18 PMstart_datetime
and end_datetime
to go together and start should comes first, not after like thisHamza Naanani
09/27/2022, 1:47 PMoverride
parameter in prefect deployment build
to change kubernetes job parameters ?Tim Helfensdörfer
09/27/2022, 2:32 PMif profile_name is None:
return profile_map.get('default', {})
elif profile_name not in profile_map:
....
So if you do not want to use a profile you have to overwrite profile_name
manually with None
in your code and then save it again. A fix would be, as an example, a checkbox next to nullable fields to set this field to null.Stephen Herron
09/27/2022, 2:44 PMChris Gunderson
09/27/2022, 3:00 PMChristian Juhl
09/27/2022, 3:34 PMfrom prefect import flow, task, get_run_logger
from prefect.orion.schemas.states import Failed
def my_function():
raise Exception('This is an exception')
@flow
def run_flow():
logger = get_run_logger()
try:
result = my_function()
<http://logger.info|logger.info>(result)
except Exception as e:
errmsg = f"Error! Type: {e.__class__.__name__} Detail: {e}"
logger.error(errmsg)
return Failed()
if __name__ == "__main__":
run_flow()
However, I get the following error;
Exception has occurred: TypeError
Unexpected result for failure state: None —— NoneType cannot be resolved into an exception
File "main.py", line 21, in <module>
run_flow()
I'm thinking I might be doing more than one thing wrong here. Can anyone enlighten me, please? Thanks!Joe Schueller
09/27/2022, 3:58 PMprefect-dbt
and can successfully execute a flow using the trigger_dbt_cli_command
task, but dbt doesn't actually do anything.Tim Helfensdörfer
09/27/2022, 4:40 PM[]
) in my yaml file the command points to my prefect entry script but it needs the UUID of the flow run. So it boils down to two questions:
1. How does the queue worker provide the UUID into the ecs task
2. What does Command
do?
(in the second screenshot I changed the task definition command from [] to python3 /app/tasks/run.py
- but it still needs the flow run uuid)chicago-joe
09/27/2022, 4:42 PM"containerDefinitions": [
"entryPoint": [
"bash",
"-c"
],
"command": [
"prefect agent start -q ecs-worker"
],
results in error:
Usage: prefect agent [OPTIONS] COMMAND [ARGS]...
Try 'prefect agent -h' for help.
Error: No such command 'start'.
"containerDefinitions": [
"entryPoint": [
"bash",
"-c"
],
"command": [
"prefect","agent","start","-q","ecs-worker"
],
results in what appears to be calling "prefect" with nothing else, as the logs just keep repeating the menu stdout:Bradley McLaughlin
09/27/2022, 4:54 PMBrett Naul
09/27/2022, 4:56 PMStephen Herron
09/27/2022, 5:12 PMConfiguration:
profiles.yml file [[32mOK found and valid[0m]
dbt_project.yml file [[32mOK found and valid[0m]
Required dependencies:
- git [[32mOK found[0m]
Nick DeCraene
09/27/2022, 6:55 PMJacob Blanco
09/27/2022, 8:44 PM[{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
The interesting part is that this only seems to have to jobs at 9AM JST and 9PM JST. Which would be midday and midnight UTC.Chris Gunderson
09/27/2022, 9:10 PM@task(name='Get Email Credentials',
retries = 2,
retry_delay_seconds = 10)
async def GetEmailCredentials(aws_credentials: AwsCredentials) -> EmailCredentials:
session = aws_credentials.get_boto3_session()
secrets_client = session.client(service_name='secretsmanager')
aws_secret = secrets_client.get_secret_value(SecretId='DEV_EMAIL')
secret_json = aws_secret['SecretString']
emailDict = json.loads(secret_json)
return EmailCredentials(emailDict)