Henry
11/17/2020, 8:11 PMGraphQLError: x-hasura-admin-secret/x-hasura-access-key required, but not found]
with a setup using a hasura admin secret - i wasn'ta ble to find a env var for it in prefect_server
https://github.com/PrefectHQ/serverHenry
11/17/2020, 8:12 PMPhilip MacMenamin
11/17/2020, 9:23 PMPatrick
11/17/2020, 11:20 PMRunConfig
is slowly taking the place of Environment
as the primary abstraction for specifying execution environments, though it’s still marked experimental. Does anyone, especially longtime users and/or maintainers, have any general advice on which to use if just getting started?
I’d like to be using Prefect for “production” work within a few weeks, so a somewhat stable API would be grand. OTOH, we don’t want to adopt conventions whose obsolescence is already planned. Also, we’re an AWS shop. Thanks in advance!Riley Hun
11/17/2020, 11:27 PMImageNotFound
Error:
[2020-11-17 23:17:14,473] INFO - agent | Pulling image <http://gcr.io/aa-mlops-dev-inm5/prefect-etl-storage:0.1.0|gcr.io/aa-mlops-dev-inm5/prefect-etl-storage:0.1.0>...
/snap/google-cloud-sdk/160/lib/third_party/requests/__init__.py:83: RequestsDependencyWarning: Old version of crypt
ography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
[2020-11-17 23:17:16,498] INFO - agent | Successfully pulled image <http://gcr.io/aa-mlops-dev-inm5/prefect-etl-storage:0.1|gcr.io/aa-mlops-dev-inm5/prefect-etl-storage:0.1>
.0...
[2020-11-17 23:17:16,517] ERROR - agent | Logging platform error for flow run f42d17a1-fb46-4556-8253-3b10f5a482e8
[2020-11-17 23:17:16,576] ERROR - agent | Error while deploying flow: ImageNotFound(HTTPError('404 Client Error: No
t Found for url: <http+docker://localhost/v1.40/containers/create',>),)
Sam Kolli
11/18/2020, 12:31 AMAiden Price
11/18/2020, 5:46 AM<https://etl.mydomain.com/prefect>
. But when I try to go to the dashboard it doesn't load, looking at Chrome's console I can see that the page is requesting all its files without my path part, e.g. <https://etl.mydomain.com>
. I tried adding the env var PREFECT_SERVER__SERVER_UI_ENDPOINT
as the full URL and path but it doesn't seem to have changed the behaviour. Does anyone have any advice? Thanks.Michelle Wu
11/18/2020, 8:34 AMEmailTask
in state_handlers
. I want to receive a email message that shows task
and new_state
when failures occur. My handler looks something like this:
def email_on_failure(task, old_state, new_state):
if new_state.is_failed():
task = EmailTask(
subject="Prefect alert: {}".format(new_state),
msg = "Task: {}; New State: {}".format(task, new_state),
email_from='xxx',
email_to='xxx',
smtp_server='smtp.gmail.com',
smtp_port=xxx,
smtp_type='STARTTLS'
).run()
In tests, I can always receive the email with the correct subject, but the task
and the new_state
was always missing in the msg
…Anyone has encountered similar problems?Saulius Beinorius
11/18/2020, 11:55 AMAlex Bussan
11/18/2020, 2:10 PMKubernetesRun
https://github.com/PrefectHQ/prefect/blob/master/src/prefect/run_configs/kubernetes.py
My goal is to be able to set a different job template on a "per Flow Run basis". It seems currently this is set up for a "per flow" basis since you have to do:
flow.run_config = KubernetesRun(...)
flow.register(project_name=project_name)
So it appears you have to re-register if you want different resource limits per flow run. Any ideas on how to best achieve this?psimakis
11/18/2020, 3:40 PM$ prefect agent fargate start cpu=256 memory=512 executionRoleArn=arn:aws:iam::xxxxxxxxx:role/ecsTaskExecutionRole networkConfiguration="{'awsvpcConfiguration':{'assignPublicIp': 'ENABLED', 'subnets': ['subnet-xxxxxx'], 'securityGroups': ['sg-xxxxxxxxxxx']}}" cluster=fargate-cluster
Below you will find the flow:
from prefect import Flow, task, Parameter
from prefect.environments.storage import Docker
@task
def square_int(x):
return x * x
@task
def sub_2(x):
return x - 2
with Flow(
name='sp-flow-test',
storage=Docker(registry_url='xxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com',
image_name='prefecttest', image_tag='latest'),
) as flow:
integer = Parameter('integer', default=2)
x_sqr = square_int(integer)
x_less_2 = sub_2(x_sqr)
flow.register(project_name='test', idempotency_key=flow.serialized_hash())
When I run the flow, the fargate task seems to run fine (check out the screenshot). The problem is that flow run stays Submitted
forever. Have you any clue?
Thanks.Carlo
11/18/2020, 3:41 PMAlex Bussan
11/18/2020, 4:29 PMflow.run_config = KubernetesRun(
job_template_path="my_custom_template.yaml"
)
vs configuring/running a flow this way (using KubernetesJobEnvironment)
flow = Flow(
"flow_w_jobenv",
environment=KubernetesJobEnvironment(job_spec_file="job_spec.yaml"),
storage=Docker()
)
I'm struggling to understand the differences/ tradoffs of configuring a job one way vs another. Both ways you can pass in information that a job spec would need. Then you register the flow and run it and the k8s agent picks it up. Is there any fundamental difference I'm missing?Sean Harkins
11/18/2020, 5:12 PMcloudpickle
serializes my Flow's dependencies with S3 storage. Based on some information outlined here https://prefect-community.slack.com/archives/C014Z8DPDSR/p1605200879483400 I've got S3 storage configured and working but it seems that my Flow's upstream dependencies are not pickled when I register the flow as indicated in https://docs.prefect.io/core/advanced_tutorials/task-guide.html#task-inputs-and-outputs. Specifically I receive the following dependency error Unexpected error: ModuleNotFoundError("No module named 'h5netcdf'")
. I can build an image with the necessary dependencies for use with the DaskExecutor
and everything works correctly but our goal is to decouple our execution environment from Flows.
Am I misunderstanding how Flow dependencies should be serialized by cloudpickle
. Is there another approach I should be considering in this case? Thanks in advance.Riley Hun
11/18/2020, 5:55 PM[2020-11-18 17:51:18+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'ETL'
[2020-11-18 17:51:20+0000] INFO - prefect.CloudFlowRunner | Flow run FAILED: some reference tasks failed.
Also when I look into the kubernetes pod logs, it references Prefect Cloud.
[2020-11-18 17:51:22+0000] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
Any guidance would be much appreciated!Marco Palmeri
11/18/2020, 6:10 PMextract_region_transactions
starts running as soon as any of the decrypt_file_task
worker tasks complete. -- See code snippet in thread.
``````Gabriel Montañola
11/18/2020, 8:03 PMdbt
error logs sent to my task log but they are coming up as blank and I've already set log_stderr
to True in the task definition.Isaac Brodsky
11/18/2020, 9:19 PM'PrefectSecret' object has no attribute 'secret_name'
which started around November 11. Possibly there was a change to my execution environment like using a newer/different Prefect agent at that point. Do I need to reinstall the Prefect Agent?Dolor Oculus
11/18/2020, 9:32 PMDolor Oculus
11/18/2020, 10:18 PMlog_stdout=True
to all of our tasks, and the log messages are showing up just fine in the agent's stdout, but we'd like them to also be available on the web ui. We've discovered that by getting the logger from the context, we can do this, but this is a pretty invasive change for our codebase (most of our modules are using standard logging and we don't want to change that). Is there any magic we can invoke to get our logger messages to show up on the web ui?John Grubb
11/18/2020, 10:35 PMJohn Grubb
11/18/2020, 10:35 PMRiley Hun
11/18/2020, 10:40 PMhost.docker.internal
to /etc/hosts
via --add-host
? Is this something we add to the running agent or the config.toml?simone
11/19/2020, 2:45 PM@task
decorator. When the function is executed in a flow everything runs fine. If I write the same code by subclassing Task
I get a pickling error:
pickle.UnpicklingError: NEWOBJ class argument isn't a type object
The issue happened in a couple of other functions. I guess there is something basic in the Task construction that I am missing. I will appreciate if you can shine some light on this issue! Thanks
Example of the code: Gist Tasks codeKevin Weiler
11/19/2020, 6:14 PMMike Grabenstein
11/19/2020, 9:01 PMWalt Wells
11/20/2020, 12:16 AMSaulius Beinorius
11/20/2020, 8:10 AMKlemen Strojan
11/20/2020, 9:18 AMAlexander
11/20/2020, 11:06 AM