YD
06/11/2021, 11:23 PMBen Muller
06/11/2021, 11:26 PMThomas Nyegaard-Signori
06/14/2021, 10:01 AMStarted following logs for <task pod>
and then nothing more other than the completion and deletion log. Meanwhile, if I check the kubernetes job pod logs (kubectl logs -n my-namespace prefect-job-...
I see the logs of the task pod showing up nicely, just not being sent back to the the Agent/Server. Anyone have a similar experience and know of that one flag I'm missing?joshua mclellan
06/14/2021, 3:17 PMFina Silva-Santisteban
06/14/2021, 6:30 PMflow.storage = GitHub(secrets=["GITHUB_ACCESS_TOKEN"],
repo="my_org/repo_name",
path="prefect_flows/flows/flow.py",
ref="trunk")
The access token is saved in prefect cloud under the same name. During the flow run prefect can’t seem to be able to find the repo??
INFO:Downloading flow from GitHub storage - repo: 'my_org/repo_name', path: 'prefect_flows/flows/flow.py', ref: 'trunk'
ERROR: Repo 'my_org/repo_name' not found. Check that it exists (and is spelled correctly), and that you have configured the proper credentials for accessing it.
ERROR: Failed to load and execute Flow's environment: UnknownObjectException(404, {'message': 'Not Found', 'documentation_url': '<https://docs.github.com/rest/reference/repos#get-a-repository>'}, {'server': '<http://GitHub.com|GitHub.com>', 'date': 'Fri, 11 Jun 2021 22:46:06 GMT', 'content-type': 'application/json; charset=utf-8', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '0', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'vary': 'Accept-Encoding, Accept, X-Requested-With', 'content-encoding': 'gzip', 'x-ratelimit-limit': '60', 'x-ratelimit-remaining': '59', 'x-ratelimit-reset': '1623455165', 'x-ratelimit-resource': 'core', 'x-ratelimit-used': '1', 'content-length': '112', 'x-github-request-id': 'C202:57D9:E1FC5:1D69CC:60C3E7AD'})
I’ve curled the github api endpoint repos/
directly using the same auth token and I do get the repo’s information. Pls advise! 🙏Jeremy Phelps
06/14/2021, 7:26 PMTask.run
method? I'd like to do something like this:
class MyTask(prefect.tasks.core.function.FunctionTask):
def run(*args, **kwargs):
try:
return self.super().run(*args, **kwargs)
except BaseException as e:
prefect.context.get('logger').error('Error on host {}: {}'.format(hostname, format_error(e)))
raise # Or better yet, mark the task as Failed
By default, Prefect catches, logs, and marks the task as failed for some errors, but for others, it puts str(e)
in the "State Message" and throws away the stack trace, and for still others, it drops the error completely and the task appears to be "running". My wrapper would fix these problems.
But you can't write the method above because there's a function called _validate_run_signature
that expressly forbids it. So the only way to do it that I can think of would be to either wait for Python to get macros, or to add code generation to my deployment process (it would insert the try/except block into the body of every function that has a @task
decorator).
Is it really impossible to write this wrapper as an ordinary method?Verun Rahimtoola
06/14/2021, 9:06 PMError while deploying flow: ValidationError({'type': ['Unsupported value: UniversalRun']
with prefect version 0.14.21
Raúl Mansilla
06/14/2021, 9:12 PMpeter zhao
06/14/2021, 9:43 PMpeter zhao
06/14/2021, 9:44 PMpeter zhao
06/14/2021, 9:45 PMrequests.exceptions.SSLError: HTTPSConnectionPool(host='<http://opensky-network.org|opensky-network.org>', port=443): Max retries exceeded with url: /api/states/all?lamin=37.15294209056667&lamax=40.75094679823332&lomin=-79.76136868215879&lomax=-75.13474242904122 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
Scott Vermillion
06/15/2021, 12:36 AMERROR - <name> | Failed to query for flow run metadata
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/prefect/agent/agent.py", line 324, in _submit_deploy_flow_run_jobs
flow_runs = self._get_flow_run_metadata(flow_run_ids)
File "/home/ubuntu/.local/lib/python3.8/site-packages/prefect/agent/agent.py", line 684, in _get_flow_run_metadata
result = self.client.graphql(query)
File "/home/ubuntu/.local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
I rebuilt my agent (was planning to anyway) but same error right away upon launching. I deleted and reinitialized my project. It doesn’t seem like I’m doing anything differently than before when the agent appeared happy but my issue was further down the line. Really mystified right now, but I’m a bit frazzled from working it so many hours…maybe I’m missing something obvious? (I’m also not the least bit experienced with Prefect - nor very much with AirFlow.)Prabin Mehta
06/15/2021, 12:00 PMThomas Nyegaard-Signori
06/15/2021, 12:50 PMazure
storage? I had a look at comparing the s3
implementation (https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/s3.py#L159) to the azure
one (https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/azure.py#L128), and it seems like the stored_as_script
flag doesnt do much on the azure
storage. Can anyone from the Prefect team confirm what I am seeing or am I on the wrong track?Tadas
06/15/2021, 1:30 PMMilly gupta
06/15/2021, 1:31 PMWai Kiat Tan
06/15/2021, 3:32 PMciaran
06/15/2021, 3:48 PMprefect agent kubernetes
in a custom Docker Image?
I can get my image running on AWS Fargate with:
command=[
"prefect",
"agent",
"ecs",
"start",
"--agent-address",
"http://:8080",
"--cluster",
cluster.cluster_arn,
"--task-role-arn",
ecs_task_role.role_arn,
],
But on AKS it's a whole different story, I can't get it running 😭 Essentially it can't find prefect but I can run the container locally and invoke prefectZach Schumacher
06/15/2021, 5:04 PMZach Schumacher
06/15/2021, 5:46 PMflow.schedule = CronSchedule("30 12 * * *")
Kathryn Klarich
06/15/2021, 7:59 PMJustin Liu
06/15/2021, 8:25 PMMatthew Blau
06/15/2021, 8:30 PMJason Prado
06/15/2021, 10:57 PMBen Muller
06/16/2021, 7:21 AMTimo
06/16/2021, 7:21 AMAyyanar Thangaraj
06/16/2021, 7:21 AMThomas Hoeck
06/16/2021, 7:23 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': '[{\'extensions\': {\'path\': \'$.selectionSet.billing_license.selectionSet.account_name\', \'code\': \'validation-failed\'}, \'message\': \'field "account_name" not found in type: \\\'billing_license\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': '[{\'extensions\': {\'path\': \'$.selectionSet.billing_license.selectionSet.account_name\', \'code\': \'validation-failed\'}, \'message\': \'field "account_name" not found in type: \\\'billing_license\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
I'm running Prefect 0.14.7 in the flows and the agent is 0.14.17Ayyanar Thangaraj
06/16/2021, 7:24 AMJeremy Phelps
06/16/2021, 10:27 AMKeyError: 'Task slug disable_sound_on_mixer_error_stores-1 not found in the current Flow; this is usually caused by changing the Flow without reregistering it with the Prefect API.'
The slug is a transformation of the name of a flow. Prefect replaced the hyphens with _
and added -1
to the end.
The error is happening when trying to run a different flow. Both flows are defined in the same Python source file.
Both flows were registered with the latest code, contrary to the suggestion in the error message.