Andrew Lawlor
04/21/2022, 6:17 PMFailed to load and execute flow run: RefreshError(TransportError("Failed to retrieve <http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/ACCOUNT/token?scopes=SCOPES> from the Google Compute Engine metadata service. Status: 500 Response:\nb'Unable to generate access token; IAM returned \\n'", <google.auth.transport.requests._Response object at 0x7f5b071fd670>))
Andrew Lawlor
04/21/2022, 6:36 PMJustin Burchard
04/21/2022, 7:25 PMFabio Scarpellini
04/21/2022, 7:41 PMAssaf Ben Shimon
04/21/2022, 11:01 PMDarshan
04/22/2022, 12:26 AMAhmed Ezzat
04/22/2022, 4:35 AMRainer Schülke
04/22/2022, 7:56 AMStefan Crnojević
04/22/2022, 11:40 AMstandard_init_linux.go:228: exec user process caused: exec format error
error in logs though, so not 100% sure the issue's hardware-related.
Any suggestions would be appreciated 🙏Malthe Karbo
04/22/2022, 12:30 PMprefectfuture[list[str]])
object)
◦ this is where I would like to yield
• Run some down stream processing on each chunk
◦ Here we use future.result() to loop over and start each task - this works 100%, but if we could use yield in the first task, we could begin processing immediately
Is something like this supported / going to be supported? I tried making it work with contextmanager and tasks, but ran into nothing but troubleJoshua Greenhalgh
04/22/2022, 12:50 PM--job-template
on the prefect agent kubernetes install
command in the hope that this would work but in fact it does not - do I need to modify the containers args which are prefect agent kubernetes start
to use the flag?Andrew Lawlor
04/22/2022, 1:23 PMXavier Babu
04/22/2022, 3:30 PMAmogh Kulkarni
04/22/2022, 4:20 PMException raised while calling state handlers: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 137, in get
value = secrets[self.name]
KeyError: 'SLACK_WEBHOOK_URL'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 64, in call_runner_target_handlers
new_state = super().call_runner_target_handlers(
File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
new_state = handler(self.task, old_state, new_state) or new_state
File "/usr/local/lib/python3.9/site-packages/toolz/functoolz.py", line 306, in __call__
return self._partial(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/notifications/notifications.py", line 299, in slack_notifier
str, prefect.client.Secret(webhook_secret or "SLACK_WEBHOOK_URL").get()
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 161, in get
raise exc
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 145, in get
result = self.client.graphql(
File "
/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
03:02:29
IArtem Vysotsky
04/22/2022, 4:23 PMPostgresFetch
in Prefect 2.0. Any pointers?Domenico Di Gangi
04/22/2022, 4:47 PMJason
04/22/2022, 4:47 PMChris Reuter
04/22/2022, 5:29 PMTom Manterfield
04/22/2022, 8:36 PMLight Luo
04/22/2022, 9:05 PMArtem Vysotsky
04/22/2022, 9:35 PMjobs
like following:
id | name | query | template
-----------------------------------------------------------------------------------------
1 | user_count_query | select count(*) as cnt from users | user count is {{cnt}}
2 | projects_count_query | select count(*) as cnt from projects.| project count is {{cnt}}
I would like to
1. Use fetch_jobs = PostgresFetch
task to get a list of all jobs in the table above
2. Fan out the results of fetch_jobs
to new task say run_job_query
which will for each row grab value of query
column and execute another PostgresFetch(query=query)
In a pseudo flow it would look something like:
fetch = PostgresFetch()
@task
def send_to_slack():
pass
with prefect.Flow() as flow:
jobs = fetch(query="select query from jobs"), fetch="many")
for job in jobs:
query_result = fetch(query=job["query"])
send_to_slack(query_result)
The above code does not obviously work because Tasks are not iterable. Mapping won’t work too, since I cannot call another task in a task
Any help appreciated.Ahmed Ezzat
04/23/2022, 6:33 AMscott simpson
04/24/2022, 8:37 AMSlackbot
04/24/2022, 1:58 PMAssaf Ben Shimon
04/24/2022, 3:24 PMTraceback (most recent call last):
File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/dockers/extract_markers.py", line 2, in <module>
from biomarkers.batch_processing import orchestrator
File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/batch_processing/orchestrator.py", line 130, in <module>
seq_markers_extraction_flow.register("neuralight-research")
File "/usr/local/lib/python3.9/site-packages/prefect/core/flow.py", line 1708, in register
registered_flow = client.register(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 839, in register
project = self.graphql(query_project).data.project # type: ignore
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
result = <http://self.post|self.post>(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
response = self._request(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 649, in _request
raise AuthorizationError(
prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
Any idea what can cause it? I've already authenticated using my API keyChris Reuter
04/24/2022, 8:52 PMJoshua Greenhalgh
04/25/2022, 10:29 AMCronSchedule("0 11 * * *")
to fire at 11 and not 12?Ilhom Hayot o'g'li
04/25/2022, 10:30 AMNuno Silva
04/25/2022, 1:47 PMMatt Delacour
04/25/2022, 2:51 PMMatt Delacour
04/25/2022, 2:51 PMalex
04/25/2022, 3:49 PMprefect-dbt
collection that will have all the functionality of the current dbt tasks in Prefect 1.0.
Here’s a blog post about Prefect 2.0 collections and our current list of collections that are available for more info!Matt Delacour
04/25/2022, 3:59 PMMatthias
04/25/2022, 5:11 PMprefect-shell
. This is the equivalent of the ordinary ShellTask
in Prefect 1.0 on which DbtShellTask
was written on top of. https://prefecthq.github.io/prefect-shell/Matt Delacour
04/25/2022, 5:31 PMprocess
directly