Tomás Emilio Silva Ebensperger
09/28/2022, 6:17 PMcommand=/home/ubuntu/project/env/bin/prefect agent start --work-queue "my queue"
This works fine but then when i run flows in the UI the worker is using the global python runtime and no the one in the virtual environment, please helpChris Gunderson
09/28/2022, 6:34 PMfrom pydantic import BaseModel
class CustomList(BaseModel):
data: list[str]
@flow(name = "Fidelity Allocations")
def FidelityAllocationsFlow(toEmail: CustomList):
#code....
if __name__ == "__main__":
emailAddress = CustomList(data=['<mailto:cgunderson@spiderrockadvisors.com|cgunderson@spiderrockadvisors.com>'])
FidelityAllocationsFlow(toEmail=emailAddress)
Chris Reuter
09/28/2022, 6:58 PMAmol Shirke
09/28/2022, 7:23 PMAndreas Nigg
09/28/2022, 7:59 PMFile "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 226, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/bd169b15-9cf0-41df-9e46-2233ca3fcfba/workspaces/f507fe51-4c9f-400d-8861-ccfaf33b13e4/task_runs/|https://api.prefect.cloud/api/accounts/bd169b15-9cf0-41df-9e46-2233ca3fcfba/workspaces/f507fe51-4c9f-400d-8861-ccfaf33b13e4/task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
Rio McMahon
09/28/2022, 8:34 PMprefect.infrastructure.process - Process 'meticulous-manatee' exited with status code: -9
Any guidance?Nick DeCraene
09/28/2022, 8:45 PM@task(name="extract(ctx.id)")
def extract(ctx):
These would then be displayed in the UI as extract(4)
for example if our flow was running for ctx.id=4.
I haven't been able to find anything like this for 2.0, am I just missing something?James Constable
09/28/2022, 10:09 PMAndrew Reeve
09/28/2022, 10:56 PMAndrew Reeve
09/28/2022, 10:57 PMAttributeError: 'Credentials' object has no attribute 'project'. Did you mean: 'project_id'?
It looks like your PR will fix this @Andrew Huang.Deepak
09/28/2022, 11:08 PMRuntimeError: Java gateway process exited before sending its port number
when I run pyspark methods inside a task.Jean
09/28/2022, 11:10 PM23:04:54.057 | INFO | prefect.engine - Created flow run 'optimal-snail' for flow 'test1'
23:04:54.057 | INFO | prefect.task_runner.dask - Creating a new Dask cluster with `distributed.deploy.local.LocalCluster`
23:04:55.980 | INFO | prefect.task_runner.dask - The Dask dashboard is available at <http://127.0.0.1:8787/status>
23:05:00.371 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-0' for task 'Execute values of the query'
23:05:00.372 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-0' immediately...
23:05:24.612 | INFO | Task run 'Execute values of the query-8165e3c8-0' - Finished in state Completed()
23:05:26.370 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-1' for task 'Execute values of the query'
23:05:26.370 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-1' immediately...
23:05:42.533 | INFO | Task run 'Execute values of the query-8165e3c8-1' - Finished in state Completed()
23:05:44.295 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-2' for task 'Execute values of the query'
23:05:44.296 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-2' immediately...
23:06:09.538 | INFO | Task run 'Execute values of the query-8165e3c8-2' - Finished in state Completed()
As you see in these logs it’s not really spawning more tasks. I was under the impression that the call to a function with the @task
decorator would be non-blocking if made within a flow @flow
which uses DaskTaskRunnerMichael Holvey
09/29/2022, 2:32 AMfrom prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command
@flow
def run_dbt() -> str:
result = trigger_dbt_cli_command("dbt run", project_dir='C:\\Users\\36350admin\\arrow_dbt-1')
return result
run_dbt()
It even tells me it's running this dbt command, and when I run that by itself in the same working directory, it works as expected!
22:29:08.084 | INFO | Task run 'trigger_dbt_cli_command-321ca940-0' - Running dbt command: dbt run --profiles-dir C:\Users\36350admin\.dbt --project-dir C:\Users\36350admin\arrow_dbt-1
marque
09/29/2022, 5:56 AMAndreas Nord
09/29/2022, 9:24 AMVadym Dytyniak
09/29/2022, 10:27 AMIgor Kotua
09/29/2022, 10:51 AMDavid
09/29/2022, 11:29 AMPedro Henrique
09/29/2022, 1:08 PMEmma Rizzi
09/29/2022, 1:25 PMRudy García
09/29/2022, 1:38 PMNic
09/29/2022, 1:42 PMChing
09/29/2022, 2:06 PMAniruddha Bharadwaj
09/29/2022, 2:20 PMFile "/usr/local/lib/python3.9/site-packages/prefect_email/credentials.py", line 138, in get_server
server = SMTP_SSL(smtp_server, smtp_port, context=context)
File "/usr/lib64/python3.9/smtplib.py", line 1050, in __init__
SMTP.__init__(self, host, port, local_hostname, timeout,
File "/usr/lib64/python3.9/smtplib.py", line 255, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib64/python3.9/smtplib.py", line 341, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib64/python3.9/smtplib.py", line 1057, in _get_socket
new_socket = self.context.wrap_socket(new_socket,
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
Slackbot
09/29/2022, 3:32 PMDavid Cupp
09/29/2022, 4:37 PM2.4.0
installed locally.
For example, if I create the deployments Foo(customer=1)
, Foo(customer=2)
and Foo(customer=3
all under the flow name Foo
, I can go to my local UI, clicked on "Flows" and then "Foo", and I see those deployments.
But in the Prefect Cloud 2.0 UI, if I go to "Flows" and click on "Foo", no deployments appear! They do appear if I click on deployments.
Is this the expected behavior of the UI? (i.e. is there some criteria for a flow deployment to appear on the flow page?)Joe
09/29/2022, 5:34 PMEsdras Lopes Nani
09/29/2022, 5:43 PMTask run 'b' - Completed()
. If I force task c to be async runs without a problem. Is that a current bug? Using Prefect 2.3.1
@task
async def a():
# do async
@task
async def b():
# do async
@task
def c():
# do sync
@flow
async def main():
a = await a()
b = await b(wait_for=[a])
c = c(wait_for=[b])
asyncio.run(main())
Thanks!Nic
09/29/2022, 5:46 PMNace Plesko
09/29/2022, 5:51 PMNace Plesko
09/29/2022, 5:51 PMBianca Hoch
09/29/2022, 7:27 PM{
task_run(
where: {
created: {_lt: "2022-09-29 00:00:00", _gte: "2022-09-01 00:00:00"},
state: {_eq: "Running"},
}
) {
id
name
flow_run_id
task {
name
id
}
}
}
mutation setTaskRunStates($input: [set_task_run_state_input!]!) {
set_task_run_states(input: {states: $input}) {
states {
id
status
message
__typename
}
__typename
}
}
{
"input": {
"task_run_id": "task_runid",
"version": 1,
"state": {
"type": "Cancelled",
"message": "marked task run as Cancelled because \"test\""
}
}
}
Nace Plesko
09/29/2022, 9:19 PMBianca Hoch
09/30/2022, 2:52 PMNace Plesko
09/30/2022, 3:05 PMBianca Hoch
09/30/2022, 3:06 PMNace Plesko
09/30/2022, 3:13 PMBianca Hoch
09/30/2022, 4:41 PMNace Plesko
09/30/2022, 4:42 PMdb_shell_task
on the screenshot is ShellTask
Does anything on the screenshot below look obviously wrong? Again, I apologize for my lack of knowledge of PrefectShellTask
doesn't seem to do anything. ShellTask doesn't seem to have Duration and the mapped child runs are also not affected by that timeout setting.
Is there a way to set timeout on mapped children from ShellTask
?