Jean
09/28/2022, 11:10 PM23:04:54.057 | INFO | prefect.engine - Created flow run 'optimal-snail' for flow 'test1'
23:04:54.057 | INFO | prefect.task_runner.dask - Creating a new Dask cluster with `distributed.deploy.local.LocalCluster`
23:04:55.980 | INFO | prefect.task_runner.dask - The Dask dashboard is available at <http://127.0.0.1:8787/status>
23:05:00.371 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-0' for task 'Execute values of the query'
23:05:00.372 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-0' immediately...
23:05:24.612 | INFO | Task run 'Execute values of the query-8165e3c8-0' - Finished in state Completed()
23:05:26.370 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-1' for task 'Execute values of the query'
23:05:26.370 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-1' immediately...
23:05:42.533 | INFO | Task run 'Execute values of the query-8165e3c8-1' - Finished in state Completed()
23:05:44.295 | INFO | Flow run 'optimal-snail' - Created task run 'Execute values of the query-8165e3c8-2' for task 'Execute values of the query'
23:05:44.296 | INFO | Flow run 'optimal-snail' - Executing 'Execute values of the query-8165e3c8-2' immediately...
23:06:09.538 | INFO | Task run 'Execute values of the query-8165e3c8-2' - Finished in state Completed()
As you see in these logs it’s not really spawning more tasks. I was under the impression that the call to a function with the @task
decorator would be non-blocking if made within a flow @flow
which uses DaskTaskRunnerMichael Holvey
09/29/2022, 2:32 AMfrom prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command
@flow
def run_dbt() -> str:
result = trigger_dbt_cli_command("dbt run", project_dir='C:\\Users\\36350admin\\arrow_dbt-1')
return result
run_dbt()
It even tells me it's running this dbt command, and when I run that by itself in the same working directory, it works as expected!
22:29:08.084 | INFO | Task run 'trigger_dbt_cli_command-321ca940-0' - Running dbt command: dbt run --profiles-dir C:\Users\36350admin\.dbt --project-dir C:\Users\36350admin\arrow_dbt-1
marque
09/29/2022, 5:56 AMAndreas Nord
09/29/2022, 9:24 AMVadym Dytyniak
09/29/2022, 10:27 AMIgor Kotua
09/29/2022, 10:51 AMDavid
09/29/2022, 11:29 AMPedro Henrique
09/29/2022, 1:08 PMEmma Rizzi
09/29/2022, 1:25 PMRudy García
09/29/2022, 1:38 PMNic
09/29/2022, 1:42 PMChing
09/29/2022, 2:06 PMAniruddha Bharadwaj
09/29/2022, 2:20 PMFile "/usr/local/lib/python3.9/site-packages/prefect_email/credentials.py", line 138, in get_server
server = SMTP_SSL(smtp_server, smtp_port, context=context)
File "/usr/lib64/python3.9/smtplib.py", line 1050, in __init__
SMTP.__init__(self, host, port, local_hostname, timeout,
File "/usr/lib64/python3.9/smtplib.py", line 255, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib64/python3.9/smtplib.py", line 341, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib64/python3.9/smtplib.py", line 1057, in _get_socket
new_socket = self.context.wrap_socket(new_socket,
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
Slackbot
09/29/2022, 3:32 PMDavid Cupp
09/29/2022, 4:37 PM2.4.0
installed locally.
For example, if I create the deployments Foo(customer=1)
, Foo(customer=2)
and Foo(customer=3
all under the flow name Foo
, I can go to my local UI, clicked on "Flows" and then "Foo", and I see those deployments.
But in the Prefect Cloud 2.0 UI, if I go to "Flows" and click on "Foo", no deployments appear! They do appear if I click on deployments.
Is this the expected behavior of the UI? (i.e. is there some criteria for a flow deployment to appear on the flow page?)Joe
09/29/2022, 5:34 PMEsdras Lopes Nani
09/29/2022, 5:43 PMTask run 'b' - Completed()
. If I force task c to be async runs without a problem. Is that a current bug? Using Prefect 2.3.1
@task
async def a():
# do async
@task
async def b():
# do async
@task
def c():
# do sync
@flow
async def main():
a = await a()
b = await b(wait_for=[a])
c = c(wait_for=[b])
asyncio.run(main())
Thanks!Nic
09/29/2022, 5:46 PMNace Plesko
09/29/2022, 5:51 PMTony Piazza
09/29/2022, 6:19 PM2022-09-29 11:50:53 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:53 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:59 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:51:05 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:51:05 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
The simple flow is decorated as follows:
@flow(
description = 'Calculates revenues for wells associated with a specific forecast',
task_runner = CoiledTaskRunner(config_name=WELL_REVENUE_CONFIG)
)
def forecast_well_revenues(request: ForecastJobRequest) -> None:
# invoke tasks here
Can someone explain why this is happening?Joe
09/29/2022, 7:39 PMDeployment.build_from_flow
requires a .prefectignore
file? Currently have all of those flows in a pip installable python module, would be nice to be able to override that rather than need to skip upload in the build, and then separate calls to apply()
and upload_to_storage(...)
.Ahmed Ezzat
09/29/2022, 8:05 PMJosh Paulin
09/29/2022, 9:11 PMAlix Cook
09/29/2022, 9:40 PMJames Constable
09/29/2022, 11:06 PM2022-09-29T21:22:51.649459194Z 21:22:51.649 | INFO | prefect.agent - Completed submission of flow run '78a45bb6-5ee3-4d6b-9a34-bf6ca56e496c'
2022-09-29T21:30:52.133263521Z 21:30:52.133 | INFO | prefect.agent - Submitting flow run 'c716e857-eba0-432a-8561-3f622cff080a'
2022-09-29T21:30:52.859579518Z 21:30:52.859 | INFO | prefect.infrastructure.process - Opening process 'scrupulous-collie'...
2022-09-29T21:30:52.860369014Z 21:30:52.859 | ERROR | prefect.agent - Failed to submit flow run 'c716e857-eba0-432a-8561-3f622cff080a' to infrastructure.
2022-09-29T21:30:52.860389714Z Traceback (most recent call last):
2022-09-29T21:30:52.860399114Z File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 233, in _submit_run_and_capture_errors
2022-09-29T21:30:52.860407714Z result = await infrastructure.run(task_status=task_status)
2022-09-29T21:30:52.860461413Z File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/process.py", line 71, in run
2022-09-29T21:30:52.860471313Z with tempfile.TemporaryDirectory(suffix="prefect") as tmp_dir:
2022-09-29T21:30:52.860480013Z File "/usr/local/lib/python3.10/tempfile.py", line 819, in __init__
2022-09-29T21:30:52.860488513Z self.name = mkdtemp(suffix, prefix, dir)
2022-09-29T21:30:52.860497213Z File "/usr/local/lib/python3.10/tempfile.py", line 357, in mkdtemp
2022-09-29T21:30:52.860526313Z prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
2022-09-29T21:30:52.860534613Z File "/usr/local/lib/python3.10/tempfile.py", line 126, in _sanitize_params
2022-09-29T21:30:52.860543013Z dir = gettempdir()
2022-09-29T21:30:52.860551213Z File "/usr/local/lib/python3.10/tempfile.py", line 299, in gettempdir
2022-09-29T21:30:52.860558913Z return _os.fsdecode(_gettempdir())
2022-09-29T21:30:52.860567613Z File "/usr/local/lib/python3.10/tempfile.py", line 292, in _gettempdir
2022-09-29T21:30:52.860575813Z tempdir = _get_default_tempdir()
2022-09-29T21:30:52.860584713Z File "/usr/local/lib/python3.10/tempfile.py", line 223, in _get_default_tempdir
2022-09-29T21:30:52.860615012Z raise FileNotFoundError(_errno.ENOENT,
2022-09-29T21:30:52.860624512Z FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/opt/prefect']
2022-09-29T21:30:52.861818706Z 21:30:52.861 | INFO | prefect.agent - Completed submission of flow run 'c716e857-eba0-432a-8561-3f622cff080a'
James Constable
09/30/2022, 2:27 AMMalavika S Menon
09/30/2022, 10:06 AMTraceback (most recent call last):
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
self._rows = await prepared_stmt.fetch(*parameters)
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
data = await self.__bind_execute(args, 0, timeout)
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
data, status, _ = await self.__do_execute(
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
return await executor(protocol)
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncpg.exceptions.ActiveSQLTransactionError: ALTER TYPE ... ADD cannot run inside a transaction block
Coming across this error while trying to run a prefect agent and orion. Has anyone come across this before?Hamza Naanani
09/30/2022, 11:16 AMNic
09/30/2022, 11:30 AMusername
and will be set within the secret at the key {fullnameOverride}-postgresql-password
.
## This argument is only relevant when using the Postgres database included in the chart.
## For an external postgres connection, you must create and use existingSecret
instead of postgresqlPassword
.
# -- password for the custom user to create. Ignored if auth.existingSecret
with key password
is provided
password: passwordPedro Machado
09/30/2022, 12:30 PM