Igor Morgunov
10/04/2022, 3:54 PMAndrei Tulbure
10/04/2022, 4:14 PMGarren Moore
10/04/2022, 4:59 PMdatabricks
currently supported as a connection/profile type when using prefect-dbt cli commands (like trigger_dbt_cli_command
?
RuntimeError: Command failed with exit code 1:
Could not find adapter type databricks!
Philip MacMenamin
10/04/2022, 6:12 PMSchuyler Duveen
10/04/2022, 6:38 PMSean Turner
10/04/2022, 7:50 PMKubernetesJob
infra blocks in a Deployment
?
e.g.
flow a
task b (uses KubernetesJob with docker image foo)
task c (uses KubernetesJob with docker image bar)
It kind of seems like I'm limited to to having one KubernetesJob
per Deployment
?Bradley McLaughlin
10/04/2022, 8:23 PMYZ
10/04/2022, 9:43 PMmy_task(run_date="20221004")
retrying run: my_task(run_date=None)
Any idea what I might have done wrong, or any workaround?
@task(retry_delay=timedelta(minutes=30), max_retries=10) # wait for up to 5 hours
def my_task(run_date: str):
Sander
10/04/2022, 9:47 PMWill Hayes
10/04/2022, 10:25 PMTan Li
10/04/2022, 11:50 PMSecret.load("github-access-token").get()
in normal python scripts?Jelle Vegter
10/05/2022, 7:32 AMHa Pham
10/05/2022, 8:02 AMHieu Tran
10/05/2022, 8:35 AMtimeout_seconds
of the flow as parameter to customize it when trigger a deployment of that flow manually from UI?
@flow(timeout_seconds = 15)
def log_flow(name: str):
log_task(name)
As timeout_seconds is in the flow decorator, it won't be shown as a parameter of the flow runErik
10/05/2022, 10:36 AMprefect agent
are:
╭─ Options ──────────────────────────────────────────────────────────────────────
│ --work-queue -q TEXT One or more work queue names for the agent to pull from. [default: None] │
│ --hide-welcome │
│ --api TEXT [default: (from PREFECT_API_URL)] │
│ --tag -t TEXT DEPRECATED: One or more optional tags that will be used to create a work queue [default: None] │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────
What I would like to do is launch an agent once, grab the flow_runs that are sitting in the queue (a single call), once those are completed, the agent would close itself. Is there any kind of option for that?Steph Clacksman
10/05/2022, 10:57 AMTypeError: Dict key must be str
but the other doesn't. The one causing the error does have ~1 million rows though.Q
10/05/2022, 12:42 PMprojname/flows/myflow.py
imports projname.utils.utilfunc
).
This flow is scheduled from a deployment and uses Process
as its infrastructure block.
I would like to ensure that the latest version of projname
is installed (to guarantee that utilfunc
is up-to-date), so I can't install the package when setting up agent environment.
I think I can install the package before running the flow if I set Process.command
to something like ["pip", "install", ".", "&&", "python3", "-m", "prefect.engine"]
, but maybe there's a better way to go about it?Aaron Lanzen
10/05/2022, 1:07 PMGarren Moore
10/05/2022, 2:49 PMERROR prefect.infrastructure.process - Process 'turquoise-woodlouse' exited with status code: -9
No more messaging beyond that.
In Prefect Cloud UI, the flow runs indefinitely and doesn’t show this log message (level set to DEBUG).
Anyone familiar with exit code -9? Can’t find any docs.Michael Shoemaker
10/05/2022, 2:58 PMImportError while loading conftest '/home/gary/prefect/tests/conftest.py'.
tests/conftest.py:32: in <module>
import prefect
src/prefect/__init__.py:23: in <module>
from prefect.flows import flow, Flow
src/prefect/flows.py:36: in <module>
from prefect.context import PrefectObjectRegistry, registry_from_script
src/prefect/context.py:34: in <module>
from prefect.client import OrionClient
src/prefect/client.py:55: in <module>
from prefect.orion.api.server import ORION_API_VERSION, create_app
src/prefect/orion/api/__init__.py:1: in <module>
from . import (
src/prefect/orion/api/block_documents.py:19: in <module>
from prefect.orion import models, schemas
src/prefect/orion/models/__init__.py:1: in <module>
from . import (
src/prefect/orion/models/block_schemas.py:12: in <module>
from prefect.blocks.core import Block
src/prefect/blocks/__init__.py:3: in <module>
import prefect.blocks.notifications
src/prefect/blocks/notifications.py:26: in <module>
class SlackWebhook(NotificationBlock):
pydantic/main.py:283: in pydantic.main.ModelMetaclass.__new__
???
../anaconda3/lib/python3.9/abc.py:106: in __new__
cls = super().__new__(mcls, name, bases, namespace, **kwargs)
src/prefect/utilities/dispatch.py:99: in _register_subclass_of_base_type
register_type(cls)
src/prefect/utilities/dispatch.py:154: in register_type
key = get_dispatch_key(cls)
src/prefect/utilities/dispatch.py:76: in get_dispatch_key
dispatch_key = dispatch_key()
src/prefect/blocks/core.py:127: in __dispatch_key__
return block_schema_to_key(cls._to_block_schema())
src/prefect/blocks/core.py:263: in _to_block_schema
return BlockSchema(
pydantic/main.py:340: in pydantic.main.BaseModel.__init__
???
pydantic/main.py:1076: in pydantic.main.validate_model
???
pydantic/fields.py:884: in pydantic.fields.ModelField.validate
???
pydantic/fields.py:1101: in pydantic.fields.ModelField._validate_singleton
???
pydantic/fields.py:1148: in pydantic.fields.ModelField._apply_validators
???
pydantic/class_validators.py:318: in pydantic.class_validators._generic_validator_basic.lambda13
???
pydantic/main.py:690: in pydantic.main.BaseModel.validate
???
E DeprecationWarning: `copy_on_model_validation` should be a string: 'deep', 'shallow' or 'none'
Kunal Tyagi
10/05/2022, 3:21 PMJason Bertman
10/05/2022, 4:54 PMKyle D
10/05/2022, 5:15 PMcreate_flow_run()
in prefect 2? I’m trying to see how the flow
parameter should be setupNathan R
10/05/2022, 6:42 PMNick McGoye
10/05/2022, 6:57 PMNace Plesko
10/05/2022, 7:37 PMShellTask
which is written in Typescript. As part of the typescript script we call python script. The prefect timeout works if it's reached while typescript is executing, but the timeout doesn't work if the process is stuck inside the python script.
Has anyone ran into the same issue before in the past? Any advice is appreciated, thank you in advance!Norman
10/05/2022, 8:12 PMJarvis Stubblefield
10/05/2022, 8:51 PMprefect agent
to execute within my Django environment (using Django ORM in the flow). I finally got the deployment created by adding my project to the sys path before running the Django setup. That works for running the flow manually, but now I’m getting another error when the Agent tries to run the flow. Full error in the thread message. So my idea at this point is to create a Django Management command (which executes in the Django environment) to run the agent through Python code. I have an example of doing this in Prefect v1, but I’m using Prefect v2 and haven’t see where this is clearly documented. LocalAgent
no longer seems to exist in Prefect v2 to run an agent through code. Any ideas of other ways to run this or how to run the agent from within Python would be super helpful and wholly appreciated!Adam
10/05/2022, 9:51 PMEmon Li
10/06/2022, 12:22 AM