Justin Martin
03/07/2022, 2:34 PM@task
decorator? Here is an extremely simplified version of what i'm doing (See thread).
Am I able to define the run_load_proc function as just a FUnctionTask within the flow without an explicit function? Also, not sure if defining the global SqlServerExecute is a total anti-pattern, let me know. Thanks for all of your help.Haseeb Ahmad
03/07/2022, 2:35 PMInspecting flow runs
from prefect.backend import FlowRunView
flow_run = FlowRunView.from_flow_run_id("4c0101af-c6bb-4b96-8661-63a5bbfb5596")
Can you please guide me on the best approach to achieve the above use case. Really appreciate your time and help.Sarah Floris
03/07/2022, 2:42 PMKevin Kho
Daniel Nilsen
03/07/2022, 4:26 PMflow.register(«myProject»)
. To connect to the server I change the server.endpoint
in config.toml to the correct url. This works. But I don’t want to change the config manualy like this. Is there a way to temporarily change it when registering? I have tried set_temporary_config({"server.endpoint": "123"})
but it does not seem to work 🤔Zhibin Dai
03/07/2022, 4:34 PMwith case
statements, or can i use just one that does something if a condition is true?Jason Motley
03/07/2022, 4:59 PM: No heartbeat detected from the remote task;
Paul Butler
03/07/2022, 5:01 PMPaul Butler
03/07/2022, 5:33 PMQuestion for the Community - I am trying to use Prefect to schedule and monitor a Dbt project/pipeline. I follow example and pass in dbt_kwargs for connecting to snowflake. My dbt project runs fine with dbt run or dbt compile command, but the DbtShellTask fails - yet does not provide any reason for error. Anywhere more detailed logging is recorded??
Sample tasks like the hello one included here, run OK. I'm using Studio Code to debug this code, but also try to run it in Python IDLE and get same error output I'm running this:
from prefect import task, Flow, Parameter
from prefect.tasks.shell import ShellTask
from prefect.tasks.dbt import DbtShellTask
@task(log_stdout=True)
def say_hi(name):
print("hello {}!".format(name))
with Flow(name="dbt_flow") as f:
name = Parameter('name')
say_hi(name)
task = DbtShellTask(
profile_name='default',
environment='dev',
dbt_kwargs={
'type': 'snowflake',
'threads': 4,
'account': 'mysnowflake.account',
'user': '<mailto:myemal@myco.com|myemal@myco.com>',
'authenticator': 'externalbrowser',
'role': 'ROLENAME',
'database': 'DBNAME',
'warehouse': 'ENGINEERING_XS',
'schema': 'DV_PROTO'
},
overwrite_profiles=False,
profiles_dir='C:\\Users\myDBTuser\.dbt'
)(command='dbt compile')
out = f.run(name='Paul')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2022-03-07 17:18:07+0000] ERROR - prefect.DbtShellTask | Command failed with exit code 1
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 1')
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Starting task run...
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | hello Paul!
[2022-03-07 17:18:07+0000] INFO - prefect.TaskRunner | Task 'say_hi': Finished task run for task with final state: 'Success'
Adam Roderick
03/07/2022, 6:06 PM...venv/lib/python3.8/site-packages/prefect/client/client.py:848: UserWarning: A flow with the same name is already contained in storage; if you changed your Flow since the last build, you might experience unexpected issues and should re-create your storage object.
I don't understand why we see that, because, we create a new Docker storage object on every deployment
flow.storage = Docker(....)
Can anyone explain what this error means, or how to address it?kevin
03/07/2022, 7:40 PMflow_run.duration
but I can’t seem to find that queryable value in the flow_run
fields on the interactive API in Prefect Cloud. Could someone help me resolve this differential?Pedro Machado
03/07/2022, 8:51 PMJean-Michel Provencher
03/07/2022, 9:20 PMprefecthq/prefect:1.0.0
and when scanning the docker image with Snyk I’m getting up to 114 security issues, as well as 8 critical vulnerabilities.
Are you guys planning on fixing them in the base docker image?Dylan
03/07/2022, 9:40 PMDylan
03/07/2022, 9:41 PMKelly Huang
03/07/2022, 11:27 PMin get_flow
from github import UnknownObjectException
ModuleNotFoundError: No module named 'github'
I'm reading differing things about whether or not github storage can access package dependencies? Correct me if I'm wrong, but shouldn't it definitely be able to access my Pipfile and therefore dependencies? Otherwise what would the point of github storage be?Andrew Moist
03/08/2022, 9:37 AMTomer Cagan
03/08/2022, 9:38 AMVadym Dytyniak
03/08/2022, 9:51 AMcheckpointing
works in prefect 1.0.0? In documentation I see that it is enabled by default, but what is the default Result
?Bruno Murino
03/08/2022, 12:48 PMTypeError: StartFlowRun.run() missing 1 required positional argument: 'self'
. I don’t know if I’m doing something wrong, but it’s a bit unclear what to doTom
03/08/2022, 1:06 PMTom
03/08/2022, 1:57 PMflow.run()
? (https://stackoverflow.com/questions/66660927/prefect-workflow-how-to-persist-data-of-previous-every-schedule-run)Thomas Opsomer
03/08/2022, 2:55 PMmanual_only
trigger), after being approved using the UI on prefect cloud, it takes a very long time before the flow is resumed (like several hours :/).
It's not the first time it happens, but currently it's happening on every flows.
Anyone having the same issue ? Is it an issue on prefect backend side or should we do something ?Marius Haberstock
03/08/2022, 3:18 PMapply_map
. Details following in the thread 🙂Max Kolasinski
03/08/2022, 3:49 PMRoger Webb
03/08/2022, 3:52 PMAdam Roderick
03/08/2022, 3:59 PMBruno Nunes
03/08/2022, 4:21 PMben
03/08/2022, 5:14 PMMartha Edwards
03/08/2022, 6:01 PM