Marvin09/18/2020, 7:09 AM
Carl R09/18/2020, 7:15 AM
Marvin09/18/2020, 7:15 AM
Robin09/18/2020, 8:26 AM
and I got the following error:
/opt/prefect/healthcheck.py:149: UserWarning: Task <Task: copy_storage> has retry settings but some upstream dependencies do not have result types. See <https://docs.prefect.io/core/concepts/results.html> for more details. result_check(flows)
. I read the documentation about result types but that did not answer several questions: • What might go wrong if I don't set a result type? • If I understood correctly, the result type can be either
. I think I want that the result type is flow, since each task should be only run once (if successful) so the result does not change during the flow. Is that correct? • Should I therefore do something like
? And what should I set
with Flow("creative_name", result=flow_result)
to? Bests Robin
Jacob Blanco09/18/2020, 9:31 AM
Vinod Sugur09/18/2020, 1:42 PM
Marvin09/18/2020, 1:42 PM
Dolor Oculus09/18/2020, 4:21 PM
server launches ok, ui looks good, I can register a simple hello world flow,
prefect backend server prefect server start prefect agent start local -p .
Registering this flow results in
with Flow("Welcome Flow") as pf: print("Hello world") pf.register(project_name="Hello, World!")
But when I run the flow from the UI, I get this error message:
Hello world Result Handler check: OK Flow: <http://localhost:8080/flow/a5ec20aa-ad9e-4add-977e-cffe9699eba3>
Any thoughts? ty 🙂
[2020-09-18 16:20:35,115] ERROR - agent | Error while deploying flow: FileNotFoundError(2, "No such file or directory: 'prefect'")
Hamza Ahmed09/18/2020, 5:34 PM
the PostgresFetch initialization doesn't work when I try to use
from prefect import Flow, task from prefect.tasks.aws.secrets_manager import AWSSecretsManager from prefect.tasks.postgres.postgres import PostgresFetch @task def print_me(to_print): print(to_print) with Flow('fetching-data') as flow: credentials = AWSSecretsManager(secret='pg/prefectsql/prefect') print_me(credentials) pg_user = credentials['username'] pg_pass = credentials['password'] pg_host = 'localhost' pg_port = credentials['port'] pg_query = 'SELECT * FROM hdb_catalog.hdb_table LIMIT 5;' runsql = PostgresFetch(db_name='prefect', user=pg_user, host='localhost') result = runsql(password=pg_pass, query=pg_query, fetch='all') print_me(result) flow.run()
, but it does when I hardcode the username, or even if I set
to the a string containing the username The error flow run produces the below output:
matt forbes09/18/2020, 5:52 PM
jars09/18/2020, 8:12 PM
Minakshi09/18/2020, 8:43 PM
I found a resolution for that online namely trying to use the latest version of daks and distributed https://github.com/dask/distributed/issues/3331. But problem here is that i need to add these to our internal repo. So i need to confirm what all are the dependent packages with versions so that i can check or add all at once as required. Where can i get the list for that?
ModuleNotFoundError: No module named 'dask.system'
Dolor Oculus09/18/2020, 9:33 PM
syntax? I'm getting key not found results on asserting state.result[e], and wondering if you have to do it in the way given in the testing-flows link above to unit test the flows.
with Flow() as flow: e = Extract() t = Transform(e) l = Load(t) state = flow.run()
Glen Trudeau09/18/2020, 10:14 PM
when running a flow on Fargate agent which I assume is due to the fact that the base docker image doesn’t have that module installed. But when we tried downloading the public image, adding the necessary modules and then uploading to ECR it doesn’t appear that the agent is utilizing the updated image even though we called it inside of the container definitions (reference python wrapper). We see the below screenshot for the new task definition in ECS which means it isn’t pulling the specified image correctly. Any suggestions?
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'sqlalchemy'")
CA Lee09/19/2020, 11:08 AM
takes in a
arg (below is some sample code):
This all works fine when I am running it on my server using
CWD = os.path.dirname(os.path.abspath(__file__)) # Get CWD file_path = os.path.join(CWD, "picture.png") # Save in CWD plt.savefig(file_path)
, however, when it is registered on cloud, invoking a Quick Run results in the error :
The error shows that
PermissionError: [Errno 13] Permission denied: '/picture.png'
is unable to be parsed to the current working directory that the script is being run from when triggered from Cloud. Would appreciate any insight.. if it helps, I am getting data from Google Sheets, plotting it and sending it to Slack, so I'd like to Prefect this workflow.
Eric09/20/2020, 1:58 AM
Eric09/20/2020, 1:58 AM
Eric09/20/2020, 1:59 AM
CA Lee09/20/2020, 9:54 AM
@task(cache_for=datetime.timedelta(days=1)) def fetch_data(): return fetched_data @task def insert_into_database(fetched_data): fetched_data.to_sql('table_name', con=db.engine)
itay livni09/20/2020, 12:24 PM
Anshaj09/20/2020, 1:10 PM
Marvin09/20/2020, 1:10 PM
Jeff Brainerd09/20/2020, 2:48 PM
with heartbeats off and Lazarus on. Is this a known issue? Should Lazarus be picking these up? Thanks!
Eric09/20/2020, 5:33 PM
Eric09/20/2020, 5:34 PM
josh09/20/2020, 10:41 PM
has been released and here are a few notable changes: 🛠️ Fixes, lots of fixes 🗜️ Data compression for S3 upload/download tasks 🤫 Quieter debug logs when starting server A big thank you to our contributors who helped out with this release! Full changelog:
Pedro Machado09/21/2020, 12:45 AM
. If these are not provided, I want to use the
from the context to generate the
. What is the cleanest way to write this logic given the fact that the parameters will only be available at run time? I currently have a task that takes both parameters and outputs a tuple of dates. This tuple has either the value of the parameters converted to a
date or the values generated from the
also as dates. Is there a better way? This is what the flow looks like now.
CA Lee09/21/2020, 1:39 AM
sark09/21/2020, 9:21 AM
hmm i am unable to set the executor for a flow like this https://docs.prefect.io/api/latest/core/flow.html#flow-2 says that it does have the
Traceback (most recent call last): File "daily.py", line 95, in <module> with Flow('start dataflow job 1', executor=LocalDaskExecutor(), storage=GCS('ghpr-dev-prefect-flows')) as startFlow1: TypeError: __init__() got an unexpected keyword argument 'executor' (prefect) bash-3.2$ prefect version 0.13.7
parameter though i am trying to increase parallelism for my flows
Greg Roche09/21/2020, 9:27 AM
). If I have two separate agents running on the machine, each started from within their own venv, they can both execute their own flows successfully. As soon as I try to cut it down to one agent listening for both flows, the agent fails to execute any flows which live inside another venv (
). I'd like to keep the separated venvs if possible and just have one agent running, which would execute each of the flows within the context of their own venv. Is this possible, or would I need to bite the bullet and have one venv for both flows?
Failed to load and execute Flow's environment: ModuleNotFoundError