Jeff Brainerd
03/26/2020, 6:43 PMquery {
flow_run {
name
parameters(path: "company_slug")
}
}
Scott Zelenka
03/26/2020, 8:19 PMschedule=Schedule(
clocks=[
IntervalClock(
start_date=pendulum.datetime(2020, 1, 1, tz='UTC'),
interval=datetime.timedelta(days=1),
parameter_defaults=dict(
sfdc_listener_name='CustomerService',
sfdc_report_url='...'
)
),
IntervalClock(
start_date=pendulum.datetime(2020, 1, 1, tz='UTC'),
interval=datetime.timedelta(days=1),
parameter_defaults=dict(
sfdc_listener_name='OneSupport',
sfdc_report_url='...'
)
)
]
),
However, it seems that when this registered with Prefect Cloud, only the first Schedule is setup.
Do I need to create two different Flows to have the same logic execute with different parameters on a schedule?Ben Fogelson
03/26/2020, 11:02 PMParameter
whose default is the value of another Parameter
? Something like
from prefect import task, Flow, Parameter
@task
def add(x, y):
return x + y
with Flow('flow') as flow:
a = Parameter('a', default=0)
b = Parameter('b', default=a)
c = add(a, b)
Christopher
03/27/2020, 8:35 AMdherincx
03/27/2020, 7:39 PMswitch
. Is it possible to access the results of the switch in a subsequent task? For example, I have a list of data (some of which contain latitude/longitude and others don't). My switch consists of getting lat/long for the records that are missing it, while records that do have coordinates are simply returned. How can I access the final, joined list after the switch?Pierre CORBEL
03/27/2020, 10:24 PMScott Zelenka
03/27/2020, 11:16 PMselenium/standalone-chrome
. But when I try to build through the CLI, it's giving me trouble (essentially spun the CPU fan for an hour, and eventually gave up because
Traceback (most recent call last):=============================>] 143.3MB/143.3MB
File "example-selenium.py", line 473, in <module>
parameters=dict(
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/core/flow.py", line 1419, in register
no_url=no_url,
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/client/client.py", line 623, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/core/flow.py", line 1228, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 282, in build
self._build_image(push=push)
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 312, in _build_image
self.pull_image()
File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 520, in pull_image
raise InterruptedError(line.get("error"))
InterruptedError: write /var/lib/docker/tmp/GetImageBlob079036145: no space left on device
Pierre CORBEL
03/29/2020, 3:26 PMCronClock
combine to a launch at startup.
Is there an easy way to achieve this? 🧐
I think the fact that the scheduler wait for the next schedule raise the need for a "fire_at_start" parameter.
In my case, if my schedule is programmed every day at 3 o'clock, I don't want to wait X hours before my DB is loaded up with data. 🕒
Does it make sense?Scott Zelenka
03/29/2020, 5:18 PM.register
generates.
I can get the dependent application(s) to run inside a Docker image with supervisord, but that would require the entrypoint of Docker to somehow trigger supervisord before it attempts to execute the Flow. I couldn't find an API in Prefect to allow for this type of functionality?
Another option is to provide the dependent application(s) as a native cloud service, and communicate over the network to access the same way I do it on my local dev machine.. but that'd be overkill for this one Flow.
Curious if anyone has been successful in wrapping another (non-Python) application inside a serialized Flow to deploy on Prefect Cloud.Chris O'Brien
03/29/2020, 10:38 PMMohit Kumar Agarwal
03/30/2020, 11:39 AMJoe Schmid
03/30/2020, 2:03 PMpip install -U prefect
then prefect server start
and brought up the UI in a browser without any trouble.
This is a huge milestone and one that the community will benefit from greatly. Anyone can now run the fully open source Prefect Core and have a UI to manage and monitor flows. This gives folks a greater spectrum of options for running Prefect, from fully open source and no cost with Core to commercial with additional features with Cloud. Great work to the Prefect team!Bob Colner
03/30/2020, 4:06 PMScott Zelenka
03/30/2020, 4:08 PMflow.register()
and it gets orchestrated to run in Cloud, it fails with the result_handler
. I'd like to simulate running the serialized flow within the Docker image it generated locally, to see if I can figure out what's different between that environment and my local machine.
Is there a proper way to do this?Leo Meyerovich (Graphistry)
03/30/2020, 5:15 PMMike Lutz
03/30/2020, 6:36 PMManuel Aristarán
03/30/2020, 6:39 PMprefect server?
I have an instance of postgres running on the default port…Preston Marshall
03/30/2020, 6:47 PMArkady Kleyner
03/30/2020, 6:59 PMKamil Okáč
03/31/2020, 9:22 AMvue-apollo.js:14 Uncaught TypeError: Cannot read property 'substr' of undefined
at Module.56d7 (vue-apollo.js:14)
at c (bootstrap:89)
at Object.0 (app.e76c6ad2.js:1)
at c (bootstrap:89)
at t (bootstrap:45)
at bootstrap:267
at app.e76c6ad2.js:1
John Faucett
03/31/2020, 1:31 PMflow = Flow('foo', tasks=[t1,t2,t3, t4])
flow.add_edge(t1,t2, key='x')
flow.add_edge(t2,t3)
flow.add_edge(t3,t4)
# now t4 needs the results of t1 not t2 or t3
Preston Marshall
03/31/2020, 2:28 PMJeff Brainerd
03/31/2020, 4:11 PMFabian Thomas
03/31/2020, 4:17 PMprefect agent start
gives me this exception: prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
Any suggestions? I had already registered an agent with Prefect Cloud and an API token before.Benjamin Filippi
03/31/2020, 4:19 PMBen Fogelson
03/31/2020, 4:22 PMThomas La Piana
03/31/2020, 5:27 PMMaxime Lavoie
03/31/2020, 5:54 PMwith Flow("My Flow") as flow:
my_param = Parameter("my_param", default='a default value')
...
flow.run()
# flow.run(parameters={"my_param": "overwritten value"})
When I run this, I get
raise ValueError(
ValueError: Flow.run received the following unexpected parameters: my_param
What am I missing?David N
03/31/2020, 7:33 PMResult Handler check: OK
/home/ec2-user/venv/prefect/lib64/python3.7/site-packages/prefect/core/task.py:258: UserWarning: DEPRECATED: all cache_* options on a Task will be deprecated in 0.11.0, and removed in 0.12.0; the options will be moved to a Task's prefect.engine.Result object.
UserWarning,
Flow: <http://localhost:8080/flow/ef63906a-6e2b-4b7b-aba5-6e9303560e5f>
But I dont see anything changing in the UI, and dont see anything under the "Flows" link. I have this running on an ec2 machine so I dont use "localhost", but can reach the UI.. Just looks like it knows nothing about the flow. What have I missed?Cab Maddux
03/31/2020, 8:49 PM