An Hoang
04/17/2020, 2:50 PMAn Hoang
04/17/2020, 2:51 PMrmax
04/17/2020, 4:02 PMJohn Ramirez
04/17/2020, 4:08 PMbatch_run_date = Parameter(
name='batch_run_date',
default=(datetime.today() - timedelta(days=1)).strftime('%Y-%m-%d')
)
in perfect 0.9.*
this would alter the default value based on the current date. But in prefect 0.10.1
the default date is fixed. Is there a a different way I can reproduce the previous functionalityAn Hoang
04/17/2020, 4:27 PMtiz.io
04/17/2020, 5:33 PMtiz.io
04/17/2020, 5:51 PMqizzle
04/17/2020, 6:27 PMqizzle
04/17/2020, 6:27 PMqizzle
04/17/2020, 6:46 PMasm
04/17/2020, 6:53 PMasm
04/17/2020, 6:54 PMasm
04/17/2020, 6:54 PMDaskExecutor
if that helpsMatthew Maldonado
04/17/2020, 8:31 PMAndrew Vaccaro
04/17/2020, 10:06 PMupstream_tasks
to the same task being called twice with different arguments. I have something like
A(abcd)
B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)
A(defg)
B.set_dependencies(upstream_tasks=[A], keyword_tasks=...)
where B must run after A, and each gets called with two separate sets of Parameters. If I call B.set_dependencies(upstream_tasks=[A])
twice, will the second B wait on the second A? And is this the idiomatic way to do this?Matthew Perry
04/17/2020, 11:09 PMBrad
04/18/2020, 1:48 AMtiz.io
04/18/2020, 2:14 AMtiz.io
04/18/2020, 3:01 AMAlexey Volochenko
04/18/2020, 8:31 PMScott Zelenka
04/18/2020, 9:43 PMHui Zheng
04/19/2020, 8:50 PMMikael
04/20/2020, 7:00 AMAn Hoang
04/20/2020, 11:53 AMbsub
or lsub
command into a prefect task. Basically how do you use prefect to submit jobs to a compute cluster that has no automatic callback API for job completion and you have to ping it continuously to check the status? How do you check if the HPC job is complete? Do you parse the output of the HPC’s status command, check existence of an output file or something else?Romain
04/20/2020, 1:39 PMMartin Guindon
04/20/2020, 2:24 PMMatias Godoy
04/20/2020, 3:27 PMretries
option with the timeout
?
Let me explain better: Inside my task I am polling an API waiting for something to finish. If one of those HTTP calls fail, I'd like to retry, but if it takes too long and that something hasn't finished, I want the entire task to timeout.
Is that how combining those two options work? I'm afraid that a timeout would cause a retry and start all over again.Christian
04/20/2020, 5:00 PMprefect core server
. I have an instance of prefect server () and an agent running on a remote machine (via a VPN connection). Then I set
export PREFECT__CLOUD__API=<http://myserver.edu:4200>
Finally, I run my dummy flow which contains a flow.register() at the end...
The output on the command is:
Result Handler check: OK │
Flow: <http://localhost:8080/flow/17b029f0-fb9d-4ac5-bc54-fc4102a44631>
Is this OK? The task does not show up on the UI. Anything I'm missing? Do I have to configure something else for this?
My local laptops' diagnostic:
{
"config_overrides": {},
"env_vars": [
"PREFECT__CLOUD__API"
],
"system_information": {
"platform": "macOS-10.14.6-x86_64-i386-64bit",
"prefect_version": "0.10.2",
"python_version": "3.8.2"
}
}
And the servers' diagnostic:
{
"config_overrides": {},
"env_vars": [],
"system_information": {
"platform": "Linux-5.3.0-40-generic-x86_64-with-glibc2.29",
"prefect_version": "0.10.2",
"python_version": "3.8.2"
}
}
Cheers,
CChristian
04/20/2020, 6:18 PMViv Ian
04/20/2020, 6:20 PM