Mark McDonald
04/02/2020, 8:11 PMManuel Aristarán
04/02/2020, 9:44 PMParameter
. My use case is building a shell command:
data_source_id = Parameter("data_source_id")
# ...
t = shell_task(command=f"some_script {data_source_id}")
Scott Zelenka
04/03/2020, 12:00 AMKamil Okáč
04/03/2020, 7:07 AMfrom prefect import Flow, task
@task
def add_ten(x):
return x + 10
with Flow('simple map') as flow:
mapped_result = add_ten.map([1, 2])
flow.register()
flow.run_agent()
When running the flow from UI, the task "add_ten (Parent)" is stuck in state "Mapped" (with description "Preparing to submit 2 mapped tasks"). What's wrong?David Ojeda
04/03/2020, 4:28 PMBenjamin Filippi
04/03/2020, 4:32 PMserver
or cloud
Options:
-h, --help Show this message and exit.Benjamin Filippi
04/03/2020, 4:32 PMChris Hart
04/03/2020, 4:45 PMBen Fogelson
04/03/2020, 6:34 PMx
?David Hogarty
04/03/2020, 6:38 PMDavid Hogarty
04/03/2020, 6:39 PMDavid Hogarty
04/03/2020, 6:39 PMDavid Hogarty
04/03/2020, 6:40 PMDavid Hogarty
04/03/2020, 6:40 PMDavid Hogarty
04/03/2020, 9:05 PMDavid Hogarty
04/03/2020, 9:05 PMPriya Rao
04/03/2020, 10:05 PMManuel Aristarán
04/03/2020, 11:33 PMprefect
typing information when importing the module?
mypy
is also complaining…Ricky
04/04/2020, 5:15 AMPierre CORBEL
04/04/2020, 5:01 PMPierre CORBEL
04/04/2020, 5:07 PMJeremiah
04/04/2020, 6:52 PMDavid Haines
04/05/2020, 6:24 AMIlay Gordon
04/05/2020, 12:28 PMParameter
. Which construct should I use? ifelse
doesn't seem fit for this, since I don't have another task for the false condition. In airflow it was solved using a DummyOperator
and it always looked somewhat redundant. Is there an idiomatic way with Prefect? UPDATE: nevermind, just read about the SKIP signal 🙂Darren Fleetwood
04/05/2020, 3:49 PMrequests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7739090a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
. I'm guessing it means the container can't find the graphql api? I'm running under Ubuntu. I'm ssh tunnelling into a jupyter notebook on an Azure VM and running from there - could that be making a difference? thanks!!Kaz
04/05/2020, 7:23 PMShellTask
doesn’t seem to work for me. I have an S3 sync that occasionally hang (working to figure that out), and as a temporary solution I was hoping to use a timeout + retry. This has solved the issue in the past. I’ve set the timeout to 600 (600 seconds, 10 minutes), and yet I’m getting a task that runs for 6 hours+ when the shell task hangs. On Cloud (Scheduler tier), I can see that my retry and retry delay parameters are set, but I’m not sure if there’s a way to check whether a timeout has been configured on a task. It’s a fairly simple task (top level, only 1 dependency, unmapped), and everything works just fine if I manually restart the task on the Cloud UI (I even see the retry count increment). I’m wondering if there are any additional inputs I needed to configure other than timeout + retry in order to get the timeout to work, or if this is an issue other people have seen before?. Thanks!Rugzo
04/06/2020, 2:47 PMlabels
Basically, some best practices around Agents in a production setting.
ThanksRichard Gu
04/06/2020, 5:08 PMliren zhang
04/06/2020, 6:35 PMpip install prefect
Building wheels for collected packages: pendulum
Building wheel for pendulum (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\zhang\appdata\local\programs\python\python38\python.exe' 'c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\zhang\AppData\Local\Temp\tmpjjs207ax'
cwd: C:\Users\zhang\AppData\Local\Temp\pip-install-2pd94jc4\pendulum
Complete output (24 lines):
Traceback (most recent call last):
File "setup.py", line 2, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'
Traceback (most recent call last):
File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 257, in <module>
main()
File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 240, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 181, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\api.py", line 57, in build_wheel
return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 56, in make_in
wb.build()
File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 82, in build
self._build(zip_file)
File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 102, in _build
self._run_build_command(setup)
File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 130, in _run_build_command
subprocess.check_call(
File "c:\users\zhang\appdata\local\programs\python\python38\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\users\\zhang\\appdata\\local\\programs\\python\\python38\\python.exe', 'setup.py', 'build', '-b', 'build']' returned non-zero exit status 1.
----------------------------------------
ERROR: Failed building wheel for pendulum
Failed to build pendulum
ERROR: Could not build wheels for pendulum which use PEP 517 and cannot be installed directly
Manuel Aristarán
04/06/2020, 6:41 PMManuel Aristarán
04/06/2020, 6:41 PMZachary Hughes
04/06/2020, 8:23 PMManuel Aristarán
04/06/2020, 9:56 PMZachary Hughes
04/07/2020, 2:13 AM