• Jeremiah

    Jeremiah

    2 years ago
    set the channel topic: Welcome to the Prefect community! Please use threads if possible so we can archive helpful conversations to our GitHub knowledge base: https://github.com/PrefectHQ/prefect/issues?q=label%3A%22Prefect+Slack+Community%22+
  • d

    David Haines

    2 years ago
    Have observed a significant difference in overall execution time between running a flow manually and running the same flow via a local agent on the same machine (triggered via prefect cloud). Consistently 20-25 times slower for this particular flow (lots of mapped tasks). Expected behaviour?
    d
    Chris White
    +1
    13 replies
    Copy to Clipboard
  • i

    Ilay Gordon

    2 years ago
    Hi guys, quick question - I would like to run a task conditioned by an optional
    Parameter
    . Which construct should I use?
    ifelse
    doesn't seem fit for this, since I don't have another task for the false condition. In airflow it was solved using a
    DummyOperator
    and it always looked somewhat redundant. Is there an idiomatic way with Prefect? UPDATE: nevermind, just read about the SKIP signal 🙂
    i
    1 replies
    Copy to Clipboard
  • d

    Darren Fleetwood

    2 years ago
    Hi all, first of all thanks for what looks like a great product so far! I'm trying to use the core server to submit jobs to the docker agent. I can register flows ok and when I run them they are picked up by the agent. But they sit at 'submitted'. Looking at the logs for the docker containers I get this:
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7739090a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    . I'm guessing it means the container can't find the graphql api? I'm running under Ubuntu. I'm ssh tunnelling into a jupyter notebook on an Azure VM and running from there - could that be making a difference? thanks!!
    d
    j
    +2
    21 replies
    Copy to Clipboard
  • k

    Kaz

    2 years ago
    Hey guys. Big fan of the product, it’s really cut down on our manual ETL work load. 1 issue: setting the time out in a
    ShellTask
    doesn’t seem to work for me. I have an S3 sync that occasionally hang (working to figure that out), and as a temporary solution I was hoping to use a timeout + retry. This has solved the issue in the past. I’ve set the timeout to 600 (600 seconds, 10 minutes), and yet I’m getting a task that runs for 6 hours+ when the shell task hangs. On Cloud (Scheduler tier), I can see that my retry and retry delay parameters are set, but I’m not sure if there’s a way to check whether a timeout has been configured on a task. It’s a fairly simple task (top level, only 1 dependency, unmapped), and everything works just fine if I manually restart the task on the Cloud UI (I even see the retry count increment). I’m wondering if there are any additional inputs I needed to configure other than timeout + retry in order to get the timeout to work, or if this is an issue other people have seen before?. Thanks!
    k
    Chris White
    3 replies
    Copy to Clipboard
  • Rugzo

    Rugzo

    2 years ago
    Hey there, first time here. Firstly, fantastic product you have built, seems to have massive potential, coming from someone who used to work with Airflow. The documentation has been super helpful and I would like to see more video demos in the future. My question is, what would you recommend would be the best way to run Agents in a production environment, say EC2, when using Prefect's Cloud feature? I mean: • should the Agent run continuously in the background of an EC2 instance? • should the Agent be embedded in the Python Flow script and run as a container • should we have several Agents running and associated to flows based on
    labels
    Basically, some best practices around Agents in a production setting. Thanks
    Rugzo
    1 replies
    Copy to Clipboard
  • r

    Richard Gu

    2 years ago
    Hello friends, I started looking into generating some performance benchmarks with prefect+dask, and I was wondering if anyone here has done that before and could provide some pointers. Right now, I only have my laptop but am willing to figure out how to set up a proper dask cluster to do this.
    r
    1 replies
    Copy to Clipboard
  • l

    liren zhang

    2 years ago
    Hi, I am just installing Prefect on my Windows 10 PC and have received the following error. I have tried many things to solve my issue and no luck so far. Would like to get some help on this.... I was running the install
    pip install prefect
    Building wheels for collected packages: pendulum
      Building wheel for pendulum (PEP 517) ... error
      ERROR: Command errored out with exit status 1:
       command: 'c:\users\zhang\appdata\local\programs\python\python38\python.exe' 'c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\zhang\AppData\Local\Temp\tmpjjs207ax'
           cwd: C:\Users\zhang\AppData\Local\Temp\pip-install-2pd94jc4\pendulum
      Complete output (24 lines):
      Traceback (most recent call last):
        File "setup.py", line 2, in <module>
          from setuptools import setup
      ModuleNotFoundError: No module named 'setuptools'
      Traceback (most recent call last):
        File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 257, in <module>
          main()
        File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 240, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "c:\users\zhang\appdata\local\programs\python\python38\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 181, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\api.py", line 57, in build_wheel
          return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
        File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 56, in make_in
          wb.build()
        File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 82, in build
          self._build(zip_file)
        File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 102, in _build
          self._run_build_command(setup)
        File "C:\Users\zhang\AppData\Local\Temp\pip-build-env-e3fac8fx\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 130, in _run_build_command
          subprocess.check_call(
        File "c:\users\zhang\appdata\local\programs\python\python38\lib\subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['c:\\users\\zhang\\appdata\\local\\programs\\python\\python38\\python.exe', 'setup.py', 'build', '-b', 'build']' returned non-zero exit status 1.
      ----------------------------------------
      ERROR: Failed building wheel for pendulum
    Failed to build pendulum
    ERROR: Could not build wheels for pendulum which use PEP 517 and cannot be installed directly
    l
    Chris White
    +1
    6 replies
    Copy to Clipboard
  • m

    Manuel Aristarán

    2 years ago
    Hi! Does anyone have any tips on running Docker tasks in a Flow running on a Docker agent? Mounting the hosts’ Docker socket does not seem to be a good practice…
    m
    Zachary Hughes
    4 replies
    Copy to Clipboard
  • Chris Hart

    Chris Hart

    2 years ago
    looking to use prefect with dask-ml on the default DaskExecutor.. how might I get the sklearn bits to use dask as the joblib backend as referenced here: https://ml.dask.org/joblib.html ?
    Chris Hart
    Jim Crist-Harif
    8 replies
    Copy to Clipboard