• m

    matt_innerspace.io

    2 years ago
    Stumbled upon prefect while debugging a few jenkins issues, and it appears to be a great tool that could replace a lot of what i'm doing in jenkins (and was hoping to migrate to airflow). The only drawback I can see, having poked around for an hour, is the lack of UI in the opensource version. While I'd love to use the cloud, the size of my workloads and sensitivity of data makes running in prefectcloud a problem. Just wanted to give some feedback, as i've seen others ask the same question.
    m
    Chris White
    +1
    13 replies
    Copy to Clipboard
  • Jake Schmidt

    Jake Schmidt

    2 years ago
    What’s the Prefectest way to use the value of a
    prefect.Parameter
    in a call to a
    prefect.ShellTask
    ? Would I use a
    StringFormatter
    task?
    Jake Schmidt
    Zachary Hughes
    +1
    4 replies
    Copy to Clipboard
  • m

    matt_innerspace.io

    2 years ago
    One simple thing i'm not grasping here - how do i deploy a flow that exists in a
    hello_world.py
    script? Lots of examples of what can happen inside the python script, but i'm missing how it would be deployed as a flow. I can see the flow runs inside a docker container, on an agent. I can see how the
    prefect
    cli tool can run a flow, but it seems the deployment and running are linked somehow? For comparison, i use openfaas python functions which also run in docker containers and are deployed wherever has room to run it, but there is an explicit
    faas-cli push ...
    to deploy to the cloud/cluster when you're ready.
    m
    Zachary Hughes
    +2
    11 replies
    Copy to Clipboard
  • s

    Sean

    2 years ago
    I have a postgres database that can only be accessed by establishing an ssh tunnel. If possible I would like to establish the tunnel as a task, then use the existing postgres tasks to query the database via the tunnel. However, I believe that a
    CreateSshTunnel
    task does not work in general, as it is stateful and local - i.e. I don't think it would work when using the Dask executor (for example), as the tunnel task could execute on a different node than the postgres task. So what would a correct way to structure this be?
    s
    a
    +1
    6 replies
    Copy to Clipboard
  • Jeff Brainerd

    Jeff Brainerd

    2 years ago
    Hi all — I’m totally new to Prefect and Dask, but ran into an issue today that burned a few hours and wanted to make others aware. I inadvertantly upgraded the dependency
    msgpack
    to
    v1.0.0rc1
    which broke Prefect on Dask. The symptom is errors like this when a flow finishes:
    distributed.protocol.core - CRITICAL - Failed to deserialize
    Traceback (most recent call last):
      File "/Users/jeff/.local/share/virtualenvs/jellyfish-u52nBq9x/lib/python3.7/site-packages/distributed/protocol/core.py", line 106, in loads
        header = msgpack.loads(header, use_list=False, **msgpack_opts)
      File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
    ValueError: tuple is not allowed for map key
    Reverting to
    msgpack 0.6.x
    solved the issue. Easy to repro on
    prefect 0.8.1
    and
    dask 2.8.1
    and
    dask 2.9.1
    . Not sure if this is something wonky in my environment or something real…
    Jeff Brainerd
    Chris White
    3 replies
    Copy to Clipboard
  • m

    matt_innerspace.io

    2 years ago
    i know it's available through graphql, but it would be handy to allow python to remove flows as well, so python can fully manage flows.
    m
    1 replies
    Copy to Clipboard
  • m

    matt_innerspace.io

    2 years ago
    Similar flows - i have dozens of flows that execute the same code on a schedule (every X seconds, forever), but with different parameters (i.e. different customers, deployments, etc). As far as i know, I need to register the same flow for each set of parameters with a unique name, which then requires a unique docker container for each. Wondering if there is a better way to do this?
    m
    1 replies
    Copy to Clipboard
  • y

    yuvipanda

    2 years ago
    hello! prefect seems really cool and a great fit for our use case, particularly around running cron-like schedules.
  • y

    yuvipanda

    2 years ago
    However, my understanding is that to have schedules (https://docs.prefect.io/core/concepts/schedules.html) work, the python process that calls that code needs to continue to run
  • y

    yuvipanda

    2 years ago
    or we need to use prefect cloud