• a

    Avi A

    1 year ago
    Hey community, We have a mapped task in our flow, spawning several thousand mapped tasks, whose output it then forwarded to one task (unmapped). We noticed that these tasks are being called twice. Is this expected in some way? This happened to me before, but I thought this was already solved. prefect 13.1 running on dask-kubernetes
    a
    m
    4 replies
    Copy to Clipboard
  • a

    Amit Gal

    1 year ago
    Hey all, New to the community and Prefect in general, but a bit in love with it already 🙂 I have a basic architectural question, and I hope this is a good place to ask it - otherwise, please let me know! In a nutshell, I am looking to
    map
    case
    . More in detail, I am converting into Prefect an existing process currently run locally through a simple CLI, to fetch and handle datasets from a remote server. One of the first things in this process is checking the dataset version, using 
    check_version(id)
     which returns a 
    bool
    . In the CLI this is then used in a simple logical block to determine how to continue. In the CLI a list of ids is handed and looped over, making that check for each element in the list (sequentially). I want to convert this into Prefect and take advantage of the parallelisation opportunities. Therefore, I thought that 
    case
     and 
    map
     would work great here, something like this (which is not possible):
    ids = [id1, id2, id3]
    with Flow('Check version') as flow:
        with case (check_version.map(id=ids), True):
            val1 = action_if_true()
        with case (check_version.map(id=ids), False):
            val2 = action_if_false()
    The issue here being that the output from 
    check_version(id)
     is mapped, and is therefore a list of bools - 
    [True, True, False]
    , which cannot be compared against a 
    bool
    . I found no option to map case. I did however see the option to map switch, and I could imagine using that if I have to, but since it must take strings as input (it uses a
    dict
    to get the different scenarios) it would require a conversion or changing the original functions in the CLI, which I would rather not do. This is a pretty simple thing, and I imagine someone else has solved this before, but could not locate anything in the docs or other places. Or perhaps I am thinking about this in the wrong way. Any input is welcome, thanks!
    a
    j
    8 replies
    Copy to Clipboard
  • s

    SImon Hawe

    1 year ago
    Hello Prefect team. I have a question regarding the prefect context. Is it possible to set prefect context in one task (so on runtime) and use those values in other tasks? It works very well when running it locally but I have no success so far when deploying it to prefect cloud. As things late dates are set dynamically, there should be some way to do that.
    s
    j
    5 replies
    Copy to Clipboard
  • m

    Mitchell Bregman

    1 year ago
    hey there — I am designing a flow that requires
    pyodbc
    to be installed in order to connect to a SQLServer instance… i have pulled Prefects
    master
    docker image and ran interactively to test out pip install pyodbc… running into gcc issues, as expected, but i am now trying to
    apt-get
    or
    apt install
    an odbc driver
    unixodbc-dev
    and getting weird “not found” type results… i have run into similar issues, but was relying on a conda environment which allowed me to
    conda install pyodbc
    but how can i replicate this here with
    pip
    ?
    m
    j
    2 replies
    Copy to Clipboard
  • a

    Arthur Duarte

    1 year ago
    Greetings! Could you please help me with a simple question? I created a duplicated prefect project, using the command:
    prefect create project "my-project"
    Can we also delete projects, using a similar command ?
    a
    james.lamb
    +1
    5 replies
    Copy to Clipboard
  • s

    Sébastien

    1 year ago
    New server, this error isn't telling much:
    #> prefect auth login -t "<my token>"
    Error attempting to communicate with Prefect Cloud
    I can confirm that
    ~/.prefect/config.toml
    isn't being created. Trying to use
    python3 start.py
    to register any flow gives:
    prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
    which unfortunately doesn't add any info. Pinging
    <http://api.prefect.io|api.prefect.io>
    works:
    PING <http://api.prefect.io|api.prefect.io> (34.96.72.220) 56(84) bytes of data.
    64 bytes from <http://220.72.96.34.bc.googleusercontent.com|220.72.96.34.bc.googleusercontent.com> (34.96.72.220): icmp_seq=1 ttl=116 time=3.71 ms
    Any ideas?
    s
    j
    17 replies
    Copy to Clipboard
  • s

    Sébastien

    1 year ago
    s
    Michael Adkins
    +2
    43 replies
    Copy to Clipboard
  • j

    Joseph

    1 year ago
    Hello all. I'm part of a team evaluating Prefect for my employer. Is there any data available on the performance of various components with very large graphs? I have scenarios that could extend up to 10M nodes and would regularly run around 1M nodes.
    j
    Chris White
    2 replies
    Copy to Clipboard
  • Dotan Asselmann

    Dotan Asselmann

    1 year ago
    Hey, is it possible to use context value as a default for Parameter?
    Dotan Asselmann
    m
    +3
    19 replies
    Copy to Clipboard
  • Edison A

    Edison A

    1 year ago
    Hello, I just read somewhere on the prefect website that prefect no-longer needs docker but I can't find a blogpost of webpage detailing that. Could you please help me with a link / confirmation? Have the tutorials been updated?
    Edison A
    Michael Adkins
    3 replies
    Copy to Clipboard