• f

    Florian Kühnlenz

    2 months ago
    Is it possible to have a Kubernetes Agent run a Flow in a different namespace? Can this be set via the job template? (In 1.0)
    f
    redsquare
    +1
    14 replies
    Copy to Clipboard
  • u

    Urban Škudnik

    2 months ago
    Hey everyone! 👋 We're doing a prototype to test out prefect and plan to go with cloud version for staging and production usage, but need to do local development story too and we want to have everything managed by docker-compose locally (and potentially for CI). Unless I'm missing something, https://hub.docker.com/r/prefecthq/prefect assumes you're going to install prefect on host machine and then
    prefect
    would start it's containers? Is that correct? If so, what's the best path to have everything as a docker-compose file that developers can spin up on request? Ar there any samples for that use case?
    u
    Anna Geller
    5 replies
    Copy to Clipboard
  • Tom Klein

    Tom Klein

    2 months ago
    Hello - We have a NodeJS script that we wish to execute as a task, the entire code (and all the NodeJS) dependencies are located on a docker image on ECR, we’re trying out an experimental alternative to running a Kubernetes namespaced job, which would be - to run the image itself as the image of the entire flow (and have the command shell to run the NodeJS script as one of the tasks of the flow) the docker image uses a docker base image of
    node:12-alpine
    (which doesn’t seem to have
    pip
    and possibly not
    python
    either) i tried to add:
    RUN apk update
    RUN apk add py-pip
    RUN pip install prefect[github,aws,kubernetes,snowflake]
    first steps finished fine, but the prefect installation seems to be taking forever (over 25 minutes already) and also has a lot of weird warning messages like:
    Collecting snowflake-connector-python>=1.8.2                                                                                                                     
      Downloading snowflake_connector_python-1.8.7-py2.py3-none-any.whl (168 kB)                                                                                     
      Downloading snowflake_connector_python-1.8.6-py2.py3-none-any.whl (161 kB)                                                                                     
      Downloading snowflake_connector_python-1.8.5-py2.py3-none-any.whl (159 kB)                                                                                     
      Downloading snowflake_connector_python-1.8.4-py2.py3-none-any.whl (161 kB)                                                                                     
      Downloading snowflake_connector_python-1.8.3-py2.py3-none-any.whl (158 kB)                                                                                     
      Downloading snowflake_connector_python-1.8.2-py2.py3-none-any.whl (157 kB)                                                                                     
    INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
    or :
    INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this 
    run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: <https://pip.pypa.io/surveys/backtracking>
    any ideas?
    Tom Klein
    Kevin Kho
    +2
    45 replies
    Copy to Clipboard
  • datamongus

    datamongus

    2 months ago
    Does anyone know of any prefect tasks for singer taps ?
    datamongus
    Kevin Kho
    2 replies
    Copy to Clipboard
  • a

    Austin Anderson

    2 months ago
    Quick note that there is a typo in the documentation - concurrency limit command examples should have dashes ("concurrency-limit") instead of underscores ("concurrency_limit")
    a
    Kevin Kho
    2 replies
    Copy to Clipboard
  • k

    Kevin

    2 months ago
    Hi! I am working on setting up a simple CI/CD process that registers new flows each time a push is made to main. My folder structure is inspired by the gitlab data team so at the root of my project i have a folder orchestrate. Within that folder i have a folder tasks - within this there are subdirectories that contain custom modules that store tasks we use throughout our flows. While registering flows via the GitHub action, the Prefect CI is complaining about not being able to find a module in the tasks folder. When I run this locally, I do not get any issues. I am pretty sure it is related to my PYTHONPATH setup within the GitHub flow but I have not had any luck troubleshooting. Has anyone out there run into a similar issue?
    k
    Khuyen Tran
    +2
    10 replies
    Copy to Clipboard
  • Chris Reuter

    Chris Reuter

    2 months ago
  • Ifeanyi Okwuchi

    Ifeanyi Okwuchi

    2 months ago
    Hello everyone, I have a task which keeps failing because the
    .map()
    is trying to run the task more times than necessary and the 4th time, the index is a string. .
    product_categories
    is a list containing 3 elements,
    flow_config
    is a dictionary and
    product_category_variables
    is also a dictionary. When the task runs in prefect cloud there are three successful task runs indexed 0, 1, 2 but then it tries to do another run with the index as a string and it fails saying
    Task 'set_dynamic_config_settings['dataset_bucket_path']': Starting task run...
    TypeError: list indices must be integers or slices, not str
    The task
    set_dynamic_config_settings
    looks like this within the flow definition
    with Flow(name="flow-name") as: 
    
       product_category_variables = get_run_variables(
            is_zero_nyp=is_zero_nyp_param,
            bucket_base=flow_config["bucket_base"],
            ltv_product_categories=product_categories,
            return_type="vars",
           )
    
       final_config = set_dynamic_config_settings.map(
            cfg=unmapped(flow_config),
            product_category_variables=unmapped(product_category_variables),
            product_category=product_categories,
            upstream_tasks=[unmapped(product_category_variables)],
           )
    Ifeanyi Okwuchi
    Kevin Kho
    7 replies
    Copy to Clipboard
  • Mansour Zayer

    Mansour Zayer

    2 months ago
    Can someone point me to the right direction please? I'm using Prefect 1.2.2 I have a flow (flow_A) that makes an API call, then flow_B will Transform some of that data, and I want these 2 flow to remain independent while running flow_B as close to flow_A as possible. What is the correct way to ensure that flow_B only starts after flow_A is finished? (I don't want any of these flows to trigger each other, I just want flow_B to wait on flow_A). I tried using
    wait_for_flow_run
    , but I get asked for a
    flow_run_id
    , while I don't have a
    flow_run_id
    , I only have a
    flow_id
    for the parent flow flow_A
    Mansour Zayer
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Constantino Schillebeeckx

    Constantino Schillebeeckx

    2 months ago
    will an agent query for flows across multiple tenants?
    Constantino Schillebeeckx
    Kevin Kho
    3 replies
    Copy to Clipboard