• s

    Sven Teresniak

    1 year ago
    I just stumbled upon the artifact API. That's awesome. The direction Prefect is heading is great. Thanks for the good work, guys!
  • t

    Tim Pörtner

    1 year ago
    Hi, is there a way of explicitly add a Parameter to a flow? I have a flow and its tasks are defined by a variable. This variable is already a Parameter but since i'm not using it in one of the tasks it doesn't show up on the web frontend. A very simple example to that would be this:
    with Flow("test-flow") as flow:
        tasks_to_run = {
            (here are information on which tasks should be added to the flow)
        }
        parameter_tasks_to_run = Parameter("tasks_to_run", default=tasks_to_run)
    Can i force it so that this Parameter shows up on the web frontend without actually using the parameter in a task within the flow?
    t
    Jenny
    5 replies
    Copy to Clipboard
  • Adam

    Adam

    1 year ago
    Hello friends, we’re using a paid plan on Prefect Cloud (1 paid seat, $100 / month) but we are now seeing this error when trying to register new flows:
    This tenant already has the maximum number of flows
    . We have 25 flows in 1 project - I don’t find any reference to an upper limit on flows on the pricing page? Any help would be much appreciated
    Adam
    Jenny
    2 replies
    Copy to Clipboard
  • d

    Dolor Oculus

    1 year ago
    I'm using the
    Docker
    flow storage with
    base_image
    , and need to add a file to the image which can only be determined at registration time (flow.storage assignment). I've tried various combinations of
    build_kwargs
    and
    extra_dockerfile_commands
    but am not having any luck. Would appreciate any feedback or pointers!
    my_new_file = "/valid/path/to/file.json"
    flow.storage = Docker( 
        base_image=valid_image_name,
        local_image=True,
        ignore_healthchecks=True,
        build_kwargs={'my_new_file': my_new_file}, 
        extra_dockerfile_commands=['ARG my_new_file', 
                                   'COPY $my_new_file .'])
    Once it gets to the COPY command I get the error
    COPY failed: stat /data/docker/tmp/docker-builder092360870/valid/path/to/file.json: no such file or directory
    I've also tried just the basename of the file as input into the build_kwargs, with no avail. How do I get a file into that temporary docker builder dir? Cheers
    d
    v
    9 replies
    Copy to Clipboard
  • Sean Talia

    Sean Talia

    1 year ago
    does anyone know if it's possible to access the labels passed to a flow during the flow's execution / within the flow context? I'd like to implement some logic that says something like:
    with Flow("test-flow") as flow:
        if "label1" in flow.labels:
            function1()
        elif "label2" in flow.labels:
            function2()
    Sean Talia
    s
    +1
    7 replies
    Copy to Clipboard
  • z

    Zach

    1 year ago
    Were there any breaking changes between prefect 12 and prefect 13? Also between prefect 13 and 14?
    z
    Jim Crist-Harif
    2 replies
    Copy to Clipboard
  • Alex Rud

    Alex Rud

    1 year ago
    👋
    Alex Rud
    1 replies
    Copy to Clipboard
  • Alex Rud

    Alex Rud

    1 year ago
    Hi all... Is there a way to implement a dynamic
    manual_only
    trigger? I'm trying to do the following: read file into dataframe -> run GreatExpectations validation on dataframe -> if validation passes continue with next task, if not allow the next task to run when manually triggered
    Alex Rud
    Jenny
    +1
    9 replies
    Copy to Clipboard
  • v

    Vincent

    1 year ago
    Hi- Does anyone know how to diagnose a k8s job failure with no error messages? After I submitted a job, the 300 worker pods successfully connect to the scheduler and the jobs are set to running in the UI. However, after 8-12 minutes the pod running the scheduler/flow-runner exits (and is resubmitted because the job is still active). The cloud-ui shows still shows the job as running. Thanks for any advice!
    v
    Adam
    4 replies
    Copy to Clipboard
  • a

    Arnulfo Soriano

    1 year ago
    Hi, Im trying to migrate some code from using Prefect enviroments to Prefect executors since enviroments have been depracated. When using DaskKubernetesEnvironment the below code registers just fine and works how we want it to work nicely.
    DaskKubernetesEnvironment(metadata={'image': 'someImage'},
                                             worker_spec_file=os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                                                           f'./{worker_spec}'),
                                             min_workers=min_workers,
                                             max_workers=max_workers,
                                             labels=['DASK'])
    However when using DaskExecutor:
    cluster = KubeCluster(make_pod_spec(image = 'someImage')).from_yaml(os.path.join(os.path.dirname(os.path.abspath(__file__)),f'{worker_spec}'))
    #somethings i've tried
    #cluster = KubeCluster.from_yaml(os.path.join(os.path.dirname(os.path.abspath(__file__)),f'{worker_spec}'))
    DaskExecutor(cluster_class=lambda: cluster,adapt_kwargs={"minimum": min_workers, "maximum": max_workers},)
    I keep getting on the registering step:
    File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/core.py", line 512, in _start
        await ClusterAuth.load_first(self.auth)
      File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/auth.py", line 85, in load_first
        raise auth_exc
      File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/auth.py", line 71, in load_first
        await auth_instance.load()
      File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/auth.py", line 125, in load
        kubernetes.config.load_kube_config(
      File "/usr/local/lib/python3.8/site-packages/kubernetes/config/kube_config.py", line 661, in load_kube_config
        loader = _get_kube_config_loader_for_yaml_file(
      File "/usr/local/lib/python3.8/site-packages/kubernetes/config/kube_config.py", line 624, in _get_kube_config_loader_for_yaml_file
        raise ConfigException(
    kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.
    a
    v
    4 replies
    Copy to Clipboard