https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Adam Brusselback

    05/05/2021, 4:27 PM
    And then later an error saying:
    could not translate host name "<Task: get_value>" to address: Unknown host
  • a

    Adam Brusselback

    05/05/2021, 4:29 PM
    My flow looks like, but I have tried multiple different ways to figure out how to do this when my stuff is only known at runtime.
    with Flow("test") as flow:
        client_name = Parameter('client_name', default='client_demo')    
        client_context = get_client_context("C:/.secrets/client/", client_name)
        db_exec = PostgresExecute(
            host=get_value(client_context, 'oltp_host'), db_name=get_value(client_context, 'oltp_database'), user=get_value(client_context, 'oltp_user'), query="SELECT do_something()"
        )
        result = db_exec(password=get_value(client_context, 'oltp_pass'))
    ✅ 1
  • b

    bastianwegge

    05/05/2021, 5:53 PM
    Hey guys, I’m currently working on a GitLab CI Pipeline that leverages the Prefect Python Client and builds and also publishes our flows as Docker images into our internal registry. In that regard I found that the environment variable documentation for
    PREFECT__CLOUD__AUTH_TOKEN
    is a bit too hidden when you’re approaching from this side. I had to dig into the Storage Registration Code (Prefect library internals) to find that the Client takes this variable (if it is specified) and uses it for registration purposes on prefect Cloud. Just a small thing to improve in the API-docs I guess that might make it a lot easier to get up and running.
    👀 2
    k
    • 2
    • 1
  • t

    Todd Lindstrom

    05/05/2021, 5:55 PM
    Dear Community - New Issue. I have created an m5.large EC2 instance and I am running conda python 3.8.8 and installed docker, docker-compose, and pip installed prefect. On the EC2 Instance I opened up port 8080 and 4200. I can get to the UI at port 8080 and I can connect to graphql, I can startup the local agent - all that works. But when I try to create a project I get errors on the console for a Foreign Key Violation..... So after a while I fired up a new ssh console and tried to create a project from the command line. And that worked fine. And then I tried to create another project again back from the UI - and this time it worked. What exactly happened? Is this normal?
    b
    • 2
    • 5
  • n

    Nathan Walker

    05/05/2021, 6:43 PM
    Hey folks, is there a way to add metadata tags/labels to Flows/Flow Groups? Adding a Label to a Flow Group has the added feature of helping decide which Agent to run on, but I'm looking for something that's purely informational. The goal is to be able to query GraphQL for flows with specific tags and do something with that list of IDs. For example, if we want to pause all jobs for Client X, but not Client Y or Z (and there's nothing in the flow naming convention to indicate), we want to be able to just run a query for all jobs with tag "Client-X" and then run a mutation to pause/delay those flows. Or maybe we want to track slow flows -- so we query for flows running 50% slower than their past monthly average runtime and add a "slow" tag so we can make some changes. Does this exist? Is there a place in a flow for arbitrary information? Maybe a "tags" parameter that just never gets used?
    m
    • 2
    • 7
  • j

    Jason Prado

    05/05/2021, 6:58 PM
    My prefect agent was unreachable for several hours and a bunch of jobs got queued up behind it. When the agent restarted, it ran a bunch of duplicate jobs in a row. They’re idempotent so I think they’re okay, but I’m worried I’ll introduce a bug that depends on being the only job of its kind running at the same time, and in this kind of situation something bad could happen. Can I specify that a
    Schedule
    try and run, and if it fails expire the job after 60 seconds of trying?
    k
    • 2
    • 5
  • r

    Robin

    05/05/2021, 8:01 PM
    Hey prefect people, a
    resource manager
    related question: I am trying to setup a resource manager to do some larger-than-memory computations within a task. I wanted to start simple, but even the example gives the following error for me:
    multiprocessing\spawn.py", line 134, in _check_not_importing_main
        raise RuntimeError('''
    RuntimeError:
            An attempt has been made to start a new process before the
            current process has finished its bootstrapping phase.
    
            This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:
    
                if __name__ == '__main__':
                    freeze_support()
                    ...
    
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.
    Untitled.py
    k
    a
    +2
    • 5
    • 35
  • a

    Adam Brusselback

    05/05/2021, 8:43 PM
    So, one more question for the day, may not have an easy solution for Prefect atm from what i've read... What is the best way to deal with needing a persistent database connection between tasks, because those tasks are doing work within a transaction? I have some Talend jobs which currently deal with that just fine, but I simply cannot see a similar way to do it in Prefect.
    ✅ 1
    k
    r
    m
    • 4
    • 22
  • j

    Jonathan Chu

    05/06/2021, 12:17 AM
    The flow run names (e.g.
    daffy-wolf
    ) already look very docker-name like It seems like it'd be pretty slick if the
    DockerAgent
    then also used the same name for launching containerized flows. Not strictly needed, and maybe only relevant for our setup where our log aggregator shows the container name that the logs came from
    n
    m
    j
    • 4
    • 19
  • s

    Samuel Hinton

    05/06/2021, 9:33 AM
    Hi all, is there a good way of deleting flows. Trying to delete a flow gives a popup error:
    We could not delete your flow. Please try again. If this problem continues, contact <mailto:help@prefect.io|help@prefect.io>
    c
    j
    v
    • 4
    • 7
  • s

    Samuel Hinton

    05/06/2021, 10:06 AM
    Hi all, Im experiencing the dread “white dashboard screen” once again (and have upgraded to v0.14.17). In the chrome extension browser I see:
    Uncaught (in promise) AuthSdkError: PKCE requires a modern browser with encryption support running in a secure context.
    The current page is not being served with HTTPS protocol. PKCE requires secure HTTPS protocol.
    Maybe this is causing issues for the dashboard (even though I can see a flows runs without issue if I navigate to the url). Currently the IP address of the server is only accessible within a VPN so we have been using http, and it used to be fine. Is there a way of saying “Dont care about this?” or a way of attaching a certificate to a given service. Very similar to this old bug here: https://github.com/PrefectHQ/ui/issues/588 Did some env vars not propagate correctly again? Confused as to why this used to work, but a redeploy from the prefect images no longer does so
    j
    • 2
    • 7
  • j

    Jeremy Phelps

    05/06/2021, 12:50 PM
    Hi all, does anyone know of any condition that should cause a flow to become totally stuck? This diagram seems to indicate that all the tasks upstream from this one are done, so I'd expect that this one (or another one with no pending upstream tasks) should start running. Currently there is only one flow run, and all tasks are either completed or pending. The flow run has been idle for 8 hours. Running Prefect 0.14.2.
    k
    m
    • 3
    • 82
  • s

    Sean Perry

    05/06/2021, 3:23 PM
    Has anyone seen this message?
    TypeError: ‘ContextualVersionConflict’ object is not iterable
    I have a flow that I am trying to call
    run()
    on to run it locally.
    k
    • 2
    • 3
  • s

    Sean Perry

    05/06/2021, 3:52 PM
    Resurrecting https://prefect-community.slack.com/archives/CL09KU1K7/p1616759310466600 When I have the following in my requirements.in file:
    boto3
    dask-cloudprovider[aws] == 2021.3.1
    prefect[aws] == 0.14.17
    Trying to run pip-compile on it fails with a complaint about botocore versioning:
    botocore<1.20.50,>=1.20.49 (from aiobotocore==1.3.0->dask-cloudprovider[aws]==2021.3.1->-r requirements/production.in (line 14))
      botocore<1.21.0,>=1.20.67 (from boto3==1.17.67->-r requirements/production.in (line 9))
      botocore<2.0a.0,>=1.12.36 (from s3transfer==0.4.2->boto3==1.17.67->-r requirements/production.in (line 9))
    This boils down to: botocore ==1.20.49,>=1.20.67 which cannot be satisfied. What is a safe version of boto3 that works with prefect and dask cloud provider? Removing the version numbers above from the requirements.in does not help, same complaints. Which is to be expected since pip is going to try the newest it can without guidance.
    c
    n
    • 3
    • 10
  • z

    Zach Schumacher

    05/06/2021, 4:16 PM
    is there a way to give docker tasks a name?
    k
    • 2
    • 6
  • t

    Todd Lindstrom

    05/06/2021, 4:18 PM
    Agent Issue. I'm just trying prefect out - using the backend server and running on an EC2 instance m5.large with 8080 and 4200 ports allowed external.  I have the server running, the UI attached, the graphql connected, and I started an agent on the same machine. For me I installed prefect using a conda activation - so to start the agent - i did the conda activation then prefect agent start command. HOWEVER, when I add a flow and run it - it fails on
    unexpected error: NameError("name 'prefect' is not defined
    This appears to me that it is not running in the condra activation and yes if thats true then the pip install will not be there. Please advise how I can set the python environment of my local agent.
    k
    • 2
    • 15
  • c

    ciaran

    05/06/2021, 5:59 PM
    What's the canonical way to provide ENV vars to DaskExecutor Schedulers/Workers? I'm assuming it's going to be in either
    cluster_kwargs
    or
    client_kwargs
    on https://docs.prefect.io/api/latest/executors.html#daskexecutor ?
    k
    • 2
    • 6
  • s

    Sean Perry

    05/06/2021, 6:02 PM
    Would there be interest in a
    FilterTask
    variant that had an extra parameter along the lines of:
    log_filtered (Callable, optional): a function to use for logging any result that did not pass `filter_func`.
    Use would be something like:
    valid_configurations = FilterTask(filter_func=some_validator, log_filtered=log_invalid_configuration)
    where
    def log_invalid_configuration(configuration):
        # log the reasons this configuration is invalid. Or some other special logging for invalid configurations.
        # Like, notifying a team to fix the configuration.
    Adding this to
    FilterTask
    would be straightforward. Turn the list comprehension info a for loop appending to
    filtered
    and call
    log_filtered
    whenever a result does not match the predicate. I am currently making 2 calls to FilterTask, once to collect valid and once to collect invalid. Which is not pretty.
    m
    • 2
    • 2
  • l

    Lukas N.

    05/06/2021, 7:16 PM
    Hello 😛refect:, I have a hard time understanding the depth-first-execution of mapped tasks with Dask executor. From my experiments it seems like the execution is bfe, but maybe I'm missing something trivial like task trigger or something? More info with code example in 🧵
    k
    • 2
    • 5
  • a

    Austen Bouza

    05/06/2021, 7:36 PM
    Hi all, I have some newbie Prefect deployment questions if anyone can help: 1. What is considered best practice for automating flow registration, docker builds, etc. when deploying new code? Is it preferable to simply lean on CI/CD tools for this, or do people use things like the docker BuildImage task as part of a multi-flow setup? I want to use something like GitHub for storage but locally-built docker images for the run config, but am not sure how to close the gap on automating builds from local Dockerfiles. 2. Is the only way to define non-python docker image dependencies for a flow simply to write a Dockerfile?
    ➕ 1
    k
    m
    • 3
    • 5
  • g

    George Tam

    05/06/2021, 11:10 PM
    Hi everyone, I'm interested in using prefect to continuously replicate a bigquery table into a postgres database. Is this possible to do within prefect?
    k
    • 2
    • 9
  • t

    Trevor Kramer

    05/07/2021, 12:39 AM
    Is it possible to retry mapped tasks? Whenever I try it I get nulls being passed as the mapped input parameter. We are using prefect cloud with prefect results.
    k
    • 2
    • 11
  • d

    dipsy wong

    05/07/2021, 8:12 AM
    Hi everyone, I am trying to push a markdown file as an artifact which contains some js in order to render a chart, but the chart disappeared in prefect UI. Actually is it possible to render scripted markdown in prefect? here are the markdown file I am trying to pushed and the flow definition. Thanks
    m
    • 2
    • 4
  • v

    vaibhav satpathy

    05/07/2021, 12:00 PM
    Hi everyone, I am trying to run prefect agent in Detached mode using supervisor as per prefect documentation, even though i am able to spawn up the prefect agent, the additional params passed is not read. I have attached the CLI command that i use to run the agent and it works as expected but at the same time when i run it using supervisor, it doesn't attach the labels or get the API for the agent to connect to. If there is any other way to run prefect agent in detached mode or if there is an error in the way i am using is could you please let me know. Thanks in advance
    agent_start.shsupervisord.conf
    k
    • 2
    • 29
  • h

    Howard Cornwell

    05/07/2021, 2:09 PM
    Hey, I’m wondering if there’s a supported way for the prefect agent to run jobs using a specific kubernetes nodeselector? For example, a flow labeled
    high-memory
    runs on a
    high-memory
    agent which spawns jobs via a
    high-memory
    nodeselector.
    m
    • 2
    • 2
  • z

    Zach Schumacher

    05/07/2021, 3:31 PM
    Anyone ever have issues using the docker side car recipe? I can’t get the docker side car to authenticate to docker hub. We are using the same registry credentials we use in job containers, and that works not problem. Thinking the docker daemon inside the sidecar needs to login also somehow? Not sure though.
    k
    • 2
    • 4
  • b

    Brian Mesick

    05/07/2021, 5:30 PM
    Hey all. We’re trying to move to github storage for our flows, but don’t use Prefect secrets. Is there a way we could set the Github access token to make this work?
    m
    t
    +2
    • 5
    • 22
  • s

    Saksham Dixit

    05/07/2021, 6:33 PM
    Hey all, I am trying to get the parameter values of a flow run inside my state handler, is there a way to fetch the parameter values ?
    k
    • 2
    • 7
  • d

    Damien Ramunno-Johnson

    05/07/2021, 8:30 PM
    Hello, I am trying to run the tutorial for prefect cloud
    In [1]: import prefect
       ...: from prefect import task, Flow
       ...:
       ...: @task
       ...: def say_hello():
       ...:     logger = prefect.context.get("logger")
       ...:     <http://logger.info|logger.info>("Hello, Cloud!")
       ...:
       ...: with Flow("hello-flow") as flow:
       ...:     say_hello()
       ...:
       ...: # Register the flow under the "tutorial" project
       ...: flow.register(project_name="tutorial")
    but get
    ClientError: [{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}]
    k
    a
    g
    • 4
    • 10
  • f

    Fina Silva-Santisteban

    05/07/2021, 10:34 PM
    Hi everyone, I’m having issues using jupyter tasks with prefect core. When I try to run a jupyter task it returns
    ImportError: Using `prefect.tasks.jupyter` requires Prefect to be installed with the "jupyter" extra.
    and when I try installing it by running
    pip3 install prefect[jupyter]
    it returns
    no matches found: prefect[jupyter]
    My current workaround is to use Prefect API and to add that as dependency during flow registration
    python_dependencies = ["prefect[jupyter]"]
    flow.storage = Docker(...
            python_dependencies=python_dependencies,
            ...)
    Is there a way to pip install the jupyter things needed to run a jupyter task using Prefect Core only?
    • 1
    • 1
Powered by Linen
Title
f

Fina Silva-Santisteban

05/07/2021, 10:34 PM
Hi everyone, I’m having issues using jupyter tasks with prefect core. When I try to run a jupyter task it returns
ImportError: Using `prefect.tasks.jupyter` requires Prefect to be installed with the "jupyter" extra.
and when I try installing it by running
pip3 install prefect[jupyter]
it returns
no matches found: prefect[jupyter]
My current workaround is to use Prefect API and to add that as dependency during flow registration
python_dependencies = ["prefect[jupyter]"]
flow.storage = Docker(...
        python_dependencies=python_dependencies,
        ...)
Is there a way to pip install the jupyter things needed to run a jupyter task using Prefect Core only?
Ok I figured out that I wasn’t pip’ing correctly, sorry about that! 😅 The command should be
pip3 install "prefect[jupyter]"
, and that works fine!
👍 3
💪🏼 1
View count: 1