https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • g

    George Coyne

    03/13/2020, 4:54 PM
    We trigger flows from lambda, and all of our lambda’s started failing. Did something change with prefect’s API?
    t
    • 2
    • 1
  • t

    Tyler Wanner

    03/13/2020, 5:10 PM
    Hi All! We are experiencing some issues as our Cloud API is rejecting some traffic. We should be fully recovered shortly and I will provide additional details
    👍 1
  • t

    Tyler Wanner

    03/13/2020, 5:32 PM
    In an effort to add additional security measures to our API, we experienced a brief outage this afternoon. This issue has been resolved, and we thank you for your patience
  • t

    Tyler Wanner

    03/13/2020, 5:33 PM
    Feel free to reach out to the channel, or to me directly, with any questions about this incident or any impact you might have had from it
    🤘 2
    👍 4
  • j

    Joe Schmid

    03/13/2020, 7:34 PM
    Question for folks doing CI/CD automation to build & register Prefect Flows. Building Docker storage for one of our flows from our CI/CD system (Screwdriver.io) fails with the exception below. Docker appears to be installed and working properly because
    docker run hello-world
    succeeds. (There is special "docker-in-docker" config we use with Screwdriver and this works well for many of our other projects.) I can also build Docker storage successfully on my laptop from inside the same image that our CI/CD system is using. I'm sure it's some simple config that I'm missing -- anybody run into this?
    15:25:50 Traceback (most recent call last):
    15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    15:25:50     "__main__", mod_spec)
    15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    15:25:50     exec(code, run_globals)
    15:25:50   File "/usr/local/lib/python3.7/site-packages/srm_datasci-0.1.0-py3.7.egg/srm/datasci/flows/util/test_flow.py", line 35, in <module>
    15:25:50     flow.register("Utilities")
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1412, in register
    15:25:50     no_url=no_url,
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 616, in register
    15:25:50     serialized_flow = flow.serialize(build=build)  # type: Any
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1221, in serialize
    15:25:50     storage = self.storage.build()  # type: Optional[Storage]
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 263, in build
    15:25:50     self._build_image(push=push)
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 293, in _build_image
    15:25:50     self.pull_image()
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 483, in pull_image
    15:25:50     client = self._get_client()
    15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 471, in _get_client
    15:25:50     return docker.APIClient(base_url=self.base_url, version="auto")
    15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 190, in __init__
    15:25:50     self._version = self._retrieve_server_version()
    15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
    15:25:50     'Error while fetching server API version: {0}'.format(e)
    15:25:50 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    l
    m
    • 3
    • 9
  • i

    itay livni

    03/14/2020, 4:00 AM
    https://stackoverflow.com/questions/60679595/how-does-one-update-a-prefect-flow-with-duplicate-parameters
    ✅ 2
    👀 2
    c
    • 2
    • 5
  • b

    Bob Colner

    03/15/2020, 11:06 PM
    Hi folks!, I'm a disgruntled airflow user here looking to learn more about prefect. I've started to port over a simple ETL workflow with some success.
    c
    • 2
    • 32
  • b

    bardovv

    03/16/2020, 7:10 AM
    Do I have to use prefect cloud to get access of the UI dashboard cause my tasks run but I can't see the UI dashboard. I haven't logged into prefect cloud.
  • b

    bardovv

    03/16/2020, 8:02 AM
    (myenv) amod@amod-ThinkPad-T430:~/prefect$ prefect run cloud --name hello-flow --project 'Hello, World!' hello-flow not found
  • b

    bardovv

    03/16/2020, 8:03 AM
    I have followed all the steps from this https://docs.prefect.io/cloud/tutorial/first.html#write-flow yet unable to work.
    t
    • 2
    • 2
  • j

    John Ramirez

    03/16/2020, 2:15 PM
    Hey - is it possible to pass a function to return a default value for a Parameter
    c
    • 2
    • 1
  • n

    Nathan Molby

    03/16/2020, 2:39 PM
    I’m getting an error of “Cannot serialize socket object”. The error says “Can’t pickle function <myFunction>: it’s not the same object as Helper_Functions.myFunction”. Any suggestions on how to fix this? I’m mapping over a task with 3 unmapped parameters which as the results of other tasks.
    j
    b
    • 3
    • 16
  • j

    John Ramirez

    03/16/2020, 7:42 PM
    hey everyone - I’m experiencing a problem where my workflow in a remote dask cluster breaks down after a mapping task. The workflow is terminated on the remote cluster but not in the cloud. any thoughts?
    j
    • 2
    • 3
  • s

    Scott Zelenka

    03/17/2020, 8:05 PM
    I'm attempting to schedule a Flow, but receiving this error message:
    prefect.utilities.exceptions.ClientError: Flows with required parameters can not be scheduled automatically.
    I understand what the error message is saying, but in the
    parameter_defaults
    for the
    InvervalClock
    of the Flow, I have the required Parameters defined. It's easy to remove the required flag for these Parameters, but for code readability I'd like to keep them in there. Why is it considered an error, when all of the
    IntervalClock
    defined in the
    Schedule
    of the Flow have a value for the required Parameters defined?
    c
    • 2
    • 3
  • t

    Tyler Wanner

    03/17/2020, 8:50 PM
    Hi All, we are rolling out a new security platform that appears to be causing some service interruptions. Please bear with us as we try to resolve this issue
  • s

    Scott Zelenka

    03/17/2020, 9:17 PM
    Hi Team, how do I troubleshoot this error message:
    March 17th 2020 at 5:13:44pm | prefect.CloudTaskRunner
    INFO 
    Task 'sfdc_download_data': Starting task run...March 17th 2020 at 5:13:44pm | prefect.CloudTaskRunner
    ERROR 
    Unexpected error: KeyError('text')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 225, in check_task_is_cached
        for key, res in (candidate_state.cached_inputs or {}).items()
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 225, in <dictcomp>
        for key, res in (candidate_state.cached_inputs or {}).items()
    KeyError: 'text'
    j
    • 2
    • 5
  • z

    Zachary Hughes

    03/17/2020, 9:37 PM
    Hi folks, The issues @Tyler Wanner mentioned have been resolved. Impact appears to have been minimal, but as always we're happy to answer any questions you may have.
    👍 1
  • b

    bardovv

    03/18/2020, 6:50 PM
    from prefect import Flow, task
    
    @task
    def add_ten(x):
        return x + 10
    
    with Flow('simple map') as flow:
        mapped_result = add_ten.map([1, 2, 3])
    I am using this code expected output is 11,12,13 in mapped_result how do I print the result?
    c
    • 2
    • 5
  • b

    bardovv

    03/18/2020, 6:58 PM
    is there any open source alternative to prefect cloud?
    l
    • 2
    • 7
  • l

    Luke Orland

    03/18/2020, 7:57 PM
    We are getting some workflow runs failing immediately on Prefect Cloud with just one log message:
    March 17th 2020 at 2:32:51am | agent
    ERROR 
    An error occurred (ThrottlingException) when calling the RunTask operation (reached max retries: 4): Rate exceeded.
    z
    • 2
    • 22
  • b

    bardovv

    03/19/2020, 8:47 AM
    Failed to load and execute Flow's environment: TypeError('an integer is required (got type bytes)')
    j
    • 2
    • 1
  • e

    emre

    03/19/2020, 10:02 AM
    Hello Everyone I’ve had a mild issue about
    ShellTask
    , in which a
    git clone
    command doesn’t return some of the stdout lines (despite using
    return_all=True
    ). The command is as follows:
    git clone https://{some_url}.git dbt_project -v -b master --depth 1
    . Running it from bash logs the following to stdout:
    Cloning into 'dbt_project'...
    POST git-upload-pack (156 bytes)
    POST git-upload-pack (165 bytes)
    remote: Counting objects: 40, done.
    remote: Compressing objects: 100% (34/34), done.
    remote: Total 40 (delta 2), reused 0 (delta 0)
    Unpacking objects: 100% (40/40), done.
    While running the same command through
    ShellTask
    , and iterating over the resulting list of stdout lines logs the following:
    [2020-03-19 09:57:50,792] INFO - prefect.Task: print_task | Cloning into 'dbt_project'...
    [2020-03-19 09:57:50,792] INFO - prefect.Task: print_task | POST git-upload-pack (168 bytes)
    [2020-03-19 09:57:50,793] INFO - prefect.Task: print_task | POST git-upload-pack (177 bytes)
    By any means, I am not well versed on bash, and this may very well be an issue about how
    git clone
    logs to stdout. Wondering if anybody else encountered a similar issue, and if they managed to come up with a fix?
    t
    • 2
    • 2
  • e

    Eric

    03/19/2020, 12:43 PM
    Does prefect or the prefect community use a common solution for job tracking? The use case is: 1. get S3 file listing 2. use <some job tracking solution> to ignore files that have been previously processed successfully, process new files, and retry failed files with < 3 previous attempts 3. kick off subsequent tasks
    s
    d
    • 3
    • 3
  • b

    Brett Naul

    03/19/2020, 4:37 PM
    is there a migration guide or anything for switching to the new result handling implied by https://docs.prefect.io/core/PINs/PIN-16-Results-and-Targets.html and https://github.com/PrefectHQ/prefect/pull/2151/files ? I think I understand the concept but I'm not sure exactly what I'm supposed to do about it 🙂 or maybe the answer is "nothing yet" but then the deprecation warning feels a bit premature...?
    c
    l
    • 3
    • 3
  • m

    Matt Juszczak

    03/19/2020, 10:39 PM
    Hi all! We are using DBT standalone (cli) right now. I’ve been doing some research on an orchestration tool that can handle retries, etc. At first I thought of Jenkins but then airflow was recommended. After even more reading, Prefect seems to one more recommended choice if you are green fielding. I’m having trouble finding further documentation to answer my questions: Does Prefect replace Airflow? Would it replace parts of DBT as well? Thank you!
    c
    • 2
    • 9
  • a

    Arsenii

    03/20/2020, 2:55 AM
    Hey Prefect community, I'm trying to understand how to integrate CICD in the flow development process, in part to prevent accidents when pushing flows from a developer's local machine. My first reflex was to create a private git repo with flows' code, and automatically register all flows inside with Cloud only after a PR was merged to `master`/a tagged release was made. This approach destroys flow versioning however, so then I need to parse git history to find only files that got added/modified recently (via Github Actions, or a step in CircleCI, for example)... Not very elegant. Did anyone else run into a similar issue? How did you go about it? Thanks!!
    👍 1
    b
    • 2
    • 6
  • b

    bardovv

    03/20/2020, 12:58 PM
    can prefect core alone be used in production systems?
    s
    a
    • 3
    • 9
  • a

    Adam Roderick

    03/20/2020, 3:18 PM
    Thank you for the Cloud Hooks feature! We will start using the slack and email hooks soon. I imagine a day when you have tons of integrations tied into this for easy integrations. Keep up the good work!
    😄 2
    🎉 7
    n
    j
    • 3
    • 2
  • b

    Braun Reyes

    03/20/2020, 3:40 PM
    Now that we have the concept of multiple schedules per flow...has there been any thought to concept of environments for CICD? Like deploy a flow then promote that version to dev -> uat -> prod? We kind of do that now but with different projects and actually registering the flow in different projects. Down side is that it’s 2 flows against your allotment. Would just have to think about ‘config’ per environment. Dbt Cloud does this. Also kind of inline with k8s namespaces and perhaps AWS elastic beanstalk
    j
    c
    +2
    • 5
    • 13
  • j

    Jeremiah

    03/20/2020, 5:52 PM
    👋 @Laura Lorenz (she/her) is hosting our first ever online cantina in a few minutes if anyone would like to join! Details over in #prefect-contributors
    :marvin: 2
    🎉 2
    :upvote: 2
Powered by Linen
Title
j

Jeremiah

03/20/2020, 5:52 PM
👋 @Laura Lorenz (she/her) is hosting our first ever online cantina in a few minutes if anyone would like to join! Details over in #prefect-contributors
:marvin: 2
🎉 2
:upvote: 2
View count: 1