https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Andrew Lawlor

    04/21/2022, 6:17 PM
    I am running pipelines in GKE using Prefect Cloud, and I intermittently see the following error, which seems to come up more often when I am running lots of flows at the same time. Can anyone help?
    Failed to load and execute flow run: RefreshError(TransportError("Failed to retrieve <http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/ACCOUNT/token?scopes=SCOPES> from the Google Compute Engine metadata service. Status: 500 Response:\nb'Unable to generate access token; IAM returned \\n'", <google.auth.transport.requests._Response object at 0x7f5b071fd670>))
    a
    • 2
    • 18
  • a

    Andrew Lawlor

    04/21/2022, 6:36 PM
    i just had a task that ran twice. i only wanted it to run once and i got duplicate data as a result. i dont have retries enabled. is this something other people see sometimes? is there a way to avoid this?
    k
    d
    • 3
    • 24
  • j

    Justin Burchard

    04/21/2022, 7:25 PM
    Hello everyone. I'm a recent adopter of Prefect 1.0. I have a use case in which I have a flow that I would like to reuse for multiple projects, with different credentials and database schemas. I'd like to use a config file to publish the flows. It seems like Parameters would be great for this, but the documentation seems to suggest can only assign them for specific flow runs. Does anyone have any suggestions as to how I could structure this?
    k
    • 2
    • 2
  • f

    Fabio Scarpellini

    04/21/2022, 7:41 PM
    Hello There. I'm trying use two concepts HERE (Reuse single docker storage) and HERE (One flow with multiple parameters) to reuse tasks and create flows with different initial parameters. I'm asking because when I use the "store_as_script=True" raise an error that not found flow specified in registration. It's possible to do that, create a flow without necessary explicit declaration of flows (do like one flow with multiple parameters) ?
    k
    n
    a
    • 4
    • 10
  • a

    Assaf Ben Shimon

    04/21/2022, 11:01 PM
    Hi all 🙂 Noob question here. I'm trying to run a very basic setup, and get the following error: AttributeError: 'Foo' object has no attribute 'set_dependencies'. (This function is called outside of the flow code) Any idea what I'm doing wrong?
    k
    • 2
    • 12
  • d

    Darshan

    04/22/2022, 12:26 AM
    Hello, has anyone heard or read about the use of flowable (https://www.flowable.com/) for the purpose of data & ML orchestration ? We recently were asked to consider flowable as an option to build data & ML pipelines. Personally I see no overlap of flowable with data & ML orchestration frameworks but trying to figure out how to address this question in best possible way to explain to individuals who are not in data & ML domain. Thanks
    a
    • 2
    • 3
  • a

    Ahmed Ezzat

    04/22/2022, 4:35 AM
    Good morning everyone Is there any way to package docker workflows in prefect 2.0 yet (to be used in Kubernetes and Dask)? something like docker storage in prefect 1.0 or at least any workaround like instructions on how to package the workflow? I'd love to use prefect 2.0 in our infra but that really prevents me to do so
    k
    • 2
    • 2
  • r

    Rainer Schülke

    04/22/2022, 7:56 AM
    Good morning all, I couldn't find anything related so far and I am sorry if there is already a thread to this but I have updated a flow which is using one of our modules for data processing. The flow file was updated properly but my module remains the same resulting in an AttributeError not finding my new method. Is Prefect caching my "old" module? Do I have to explicitly update it?
    e
    • 2
    • 5
  • s

    Stefan Crnojević

    04/22/2022, 11:40 AM
    Hey everyone! Does anyone know what are the resource requirements when deploying the prefect-server helm chart? I tried to deploy it on two Raspberry Pi 4s with 8GB RAM that are running k3s, and most of the pods are unable to start, which results in 0 deployments working. All the dead pods are giving the
    standard_init_linux.go:228: exec user process caused: exec format error
    error in logs though, so not 100% sure the issue's hardware-related. Any suggestions would be appreciated 🙏
    a
    • 2
    • 6
  • m

    Malthe Karbo

    04/22/2022, 12:30 PM
    Hi, in prefect 2.0 - is there a way to perform 'yields' - we have some (unfortunately) long running tasks where we stream some content, which takes a while (+30 min, very large zipped json files) - and I can't seem to figure out if I can yield partial result futures in my task incrementally and start processing them in down stream tasks. Currently, the work flow is: • Stream read big annoying file into small chunks, save them in S3 incrementally in a separate thread inside the task, and return list of all the chunked file URI's as single result (= single
    prefectfuture[list[str]])
    object) ◦ this is where I would like to yield • Run some down stream processing on each chunk ◦ Here we use future.result() to loop over and start each task - this works 100%, but if we could use yield in the first task, we could begin processing immediately Is something like this supported / going to be supported? I tried making it work with contextmanager and tasks, but ran into nothing but trouble
    a
    • 2
    • 12
  • j

    Joshua Greenhalgh

    04/22/2022, 12:50 PM
    Is there a way to specify a default job template for kubernetes agent in the manifests? So I am looking at https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template - and I tried to use the
    --job-template
    on the
    prefect agent kubernetes install
    command in the hope that this would work but in fact it does not - do I need to modify the containers args which are
    prefect agent kubernetes start
    to use the flag?
    k
    a
    • 3
    • 12
  • a

    Andrew Lawlor

    04/22/2022, 1:23 PM
    how does everybody separate production vs development environments? i have separate projects for each, with separate labels and an agent in each env querying for the specified label. but yesterday, i was testing a big process in dev where i tried to run a large amount of flows, and it impacted my production environment, where nothing ran. should i be separating envs in another way?
    a
    • 2
    • 1
  • x

    Xavier Babu

    04/22/2022, 3:30 PM
    How can we read log of a task or flow/subflow using Orion REST APIs? Also, Do we need to do any specific configuration to achieve it via REST APIs?
    a
    • 2
    • 12
  • a

    Amogh Kulkarni

    04/22/2022, 4:20 PM
    Hi Prefect Team. Yesterday early morning, two of our flows failed at the same time at 3:00AM PST with the same error. Both the flows ran fine when we restarted it in the morning at around 9AM. This is a really weird non-deterministic issue. We didn’t change or modify the secret SLACK_WEBHOOK_URL in the morning when we re-ran the flow but still it the tasks ran fine. We did root cause analysis but still couldn’t find out why the flows failed. Can you please take a look at it and help us out over here? 1. https://cloud.prefect.io/immersa-prod/flow-run/0ae8fb55-fa89-4a1b-aa86-9d122f490751 2. https://cloud.prefect.io/immersa-prod/flow-run/7c59423a-f7b7-4438-959b-9259058d22b1
    Exception raised while calling state handlers: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 137, in get
        value = secrets[self.name]
    KeyError: 'SLACK_WEBHOOK_URL'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/engine/cloud/task_runner.py", line 64, in call_runner_target_handlers
        new_state = super().call_runner_target_handlers(
      File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
        new_state = handler(self.task, old_state, new_state) or new_state
      File "/usr/local/lib/python3.9/site-packages/toolz/functoolz.py", line 306, in __call__
        return self._partial(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/prefect/utilities/notifications/notifications.py", line 299, in slack_notifier
        str, prefect.client.Secret(webhook_secret or "SLACK_WEBHOOK_URL").get()
      File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 161, in get
        raise exc
      File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 145, in get
        result = self.client.graphql(
      File "
    /usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 570, in graphql
        raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    03:02:29
    I
    k
    r
    • 3
    • 13
  • a

    Artem Vysotsky

    04/22/2022, 4:23 PM
    Hi all, I’m trying to find an similar class to
    PostgresFetch
    in Prefect 2.0. Any pointers?
    k
    • 2
    • 2
  • d

    Domenico Di Gangi

    04/22/2022, 4:47 PM
    Hi all, I am experimenting with tasks and subflows in prefect 2.0. While in tasks I can use the wait_for keyword to enforce task dependency, the same keyword does not work for subflows. I did not found much on the subflows docs, but it seems that in prefect 1.0 dependency between subflows was possible. Is such a feature available also in prefect 2 ?
    m
    m
    • 3
    • 9
  • j

    Jason

    04/22/2022, 4:47 PM
    Where does the Docker storage inherit its credentials from? I assume boto3, but I'm getting a weird behavior on Github Actions. Using the manual docker build and push with the ECR Login action (https://github.com/aws-actions/amazon-ecr-login#usage), I can push my image just fine to ECR. However, when I run this with flow.register() it fails on no authorization.
    k
    a
    m
    • 4
    • 27
  • c

    Chris Reuter

    04/22/2022, 5:29 PM
    meme Friday going on in #random and it is hothothot 🔥
  • t

    Tom Manterfield

    04/22/2022, 8:36 PM
    I’m bumping into all sorts of issues with Deployments on 2.0, just wanted to check if these are known problems or something awry with my config?
    m
    • 2
    • 23
  • l

    Light Luo

    04/22/2022, 9:05 PM
    Hey guys seems our dev tenant (id 023b51a2-d0f4-4b3c-9f36-6eb176e50e46) can only accept 10 users (notifies me about a '10-user limit' when inviting more users). Is this something you can help increase? (A 100-user limit should be nice : )
    k
    • 2
    • 5
  • a

    Artem Vysotsky

    04/22/2022, 9:35 PM
    Hi all, here is what I’m trying to do with 1.0 version. I have a database table
    jobs
    like following:
    id | name                 | query                                | template
    -----------------------------------------------------------------------------------------
    1  | user_count_query     | select count(*) as cnt from users    | user count is {{cnt}}
    2  | projects_count_query | select count(*) as cnt from projects.| project count is {{cnt}}
    I would like to 1. Use
    fetch_jobs = PostgresFetch
    task to get a list of all jobs in the table above 2. Fan out the results of
    fetch_jobs
    to new task say
    run_job_query
    which will for each row grab value of
    query
    column and execute another
    PostgresFetch(query=query)
    In a pseudo flow it would look something like:
    fetch = PostgresFetch()
    
    @task
    def send_to_slack():
      pass
    
    with prefect.Flow() as flow:
        jobs = fetch(query="select query from jobs"), fetch="many")
        for job in jobs: 
           query_result = fetch(query=job["query"])
           send_to_slack(query_result)
    The above code does not obviously work because Tasks are not iterable. Mapping won’t work too, since I cannot call another task in a task Any help appreciated.
    k
    k
    • 3
    • 12
  • a

    Ahmed Ezzat

    04/23/2022, 6:33 AM
    Does Orion support canceling/restarting runs?
    a
    • 2
    • 1
  • s

    scott simpson

    04/24/2022, 8:37 AM
    Hey, new user here trying to get my head around Prefect. I'm using Orion. Can I return a value from a subflow and use it? I'm trying to do it by assigning the function call but the subflow is returning a State rather than the return value of the subflow function. Any ideas?
    a
    • 2
    • 2
  • s

    Slackbot

    04/24/2022, 1:58 PM
    This message was deleted.
    a
    a
    • 3
    • 2
  • a

    Assaf Ben Shimon

    04/24/2022, 3:24 PM
    I'm getting the following error when registering a flow:
    Traceback (most recent call last):
      File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/dockers/extract_markers.py", line 2, in <module>
        from biomarkers.batch_processing import orchestrator
      File "/app/biomarkers/extract_markers_image.binary.runfiles/biomarkers/biomarkers/batch_processing/orchestrator.py", line 130, in <module>
        seq_markers_extraction_flow.register("neuralight-research")
      File "/usr/local/lib/python3.9/site-packages/prefect/core/flow.py", line 1708, in register
        registered_flow = client.register(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 839, in register
        project = self.graphql(query_project).data.project  # type: ignore
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
        response = self._request(
      File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 649, in _request
        raise AuthorizationError(
    prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
    Any idea what can cause it? I've already authenticated using my API key
    a
    k
    • 3
    • 13
  • c

    Chris Reuter

    04/24/2022, 8:52 PM
    https://prefect-community.slack.com/archives/C036FRC4KMW/p1650831951700959 The Pizza Patrol strikes again! If you want to feed your open source meetup, send me a message here on Slack or fill out our form on our Community page!
    👍 1
  • j

    Joshua Greenhalgh

    04/25/2022, 10:29 AM
    How do I get
    CronSchedule("0 11 * * *")
    to fire at 11 and not 12?
    🙌 1
    a
    o
    j
    • 4
    • 15
  • i

    Ilhom Hayot o'g'li

    04/25/2022, 10:30 AM
    HI everyone!. I am newbie here and have trouble signin in to the prefect.io cloud from chrome(100.0.4896.127). Stuck at loader but cannot open the dashboard. Yet from other devices(mobile same chrome version) is opening. Tried to delete cookies but not working so stuck here (lost 2 days). Any suggestions?
    :discourse: 1
    a
    • 2
    • 23
  • n

    Nuno Silva

    04/25/2022, 1:47 PM
    Hello. Is there a way to add a taint to a KubernetesRun without creating a custom job_template.yaml?
    k
    • 2
    • 3
  • m

    Matt Delacour

    04/25/2022, 2:51 PM
    Morning 👋 Is there an equivalent of DbtShellTask in Orion (Prefect 2.0) ?
    a
    m
    • 3
    • 6
Powered by Linen
Title
m

Matt Delacour

04/25/2022, 2:51 PM
Morning 👋 Is there an equivalent of DbtShellTask in Orion (Prefect 2.0) ?
a

alex

04/25/2022, 3:49 PM
Hey @Matt Delacour! We don’t have an equivalent of the DbtShellTask in Prefect 2.0 yet, but it is on our roadmap. We’ll be creating a
prefect-dbt
collection that will have all the functionality of the current dbt tasks in Prefect 1.0. Here’s a blog post about Prefect 2.0 collections and our current list of collections that are available for more info!
m

Matt Delacour

04/25/2022, 3:59 PM
Cool thank you
m

Matthias

04/25/2022, 5:11 PM
I guess for now, you could use
prefect-shell
. This is the equivalent of the ordinary
ShellTask
in Prefect 1.0 on which
DbtShellTask
was written on top of. https://prefecthq.github.io/prefect-shell/
:upvote: 1
m

Matt Delacour

04/25/2022, 5:31 PM
yes perfect. So that I will avoid using
process
directly
Do you have any AWS prefect2.0 already usable ? Equivalent to the following in 1.0
cc @Julian Mehnle
👋 1
View count: 11