https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • n

    Nathaniel Russell

    10/13/2022, 7:26 PM
    What is the prefect API URL to log in as a service account? (It only seems to give the key)
    ✅ 1
    c
    • 2
    • 1
  • m

    Michał Augoff

    10/13/2022, 9:29 PM
    hi team! is there a plan to add something similar to the
    Projects
    abstraction from Prefect 1? I found it useful to be able to group flows and check system dashboard per project. Or would you recommend to just use tags and custom views instead? (which is only possible with flow runs, not with flows or deployments)
    ✅ 1
    👍 1
    k
    a
    • 3
    • 3
  • s

    Santhosh Solomon (Fluffy)

    10/14/2022, 3:44 AM
    Hi, I am starting with prefect and stuck in a point. I am migrating my scraping system in prefect where I have to run my tasks in different remote servers. As per the sample flow attached, for the sake of distribution I need my second task to run in two remote servers in parallel. Can someone help me with right resource to refer?
    c
    j
    • 3
    • 4
  • z

    Zac Hooper

    10/14/2022, 4:49 AM
    Does anybody have any working examples of RRuleSchedules starting at a half past? So for example I want a flow to run every 5 minutes, between 10:30am and 4pm, Monday to Friday. My first thought is to use a rruleset but so far I haven't been able to successfully stringify
    ✅ 1
    j
    z
    • 3
    • 3
  • m

    Malavika S Menon

    10/14/2022, 8:41 AM
    I have a main flow with multiple sub flows, where each sub flow is expected to run one after the other. Ideally if one of them fails, the rest shouldnt execute and that was how it was. However, when I pass the argument "return_state=True" to the subflows, it continues the execution of the next flow even if the previous one has failed. Is there a workaround with this?
    c
    j
    • 3
    • 5
  • w

    wonsun

    10/14/2022, 9:11 AM
    Hello experts~ :marvin: I'm using prefect 1.x and Cloud. I made it to run a certain flow registered in the cloud with the GraphQL API. In this time i want to use the flow run_name in that flow. Every flow automatically creates a run_name as it is executed, right? Can't we put in the a task inside that flow which gets the flow's run name that is executed by API? There is only one flow here, and I asked because I wanted the output of that flow to be specified as the flow's run_name.
    ✅ 1
    c
    • 2
    • 4
  • s

    Simon Macklin

    10/14/2022, 10:01 AM
    hey prefect. We are using v1 right now and the PG database is getting pretty large fast and appears to impact performance. Do you see any issues with archiving the database tables? Such as the flow, log, message, flow_run, flow_run_state tables?
    ✅ 1
    c
    • 2
    • 2
  • r

    Romain

    10/14/2022, 11:40 AM
    Hello folks, I'm trying to migrate from v1 to v2, and I'm currently trying to figure out what to do with
    apply_map
    . Is is correct to assume that I can easily change my apply_map uses, by subflow runned in parallel?
    ✅ 1
    a
    • 2
    • 1
  • j

    John Mizerany

    10/14/2022, 1:44 PM
    Hi all, I am currently trying to make a dbt block in prefect 2 but whenever I register it through the CLI I get
    Successfully registered 0 blocks
    and the block is not created in the Cloud UI. I have been following the instructions given in the collections documentation and the prefect-dbt github
    ✅ 1
    a
    • 2
    • 5
  • i

    Ilya Galperin

    10/14/2022, 3:03 PM
    Hi all - we’re noticing some strange behavior on flow run links that are being generated in Prefect Cloud. Some flow runs have a URL pattern of:
    <https://app.prefect.cloud/account/ACCOUNTID/workspace/WORKSPACEID/flow-runs/flow-run/FLOWRUNID>
    while others use:
    <https://app.prefect.cloud/account/ACCOUNTID/workspace/WORKSPACEID/flow-run/FLOWRUNID>
    Where the first example has
    flow-runs/flow-run
    and the second just uses a
    flow-run
    format. Some of our users get a 404 error on all links of the first type, and others get 404 on all links of the second type. Does anyone know what might be happening or how to fix this?
    m
    z
    a
    • 4
    • 14
  • s

    Sander

    10/14/2022, 4:19 PM
    Hi, I was wondering if there are any plans for supporting nomad by hashicorp as a kubernetes alternative.
    ✅ 1
    a
    • 2
    • 2
  • k

    Kyle McChesney

    10/14/2022, 4:37 PM
    Interesting question, I have the following flow state function (simplified for example)
    def flow_updater(flow, current_state, next_state):
        if isinstance(next_state, Failed):
            error_information = {k.name: v.message for k, v in next_state.result.items() if type(v) is Failed}
        else:
             error_information = None
        report(error_information)
    When I run this flow locally using
    prefect run -p path/to/flow.py
    with a simulated error, I get the expected error information, however when I run it on our deployment (not cloud, backend in AWS). I get error_information equal to
    {}
    (i.e. no task results have a result of type
    Failed
    ). Is there some other type that gets used when running connected to server? (I am using
    type(v) is Failed
    instead of
    isinstance
    cause I don want to report
    TriggerFailed
    , etc)
    1️⃣ 1
    • 1
    • 1
  • p

    Pedro Henrique

    10/14/2022, 4:39 PM
    hi people, i have a problem after update to prefect 2.6.0 and my script stop working the message of error says .fn() for call my function outside the flow
  • p

    Pedro Henrique

    10/14/2022, 4:41 PM
    after adding call my functions the following error presented when execute File "/home/cpfl0dev/.local/lib/python3.8/site-packages/prefect/logging/loggers.py", line 110, in get_run_logger raise MissingContextError("There is no active flow or task run context.") prefect.exceptions.MissingContextError: There is no active flow or task run context.
    a
    s
    • 3
    • 6
  • p

    Pedro Henrique

    10/14/2022, 4:41 PM
    in version 2.4.2 working
  • j

    Josh

    10/14/2022, 4:48 PM
    Has anyone ever tried running prefect flows through GCP lambda functions? I’m exploring how to support a mapped run of 1000s of tasks in parallel, and am not sure whether to go the Google Kubernetes Engine route (which Prefect has an agent for) or lamda functions. Would appreciate to anyone’s experience.
    r
    • 2
    • 5
  • j

    Josh Paulin

    10/14/2022, 5:16 PM
    What should be the expected behaviour if I do the following: 1. Run a flow with caching enabled on the tasks 2. Delete the cache files 3. Rerun the flow This would be using the built in
    task_input_hash
    as the
    cache_key_fn
    ? On step 3 I’m seeing the flow fail that it can’t find the cache file…
    ✅ 1
    c
    • 2
    • 24
  • n

    Nace Plesko

    10/14/2022, 5:30 PM
    Hi again, This is probably very basic python question, but I'm having trouble that Prefect can't find my module. I already have module
    database
    and a few files in it and all those work fine, but now that I created a new one, the run on the Prefect is saying that it can't find it. I'm probably just missing something obvious, but because of my lack of knowledge of both Python and Prefect I'd really appreciate help on this. Thank you in advance!
    ✅ 1
    b
    • 2
    • 6
  • j

    Josh Paulin

    10/14/2022, 5:54 PM
    Noticing a change in behaviour for results since upgrading to 2.6. A task that would return an empty list now returns
    None
    . Wondering if this is intentional. Minimal example in the thread. Also found out I can’t downgrade from 2.6 to 2.5 because of what looks like some database migrations? Running
    prefect orion database reset -y
    on a database with 2.5 after upgrading to 2.6 and it fails.
    👀 1
    :thank-you: 1
    ✅ 1
    m
    l
    • 3
    • 14
  • r

    redsquare

    10/14/2022, 7:54 PM
    Hey, how are people dealing with Prefect 2.0 CI/CD and agents with deployments/schedules/params, are there any good github action examples floating around?
    ✅ 1
    s
    a
    • 3
    • 7
  • t

    Taylor Babin

    10/14/2022, 8:32 PM
    Hi, I'm having an issue where I was trying to set up a profile with a workspace. I'm able to do this locally. And for some reason I'm stuck in this loop of trying to delete my profile for this account. But everytime i try and delete the profile the profile keeps adding '\\\' and i cant seem to call the profile to delete it. Is there a work around where i can delete the profiles?
    ✅ 1
    b
    n
    • 3
    • 7
  • o

    Omar Sultan

    10/15/2022, 8:41 AM
    Hey everyone, im trying to run orion on my local dev machine and keep running into Its Windows 11, Python 3.9 I have the same configuration on another windows machine and its working fine. Any ideas what might be causing this?
    Error: Got unexpected extra argument (prefect.orion.api.server:create_app)
    ✅ 1
    j
    • 2
    • 3
  • s

    Sander

    10/15/2022, 10:16 AM
    Hi, does anyone know if the prefect-dask lib should be able to handle the operator.KubeCluster from dask_kubernetes?
    ✅ 1
    a
    s
    • 3
    • 5
  • k

    Kostiantyn Liepieshov

    10/15/2022, 10:34 AM
    Hey everybody, I have just started with prefect graphql interface, but I cannot find how I should correctly do the pagination with it? Does anybody knows how to paginate response in prefect graphql?
    r
    • 2
    • 2
  • a

    Andrei Tulbure

    10/15/2022, 3:11 PM
    Hello all. Any idea why the Prefect Flows are freezing when running them in ECS with Prefect Cloud ? They freeze and in the cloud they appear as they are not finished, but they never actually do anything after a certain period. And the ECS machines I use are the largest one possible. ?
    a
    • 2
    • 5
  • f

    Francesco Bartoli

    10/15/2022, 4:29 PM
    Hi all, I started a fresh new installation with this Docker composition template https://github.com/fraibacas/prefect-orion pointing to the latest build. It is working as expected but the Blocks page is empty. Then using this API call
    curl -X 'POST' \
      '<http://prefect-server:4200/api/block_types/install_system_block_types>' \
      -H 'accept: application/json' \
      -H 'x-prefect-api-version: 2.6.1' \
      -d ''
    I’m getting few of them available from that page, the Remote File System one doesn’t come up though. Is this expected with the latest version?
    ✅ 1
    a
    • 2
    • 5
  • s

    Sander

    10/15/2022, 8:02 PM
    Hi, I’m trying to hook up a min.io tenant running on a k8s as a storage block. Is that supported at the moment? Seems the aws s3-bucket doesn’t have get/put-directory?
    ✅ 1
    j
    a
    a
    • 4
    • 62
  • m

    Marcos

    10/16/2022, 12:51 AM
    Hi! I think I found a bug (it happens since 2.6.1 version), when a flow has consecutive
    .map
    calls, if one task fails in the first map, then all dependent tasks stay in NotReady state. Example flow in the comments
    ✅ 1
    a
    • 2
    • 7
  • c

    Christopher

    10/16/2022, 2:59 PM
    I'm dipping my toe back into prefect after last playing with 1.x. I'm trying to get my head around the deployment model. Previously I had a prefect agent running persistently in ECS, which would spin up new containers on demand to handle prefect jobs. I used "docker" storage to do that, which appears to not be a option currently. I can't tell if that's because it's not needed to achieve what I want to do... I'm open to using other storage options like S3, but I guess that's difficult if my flow code has external dependencies? I'm also not much of a python dev so I don't have much of an intuition on how bundling works in the python world!
    ✅ 1
    h
    a
    k
    • 4
    • 11
  • f

    Fady Khallaf

    10/16/2022, 3:31 PM
    Hello everybody, I am trying to make the orion db persistent by linking a volume to orion server, but the db still ephemeral and each time the container is crashed, data is missed. here is the manifest file I am using, looking forward to helping me to figure out what is the issue
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: prefect
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 3Gi
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      namespace: prefect
      name: orion
    spec:
      selector:
        matchLabels:
          app: orion
      replicas: 1  
      template:
        metadata:
          labels:
            app: orion
        spec:
          containers:
          - name: api
            image: prefecthq/prefect:2.6.0-python3.10
            command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
            ports:
            - containerPort: 4200
            env:
              - name: PREFECT_API_URL
                value: '<https://mydomain/api>'
            volumeMounts:
              - mountPath: ~/.prefect
                name: prefect
          restartPolicy: Always
          volumes:
            - name: prefect
              persistentVolumeClaim:
                claimName: prefect
    ✅ 1
    q
    • 2
    • 2
Powered by Linen
Title
f

Fady Khallaf

10/16/2022, 3:31 PM
Hello everybody, I am trying to make the orion db persistent by linking a volume to orion server, but the db still ephemeral and each time the container is crashed, data is missed. here is the manifest file I am using, looking forward to helping me to figure out what is the issue
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prefect
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: prefect
  name: orion
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1  
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.6.0-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        ports:
        - containerPort: 4200
        env:
          - name: PREFECT_API_URL
            value: '<https://mydomain/api>'
        volumeMounts:
          - mountPath: ~/.prefect
            name: prefect
      restartPolicy: Always
      volumes:
        - name: prefect
          persistentVolumeClaim:
            claimName: prefect
✅ 1
q

Q

10/17/2022, 12:17 AM
You're trying to mount your PV to
~/.prefect
, but
~
does not get expanded by K8s (tilde pointing to homedir is a shell concept), so your volume gets mounted into a folder called
/~
.
df -h
# Filesystem                Size      Used Available Use% Mounted on
# /dev/storageclass/pvcname
#                           2.9G      6.0M      2.9G   0% /~/.prefect
Try changing
mountPath
to
/root/.prefect
or setting `PREFECT_HOME`(ref).
🙌 2
f

Fady Khallaf

10/18/2022, 1:39 PM
Thanks so much @Q, it works properly now
👍 2
View count: 1