prefect-community #prefect-community

Docs

Join the conversation Join Slack

prefect-community

Nathaniel Russell

10/13/2022, 7:26 PM

What is the prefect API URL to log in as a service account? (It only seems to give the key)

✅ 1

Michał Augoff

10/13/2022, 9:29 PM

hi team! is there a plan to add something similar to the

Projects

abstraction from Prefect 1? I found it useful to be able to group flows and check system dashboard per project. Or would you recommend to just use tags and custom views instead? (which is only possible with flow runs, not with flows or deployments)

✅ 1

👍 1

Santhosh Solomon (Fluffy)

10/14/2022, 3:44 AM

Hi, I am starting with prefect and stuck in a point. I am migrating my scraping system in prefect where I have to run my tasks in different remote servers. As per the sample flow attached, for the sake of distribution I need my second task to run in two remote servers in parallel. Can someone help me with right resource to refer?

Zac Hooper

10/14/2022, 4:49 AM

Does anybody have any working examples of RRuleSchedules starting at a half past? So for example I want a flow to run every 5 minutes, between 10:30am and 4pm, Monday to Friday. My first thought is to use a rruleset but so far I haven't been able to successfully stringify

✅ 1

Malavika S Menon

10/14/2022, 8:41 AM

I have a main flow with multiple sub flows, where each sub flow is expected to run one after the other. Ideally if one of them fails, the rest shouldnt execute and that was how it was. However, when I pass the argument "return_state=True" to the subflows, it continues the execution of the next flow even if the previous one has failed. Is there a workaround with this?

wonsun

10/14/2022, 9:11 AM

Hello experts~ :marvin: I'm using prefect 1.x and Cloud. I made it to run a certain flow registered in the cloud with the GraphQL API. In this time i want to use the flow run_name in that flow. Every flow automatically creates a run_name as it is executed, right? Can't we put in the a task inside that flow which gets the flow's run name that is executed by API? There is only one flow here, and I asked because I wanted the output of that flow to be specified as the flow's run_name.

✅ 1

Simon Macklin

10/14/2022, 10:01 AM

hey prefect. We are using v1 right now and the PG database is getting pretty large fast and appears to impact performance. Do you see any issues with archiving the database tables? Such as the flow, log, message, flow_run, flow_run_state tables?

✅ 1

Romain

10/14/2022, 11:40 AM

Hello folks, I'm trying to migrate from v1 to v2, and I'm currently trying to figure out what to do with

apply_map

. Is is correct to assume that I can easily change my apply_map uses, by subflow runned in parallel?

✅ 1

John Mizerany

10/14/2022, 1:44 PM

Hi all, I am currently trying to make a dbt block in prefect 2 but whenever I register it through the CLI I get

Successfully registered 0 blocks

and the block is not created in the Cloud UI. I have been following the instructions given in the collections documentation and the prefect-dbt github

✅ 1

Ilya Galperin

10/14/2022, 3:03 PM

Hi all - we’re noticing some strange behavior on flow run links that are being generated in Prefect Cloud. Some flow runs have a URL pattern of:

<https://app.prefect.cloud/account/ACCOUNTID/workspace/WORKSPACEID/flow-runs/flow-run/FLOWRUNID>

while others use:

<https://app.prefect.cloud/account/ACCOUNTID/workspace/WORKSPACEID/flow-run/FLOWRUNID>

Where the first example has

flow-runs/flow-run

and the second just uses a

flow-run

format. Some of our users get a 404 error on all links of the first type, and others get 404 on all links of the second type. Does anyone know what might be happening or how to fix this?

Sander

10/14/2022, 4:19 PM

Hi, I was wondering if there are any plans for supporting nomad by hashicorp as a kubernetes alternative.

✅ 1

Kyle McChesney

10/14/2022, 4:37 PM

Interesting question, I have the following flow state function (simplified for example)

def flow_updater(flow, current_state, next_state):
    if isinstance(next_state, Failed):
        error_information = {k.name: v.message for k, v in next_state.result.items() if type(v) is Failed}
    else:
         error_information = None
    report(error_information)

When I run this flow locally using

prefect run -p path/to/flow.py

with a simulated error, I get the expected error information, however when I run it on our deployment (not cloud, backend in AWS). I get error_information equal to

{}

(i.e. no task results have a result of type

Failed

). Is there some other type that gets used when running connected to server? (I am using

type(v) is Failed

instead of

isinstance

cause I don want to report

TriggerFailed

, etc)

1️⃣ 1

Pedro Henrique

10/14/2022, 4:39 PM

hi people, i have a problem after update to prefect 2.6.0 and my script stop working the message of error says .fn() for call my function outside the flow

Pedro Henrique

10/14/2022, 4:41 PM

after adding call my functions the following error presented when execute File "/home/cpfl0dev/.local/lib/python3.8/site-packages/prefect/logging/loggers.py", line 110, in get_run_logger raise MissingContextError("There is no active flow or task run context.") prefect.exceptions.MissingContextError: There is no active flow or task run context.

Pedro Henrique

10/14/2022, 4:41 PM

in version 2.4.2 working

Josh

10/14/2022, 4:48 PM

Has anyone ever tried running prefect flows through GCP lambda functions? I’m exploring how to support a mapped run of 1000s of tasks in parallel, and am not sure whether to go the Google Kubernetes Engine route (which Prefect has an agent for) or lamda functions. Would appreciate to anyone’s experience.

Josh Paulin

10/14/2022, 5:16 PM

What should be the expected behaviour if I do the following: 1. Run a flow with caching enabled on the tasks 2. Delete the cache files 3. Rerun the flow This would be using the built in

task_input_hash

as the

cache_key_fn

? On step 3 I’m seeing the flow fail that it can’t find the cache file…

✅ 1

Nace Plesko

10/14/2022, 5:30 PM

Hi again, This is probably very basic python question, but I'm having trouble that Prefect can't find my module. I already have module

database

and a few files in it and all those work fine, but now that I created a new one, the run on the Prefect is saying that it can't find it. I'm probably just missing something obvious, but because of my lack of knowledge of both Python and Prefect I'd really appreciate help on this. Thank you in advance!

✅ 1

Josh Paulin

10/14/2022, 5:54 PM

Noticing a change in behaviour for results since upgrading to 2.6. A task that would return an empty list now returns

None

. Wondering if this is intentional. Minimal example in the thread. Also found out I can’t downgrade from 2.6 to 2.5 because of what looks like some database migrations? Running

prefect orion database reset -y

on a database with 2.5 after upgrading to 2.6 and it fails.

👀 1

:thank-you: 1

✅ 1

redsquare

10/14/2022, 7:54 PM

Hey, how are people dealing with Prefect 2.0 CI/CD and agents with deployments/schedules/params, are there any good github action examples floating around?

✅ 1

Taylor Babin

10/14/2022, 8:32 PM

Hi, I'm having an issue where I was trying to set up a profile with a workspace. I'm able to do this locally. And for some reason I'm stuck in this loop of trying to delete my profile for this account. But everytime i try and delete the profile the profile keeps adding '\\\' and i cant seem to call the profile to delete it. Is there a work around where i can delete the profiles?

✅ 1

Omar Sultan

10/15/2022, 8:41 AM

Hey everyone, im trying to run orion on my local dev machine and keep running into Its Windows 11, Python 3.9 I have the same configuration on another windows machine and its working fine. Any ideas what might be causing this?

Error: Got unexpected extra argument (prefect.orion.api.server:create_app)

✅ 1

Sander

10/15/2022, 10:16 AM

Hi, does anyone know if the prefect-dask lib should be able to handle the operator.KubeCluster from dask_kubernetes?

✅ 1

Kostiantyn Liepieshov

10/15/2022, 10:34 AM

Hey everybody, I have just started with prefect graphql interface, but I cannot find how I should correctly do the pagination with it? Does anybody knows how to paginate response in prefect graphql?

Andrei Tulbure

10/15/2022, 3:11 PM

Hello all. Any idea why the Prefect Flows are freezing when running them in ECS with Prefect Cloud ? They freeze and in the cloud they appear as they are not finished, but they never actually do anything after a certain period. And the ECS machines I use are the largest one possible. ?

Francesco Bartoli

10/15/2022, 4:29 PM

Hi all, I started a fresh new installation with this Docker composition template https://github.com/fraibacas/prefect-orion pointing to the latest build. It is working as expected but the Blocks page is empty. Then using this API call

curl -X 'POST' \
  '<http://prefect-server:4200/api/block_types/install_system_block_types>' \
  -H 'accept: application/json' \
  -H 'x-prefect-api-version: 2.6.1' \
  -d ''

I’m getting few of them available from that page, the Remote File System one doesn’t come up though. Is this expected with the latest version?

✅ 1

Sander

10/15/2022, 8:02 PM

Hi, I’m trying to hook up a min.io tenant running on a k8s as a storage block. Is that supported at the moment? Seems the aws s3-bucket doesn’t have get/put-directory?

✅ 1

Marcos

10/16/2022, 12:51 AM

Hi! I think I found a bug (it happens since 2.6.1 version), when a flow has consecutive

.map

calls, if one task fails in the first map, then all dependent tasks stay in NotReady state. Example flow in the comments

✅ 1

Christopher

10/16/2022, 2:59 PM

I'm dipping my toe back into prefect after last playing with 1.x. I'm trying to get my head around the deployment model. Previously I had a prefect agent running persistently in ECS, which would spin up new containers on demand to handle prefect jobs. I used "docker" storage to do that, which appears to not be a option currently. I can't tell if that's because it's not needed to achieve what I want to do... I'm open to using other storage options like S3, but I guess that's difficult if my flow code has external dependencies? I'm also not much of a python dev so I don't have much of an intuition on how bundling works in the python world!

✅ 1

Fady Khallaf

10/16/2022, 3:31 PM

Hello everybody, I am trying to make the orion db persistent by linking a volume to orion server, but the db still ephemeral and each time the container is crashed, data is missed. here is the manifest file I am using, looking forward to helping me to figure out what is the issue

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prefect
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: prefect
  name: orion
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1  
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.6.0-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        ports:
        - containerPort: 4200
        env:
          - name: PREFECT_API_URL
            value: '<https://mydomain/api>'
        volumeMounts:
          - mountPath: ~/.prefect
            name: prefect
      restartPolicy: Always
      volumes:
        - name: prefect
          persistentVolumeClaim:
            claimName: prefect

✅ 1

Title

Fady Khallaf

10/16/2022, 3:31 PM

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prefect
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: prefect
  name: orion
spec:
  selector:
    matchLabels:
      app: orion
  replicas: 1  
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - name: api
        image: prefecthq/prefect:2.6.0-python3.10
        command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
        ports:
        - containerPort: 4200
        env:
          - name: PREFECT_API_URL
            value: '<https://mydomain/api>'
        volumeMounts:
          - mountPath: ~/.prefect
            name: prefect
      restartPolicy: Always
      volumes:
        - name: prefect
          persistentVolumeClaim:
            claimName: prefect

✅ 1

10/17/2022, 12:17 AM

You're trying to mount your PV to

~/.prefect

, but

does not get expanded by K8s (tilde pointing to homedir is a shell concept), so your volume gets mounted into a folder called

/~

df -h
# Filesystem                Size      Used Available Use% Mounted on
# /dev/storageclass/pvcname
#                           2.9G      6.0M      2.9G   0% /~/.prefect

Try changing

mountPath

/root/.prefect

or setting `PREFECT_HOME`(ref).

🙌 2

Fady Khallaf

10/18/2022, 1:39 PM

Thanks so much @Q, it works properly now

👍 2

View count: 1