https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-ui
  • d

    DK

    03/08/2022, 7:26 PM
    Hello! Is there a way to schedule the Last Tuesday of the Month in the Cron scheduler on the UI? like 2L in the Day of the Week field?
    k
    • 2
    • 3
  • d

    David Beck

    03/08/2022, 10:22 PM
    Hi Prefect Team! I'm looking to disable cloud logging since our implementation will be routing our logs via k8s directly to splunk. I tried setting an env value of
    PREFECT__CLOUD__SEND_FLOW_RUN_LOGS=False
    in our global container config which I see being recognized by Prefect Cloud, however there are logs still populating for the run (see attached). Do I perhaps have the wrong value defined or could I be missing something?
    k
    m
    +2
    • 5
    • 7
  • d

    Daniel Nilsen

    03/11/2022, 12:16 PM
    Hi I deployed prefect to my Kubernetes cluster using the Helm chart. It has been working for a while now. But I am now getting this error when visiting the UI:
    TypeError: Cannot set properties of undefined (setting 'role')
    Anyone know what this means?
    a
    • 2
    • 10
  • a

    Amogh Kulkarni

    03/14/2022, 8:16 PM
    Hi Prefect Team. I have a flow with tasks task1->task2->task3->task4. It is scheduled to run everyday and the flow runs just fine. But I want to use the UI to do the below activities once in a while. Can you let me know if I can do the below activities? 1. Sometimes I want to run just one task. Is it possible to run task3 from the UI? If yes, then how do I do that? 2. Assume that task2 has failed. If I run the flow once again, all the tasks will run from the beginning. Can I re-run the flow from the UI by ensuring that the tasks will run only from the failed task? i.e task1 will be skipped because it was successfully completed last time. the flow will run from task2 onwards.
    k
    a
    • 3
    • 5
  • r

    Russell Brooks

    03/16/2022, 10:53 PM
    Hi. I am using prefect server 1.0. Flows and tasks have been running ok. But now they only show log entries for the very beginning and end. I.e. I lost the richness of all those logger.info I made. Logs do show up if I run locally. What have I done and how to fix?
    k
    • 2
    • 18
  • r

    Russell Brooks

    03/16/2022, 10:56 PM
    Also on a separate topic, the need to know the slug in advance and to know what quirky append it has, e.g. “-1” or “-copy” is a problem when using create_flow_run and get task results. I think you have an old ticket for this but I lost a couple of hours on this yesterday. Also when to use startflowrun or create_flow_run is a confusing to me. I think is is in the wrong forum as it is not UI. I am new to slack. Can someone help move it?
    k
    • 2
    • 15
  • r

    Ruslan

    03/19/2022, 2:14 PM
    Hi! How to get project name from running flow?
    a
    • 2
    • 1
  • y

    Yury Cheremushkin

    03/21/2022, 4:02 PM
    Hi! I've run into an issue with parameters set in Cloud UI. When I update existing parameters via UI it doesn't take effect immediately. As far as I understand the reason is that Prefect Cloud schedule 10 runs upfront. So right after parameters update I need to wait 10 runs before my changes will be applied. Or I need to change either something in a Flow's code, or in a Flow's schedule so Prefect Cloud will re-schedule this 10 runs with updated parameters. It seems very strange behaviour. Is there any way to force this "re-scheduling" after each update in parameters via UI? To be more precise. Let's say I've got such Flow:
    import pendulum
    from datetime import timedelta
    from prefect import Parameter, task, Flow
    from prefect.schedules import Schedule
    from prefect.schedules.clocks import IntervalClock
    
    schedule = Schedule(clocks=[IntervalClock(start_date=pendulum.datetime(2021, 9, 28, 8, 0, 0, tz="UTC"), interval=timedelta(minutes=10))])
    
    params = {
        "message": "message version 1"
    }
    
    with Flow("test_changing_parameters_from_ui", schedule=schedule) as flow:
        p = Parameter("params", default=params)
        task(lambda x: print(x))(p["message"])
    I register it in Prefect Cloud. And right after registration Prefect Cloud schedules 10 runs of my Flow. After that i change
    message
    in this Flow via Cloud UI. For example, set
    "message": "message version 2"
    . I expect that in the next run updated message will be printed for me. But in fact I need to wait for 10 more runs before i'll be able to see my "message version 2".
    k
    • 2
    • 19
  • a

    Architha Rao

    03/23/2022, 6:41 AM
    Hi. I have prefect server set up. I see the scheduled flows are submitted bu the agent never moves it to run state. Any idea why and what the fix could be?
    ✅ 1
  • a

    Adam Knust

    03/31/2022, 6:02 PM
    Hey all! We are running Prefect/Apollo/Hasura/Prefect-Agent (v 0.14.16). on Kubernetes on EC2 in AWS We run our Prefect-Agents as Local, not as K8S Agents. We find that when a Pod dies and is recreated by K8S a new Agent is created, registered, and becomes visible in the UI; however, the Agent that was killed when the Pod died turns into a Zombie and never gets cleaned up until we manually hit the Clean button a couple of times. When the agent count gets very high the entire platform bogs down, and the UI has trouble rendering. I thought that Towel was a service to help with this kind of thing - is there something special we need to do to get this set up?
    k
    • 2
    • 5
  • l

    Leonardo Rocha

    04/04/2022, 8:49 PM
    Hey all! I recently came across this page (https://orion-docs.prefect.io/concepts/task-runners/) that showed a way of using dask.annotation to set the priority of each task, but I can't get it to work. Since I couldn't import DaskTaskRunner for the life of me I tried using LocalDaskExecutor, but I'm not sure if it does the same thing:
    @task
    def file_write(x):
        print(x)
        with open('test_result.txt', 'a') as f:
            f.write(str(x))
        return x
    
    with Flow("priority_test", executor=LocalDaskExecutor()) as flow:
        with dask.annotate(priority=-10):
            written1 = file_write(2)
        with dask.annotate(priority=0):
            written2 = file_write(1)
        with dask.annotate(priority=10):
            written3 = file_write(0)
    I expected it to write "012", but am consistently getting "201"
    k
    m
    • 3
    • 2
  • d

    Dev Dua

    04/05/2022, 4:55 AM
    Hi all! I’m using Prefect 2.0b and I am running it on my dev machine.. I want to access the Orion UI from another machine but it shows me no data because it’s making API calls to localhost:4200 instead of the API URL I set (using
    PREFECT_API_URL
    ) to
    0.0.0.0:4200/api
    Is there another way to accomplish this?
    a
    • 2
    • 6
  • r

    Russell Brooks

    04/06/2022, 7:37 AM
    I am on v1.2 and Core. If I used cli to register a flow from a python script to a remote server it seems to work (except it ignores the scheduler). But when I use flow.register from within the code it gives an error. Graphql validation fails unknown type “register_tasks_input”. Any idea what I am doing wrong?
    a
    k
    • 3
    • 3
  • m

    Meng Si

    04/06/2022, 4:52 PM
    Hi there, I have a large data set that I want to run. It’s taking a long time. I want to break down my dataset to smaller batches, and run each batch in parallel. Is there a good way to do it using Prefect workflow? Thank you!
    k
    • 2
    • 10
  • s

    Sam Garvis

    04/06/2022, 6:30 PM
    Hi all! We are transitioning our python scripts and others from Heroku to Prefect. We have prefect cloud, but what is the best way to run a prefect agent with the needed dependencies for our python files. We were just using prefect docker agent start on the VM, and pointing our flows to a dockerfile with the correct dependencies. This was working but would always stop working after a few flow runs. Does anyone know of any source code for a good docker compose for prefect agents to run on a VM, or where we should look for this info? Thanks!!
    k
    • 2
    • 9
  • v

    Vadym Dytyniak

    04/07/2022, 12:41 PM
    Hi. Is Prefect team know that there is issue with Flow Logs and scrolling there. In Live mode it sometimes automatically scrolling up and down many times.
    a
    • 2
    • 1
  • r

    RAISS Zineb

    04/07/2022, 2:05 PM
    I am a beginner with Prefect, so I searched the internet and followed tutorials to see UI and server, during installation and configuration I ran the command "prefect server start" but I got a message of error, have you please on that? I have docker and docker compose install
    k
    • 2
    • 15
  • a

    Atsushi Saito

    04/07/2022, 5:50 PM
    Hi, folks, a work flow finished correctly but UI does not show anything.
    k
    • 2
    • 7
  • k

    Konstantin

    04/12/2022, 2:29 PM
    Hi, team Prefect! Updated prefect-core to 1.2.0 version. The problem is that I don't see registered FLOWs in the UI
    :discourse: 1
    a
    y
    • 3
    • 17
  • r

    RAISS Zineb

    04/15/2022, 2:35 PM
    Hello , when I restarted my laptop, I tried to run Prefect on local, so I typed the command perefct server start, but an error showed me "prefect: command not found" do you have an idea?
    k
    a
    a
    • 4
    • 36
  • t

    Tim Helfensdörfer

    04/19/2022, 7:49 AM
    Hey everyone! Everything seems to be setup correctly so far, but I get the error "Flow is not contained in this Storage". Even though, I can docker exec into the storage in question and execute the python file manually. I've written a more detailed description with the actual code in the GitHub discussion board https://github.com/PrefectHQ/prefect/discussions/5680 - but did not get any responses so far. Does anyone have any idea what I did wrong?
    a
    • 2
    • 2
  • a

    Arkady K.

    04/19/2022, 1:17 PM
    Hi all,
  • a

    Arkady K.

    04/19/2022, 1:18 PM
    Is there a way to schedule on the dask executor and still use the Prefect Server GUI to see the flows ?
    k
    • 2
    • 10
  • d

    Domenico Di Gangi

    04/19/2022, 1:37 PM
    Hi all, It seems that in the orion-ui Run Time is not an average, as the "AVG" after the duration seems to indicate, but a sum of run times. Is this meant to be an average run time or a total run time?
    k
    • 2
    • 3
  • b

    Benoit Chabord

    04/19/2022, 8:43 PM
    I am wondering if there is a pattern that exists in prefect 2.0 to say that a Flow had nothing to do. My use case is that I have a Flow checking every 5min for files in the s3 bucket of a partner and when they are files I process them with a series of tasks. The dashboard is showing a lot of successful tasks. Would using a subflow be a good pattern there? One flow "checking" and one flow "processing" ? Maybe having a sub flow status "success nothing to do" would work. I also have another use case, in the same flow I am processing those files and 0.1% of the records are not "valid" I am logging them as warnings but wondering if there is a way to report on those "invalid" records, seems to be a pretty common use case
    👍 1
    k
    j
    • 3
    • 10
  • j

    Joseph Mathes

    04/22/2022, 4:57 PM
    Is there a way to see the content of a flow I've registered to prefect cloud? To double check what version of it is up there Edit: IIRC the source code of the flow never gets uploaded, right? So I guess a better version of the question would be "can I see where the flow is stored so I can grab and unpickle it myself"
    k
    b
    • 3
    • 4
  • s

    Sander

    04/23/2022, 6:18 PM
    Hi, is it correct that setting the task name using task_run_name from prefect 1 does not exist yet in prefect 2?
    k
    a
    • 3
    • 11
  • s

    Sander

    04/23/2022, 6:36 PM
    Does anyone know if we should be able to click through a task run and see it's log? Currently it seems the log is only on the flow run level?
    a
    • 2
    • 7
  • s

    Sander

    04/23/2022, 7:49 PM
    I'm not sure if this is the right channel, but I was wondering why Postgres 12 is not sufficient for Orion?
    m
    a
    • 3
    • 5
  • b

    Blake Enyart

    04/25/2022, 3:40 PM
    Perhaps my Google search skills aren't quite up to the task, but I'm trying to find the IP range that Prefect Cloud uses to communicate with other services. Right now, we were running into difficulties using the AirbyteConnectionTask without opening our AWS Security Group up to public access for the task to work which we aren't super excited about. I have the logs added in the thread, but in summary, it fails and shows the below stack trace when we only have port 22 open to a specific IP range and it succeeds when we open up the SG. Has anybody else run into this using the AirbyteConnectionTask?
    k
    k
    • 3
    • 15
Powered by Linen
Title
b

Blake Enyart

04/25/2022, 3:40 PM
Perhaps my Google search skills aren't quite up to the task, but I'm trying to find the IP range that Prefect Cloud uses to communicate with other services. Right now, we were running into difficulties using the AirbyteConnectionTask without opening our AWS Security Group up to public access for the task to work which we aren't super excited about. I have the logs added in the thread, but in summary, it fails and shows the below stack trace when we only have port 22 open to a specific IP range and it succeeds when we open up the SG. Has anybody else run into this using the AirbyteConnectionTask?
Removed the EC2 IP for potential security impact at this time from the stack trace.
Abbreviated stack track output.txt
k

Kevin Mullins

04/25/2022, 3:46 PM
Prefect Cloud itself wouldn’t communicate to other services. This would all occur on the Prefect Agent that picked up the flow. For instance, in my environment I use the Kubernetes agent which gets the egress IP I have configured for the cluster. What type of agent are you using?
👍 1
b

Blake Enyart

04/25/2022, 3:55 PM
We are using the local agent. This was my understanding of the communication mechanism in Prefect as well. With that, what should we do? Because it does work when we open the security group and it doesn't work when we only have it open to port 22. Is there any other information I can send over or details on this that would provide additional context?
It is specifically the health checks for Airbyte sent to the EC2 instance we have Airbyte hosted on that are failing. So when we have public communication it works fine, but when we shut it down to just port 22 it fails
k

Kevin Mullins

04/25/2022, 3:57 PM
If I am understand correctly, that local agent had to be installed somewhere (your laptop for dev, a server or vm, ec2, etc.). Where the local agent was installed would dictate the networking configuration. Do you happen to know where the local agent was installed/is running?
b

Blake Enyart

04/25/2022, 3:59 PM
The local agent is installed on the EC2 instance hosting Airbyte
k

Kevin Mullins

04/25/2022, 4:01 PM
Gotcha, that’s interesting, It should be using that EC2's public ip address. I haven’t worked with AWS in a minute but I think you may want to look at how that EC2 instance is configured to see if it is part of a virtual network of some sort. That virtual network should be added to networking rules to allow it access. In this case since it sounds like the same server I’m wondering if it’s bouncing out and getting a public IP address
As a test, you could find out what the public IP address is of the EC2 instance and add a rule to the security group allow it for that ip address to the necessary ports that Airbyte uses.
b

Blake Enyart

04/25/2022, 4:06 PM
Just to make sure I'm understanding this, the security group attached to the EC2 instance which hosts Airbyte should have an ingress rule which allows traffic from the public IP address attached to the EC2 instance? Is that correct?
k

Kevin Kho

04/25/2022, 4:07 PM
The discussion so far is right that the Agent machine just needs HTTP out in order to ping Prefect Cloud, and then it finds a flow to run and then execute it. So you shouldn’t need the IP ranges of Prefect Cloud in order to execute anything. I am confused why the Airbyte task would go through the internet if they are in the same box? Wouldn’t it hit a local port on the same machine? You can do
localhost
for the server host right?
k

Kevin Mullins

04/25/2022, 4:07 PM
I was gonna say my guess is because the URL is using the full public EC2 instance IP address.
:upvote: 1
Yea, you could technically use “localhost”, but seems like there may be some scalability type decisions there, can you always assume airbyte will be on the same machine as the agent, do you need multiple agents/airbyte servers, etc
Prolly should have led with that. I was thinking from the perspective of a solution that would work if it started on the same machine but moved to another or expanded in other ways, using vnet type rules would prolly be ideal between ec2 instances imho
k

Kevin Kho

04/25/2022, 4:12 PM
Ahhh I understand what you are saying
k

Kevin Mullins

04/25/2022, 4:13 PM
But, that was a bit presumptuous of me lol. I could see labeled agents always on the airbyte hosts where localhost would be fine. So that may be the best first step. Sorry for any confusion
View count: 9