https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-getting-started
  • k

    Khuyen Tran

    11/10/2022, 7:39 PM
    Tip of the day: You can add
    wait_for
    to subflows so that they can wait for upstream tasks. In the example below,
    subflow
    didn’t run because the
    fails
    task failed. This feature is available in Prefect 2.6.6.
    :party-parrot: 3
    :gratitude-thank-you: 5
    πŸ™Œ 6
    πŸŽ‰ 2
    πŸ‘ 3
    πŸš€ 1
  • k

    Khuyen Tran

    11/11/2022, 5:32 PM
    Tip of the day: You can now read Prefect documentation in dark mode! πŸŒ™
    πŸ”₯ 4
    πŸ•ΆοΈ 3
    j
    b
    2 replies Β· 3 participants
  • k

    Khuyen Tran

    11/11/2022, 11:05 PM
    Are you trying to wrap your head around a deployment (i.e, what it does, and how to use it)? In this series of short videos, you will get some basic understanding of deployment: β€’ What is a deployment β€’ Create a deployment β€’ Run a deployment
    :gratitude-thank-you: 5
    πŸ™Œ 5
    πŸ‘ 5
    :marvin: 4
    πŸŽ‰ 2
  • k

    Kalise Richmond

    11/15/2022, 4:46 PM
    πŸ“Ί Join us today at 10 AM PST for Prefect Beginners Demo by @Taylor Curran https://www.youtube.com/c/PrefectIO
    :blob-attention-gif: 2
    :party-parrot: 1
  • k

    Khuyen Tran

    11/15/2022, 4:59 PM
    Prefect now supports colored log levels in the 2.6.7 version. 🌈 Test code:
    import asyncio
    from prefect import flow, get_run_logger
    
    @flow
    async def test_async_flow():
        await asyncio.sleep(3)
        logger = get_run_logger()
        logger.debug("a debug")
        <http://logger.info|logger.info>("my info")
        logger.warning("some warning")
        logger.critical("super critical")
        logger.error("error!")
        print('all done')
    
    @flow
    def test_sync_flow():
        test_async_flow()
    
    test_sync_flow()
    View more here. Kudos to @Andrew Huang for this cool feature.
    :terminal: 6
    🎨 3
    🀩 7
  • a

    Anjaneyulu K

    11/16/2022, 1:34 AM
    Hi #prefect-getting-started, Can we create dependency flows in prefect like one after other based on action completion?
    a
    1 reply Β· 2 participants
  • c

    Colin Bieberstein

    11/17/2022, 3:29 AM
    Hi everyone. I see lots of examples where people are using Pandas / SQL Alcehemy to do an extract / Load operation. What I haven’t seen are examples of how to handle data sets larger than memory for these operations. Do you advocate to running pyspark, Dask clusters or is there a mechanism to do something with ACI / ECS fargate so that a just in time just big enough worker can be launched?
    a
    1 reply Β· 2 participants
  • a

    Anderson Luiz Souza

    11/17/2022, 11:44 AM
    Hi all! A basic question: I really would like to replace those random suffixes of flows and tasks with more informative names (for example: datetime now or git commit hash). How could I do this? Thanks!
    p
    k
    +1
    8 replies Β· 4 participants
  • k

    Khuyen Tran

    11/17/2022, 4:11 PM
    Tips of the day: In Prefect 2.6.7, you can add timeout to tasks πŸ• . If
    timeout_seconds=n
    and the task takes longer than
    n
    seconds to execute, Prefect will raise a TimeOut error. View more here. Kudos to @Peyton Runyan for this feature.
    πŸ™Œ 6
    :party-parrot: 1
    ⏲️ 1
    :gratitude-thank-you: 4
  • j

    jmill

    11/21/2022, 7:27 PM
    I want to deploy prefect + UI on a remote machine i own and be able to tunnel into it from my main machine, ideally using docker. is there a good tutorial for this? (e
    βœ… 1
    r
    6 replies Β· 2 participants
  • b

    Boris Tseytlin

    11/28/2022, 3:13 PM
    Hey guys. I am creating a block from code and running
    block.save("name", overwrite=True)
    , but I don’t see it in UI. How can I know whatt is wrong?
  • b

    Boris Tseytlin

    11/28/2022, 3:14 PM
    storage_block = RemoteFileSystem(
            basepath="<s3://prefect-flows>",
            key_type="hash",
            settings=dict(
                use_ssl=False,
                key=config.STORAGE_MINIO_USER,
                secret=config.STORAGE_MINIO_USER,
                client_kwargs=dict(endpoint_url=f"http://{config.STORAGE_MINIO_URL}"),
            ),
        )
        storage_block.save("flow-storage", overwrite=True)
        <http://logging.info|logging.info>("Created storage block")
    m
    2 replies Β· 2 participants
  • k

    Kalise Richmond

    11/29/2022, 4:41 PM
    πŸ“Ί Come join @Matt Conger at 1 PM EST for a Prefect Beginners Demo live on youtube. https://www.youtube.com/c/PrefectIO
    :prefect-duck: 3
    :prefect: 3
    :gratitude-thank-you: 1
    πŸ™Œ 1
  • s

    Sebastian Steele

    11/29/2022, 7:10 PM
    hey everyone question, let say I am importing a helper file
    import helper_file.py
    
    '''
    flow that uses helper_file
    '''
  • s

    Sebastian Steele

    11/29/2022, 7:11 PM
    If i make changes to the helper_file do I have to redeploy my flow?
    r
    1 reply Β· 2 participants
  • j

    Jordy Wegman

    11/29/2022, 8:05 PM
    Hey, is there maybe a video on how to get the UI up and running? I can't seem to get 'prefect orion start' to work. My terminal keeps giving this error. I'm probably doing something stupid
    βœ… 2
    j
    a
    13 replies Β· 3 participants
  • a

    ar

    11/30/2022, 3:00 AM
    asking for some feedback is this simple flow is correct to explain how prefect works to others.
    ❀️ 3
    j
    a
    +1
    8 replies Β· 4 participants
  • k

    Khuyen Tran

    11/30/2022, 4:30 PM
    Tips of the week Prefect logger allows you to leverage Rich to insert their own custom color / styling. Here is an example:
    from prefect import flow, get_run_logger
    
    @flow
    def cow_say():
        logger = get_run_logger()
        <http://logger.info|logger.info>("I'm in [red]danger[/red]!")
    
    if __name__ == "__main__":
        cow_say()
    View more styles in Rich’s documentation.
    :gratitude-thank-you: 4
    πŸŽ‰ 4
    πŸ™Œ 5
    r
    1 reply Β· 2 participants
  • l

    L

    11/30/2022, 5:02 PM
    Is there a page in the documentation showing how to properly implement cicd with github ? I'm kind of confused about how to used deployment yaml files when it comes to automation. Thanks
    βœ… 1
    a
    8 replies Β· 2 participants
  • u

    Uzoma Emuchay

    12/01/2022, 2:26 AM
    Hey all, is there any documentation beyond the getting started page on best practice for my use case. I'm attempting to deploy data bricks jobs that create tasks within a job dynamically based on certain conditions. If anyone can point me in the right direction, that would be greatly appreciated.
  • d

    Dragan

    12/01/2022, 11:10 AM
    Hi all, maybe a basic question, but here goes We have a flow that is triggered to run every hour In random situations we have longer calculations that make the job run for more than an hour Is there a validation that can be added that a flow should not start before previous one finishes, something like
    wait until finish
    if it is running in same flow, we can check triggers, but can we do it on the flow level?
    k
    7 replies Β· 2 participants
  • a

    ar

    12/02/2022, 5:02 AM
    Hi again, may i know based on https://docs.prefect.io/tutorials/deployments/ how can i leverage the use of config.json file ? is it customable on storage like prefect parameters? tq for your advice
    βœ… 1
    j
    2 replies Β· 2 participants
  • j

    Jordy Wegman

    12/02/2022, 8:24 PM
    Hey, I'm working my way trough the tutorial and I'm getting some errors on the 'flow and task configuration'. It keeps telling me it 'expected an indented block'. But the code seems fine as it's copied from the Prefect site. I found this link which explains it well, but it doesn't solve my issue since everything looks good. All other script examples do work for me. Anyone got an idea?
    βœ… 2
    j
    p
    14 replies Β· 3 participants
  • z

    Zack

    12/03/2022, 7:08 PM
    Is there way to pick up cloudwatch events through prefect and communicate the information via email or slack notification?
    βœ… 1
    a
    1 reply Β· 2 participants
  • a

    Anthony

    12/07/2022, 3:41 AM
    So our DevOps team has implemented Prefect 2.0 on Kubernetes. How do I go about connecting to the Kubernetes cluster to build and deploy prefect code?
    βœ… 1
    k
    8 replies Β· 2 participants
  • m

    Matthew Ferry

    12/07/2022, 12:42 PM
    I've been struggling to understand how different pieces of Prefect communicate with each other. Is this an accurate representation of the basics? I want to make sure I know where traffic is going to ensure my deployment has the right networking and access. If I have this right, I'd like to expand it to cover the development workflow.
    😍 2
    j
    1 reply Β· 2 participants
  • d

    Daryl Thomas

    12/08/2022, 2:09 PM
    I have run prefect server on a single VM, On accessing the ui everything seems to be loading very slowly. In the console i see a lot of Err Connection timed out for the 4200 graphql. I have followed the steps in https://docs-v1.prefect.io/orchestration/server/deploy-local.html#ui-configuration. im running on an aws ec2 instance. When i run the sample flow it is stuck at waiting for flow runs
    1️⃣ 1
    c
    4 replies Β· 2 participants
  • Delete a custom Prefect Block
    n

    Nils

    12/08/2022, 2:27 PM
    I was playing around with custom blocks. Registered a couple of with the Python command
    register_type_and_schema()
    . However, now I want to delete those blocks but I can not find a way to do so..
    βœ… 1
    r
    1 reply Β· 2 participants
  • b

    Bryce Morrow

    12/08/2022, 6:07 PM
    Hey everyone, been playing around with prefect the last few days and really enjoying using it so far, however, I'm running into some issues with my workflow / setup and would appreciate some feedback / help πŸ™‚ My environment is setup like this β€’ I'm working in jupyter-lab environment (installed globally). β€’ I have a python 3.10.8 conda kernel installed in this environment, and I'm using this kernel to run my notebooks. My workflow is like this β€’ Define a number of tasks. Some tasks involve expensive api calls, so for these I set
    persist_result=True
    , which helps speed up my workflow considerably. For simpler data transformations on the results of these expensive tasks, I don't tend to persist the results. β€’ For the tasks that I am persisting, their implementations often change as I iterate on ideas. So, for example, maybe I change the implementation of the task
    one
    to return
    10
    instead of
    5
    . Whenever I change the implementation of one of my tasks with persisted results, I change the cache key. For this example, I'd change
    lambda *args: "one"
    to
    lambda *args: "one1"
    . A simplified example:
    @task(persist_result=True, cache_key_fn=lambda *args: "one")
    async def one(): 
        return 5
    
    @task(persist_result=True, cache_key_fn=lambda *args: "two")
    async def two(): 
        return 2
    
    @flow(cache_result_in_memory=False)
    async def flow_test():
        a = await one()
        b = await two()
        return a + b 
    
    await flow_test()
    This seems to be working at the start, but eventually, I start running into an issue where tasks fail to run due to an error in decrypting their block data. I'll share the exact error log in the replies. I haven't yet determined the exact cause of the problem but I have some suspicions / notes. The first concerning thing is that when I re-run this cell, say after changing a task implementation, I get some warnings that look like this:
    /opt/miniconda3/envs/flywheel/lib/python3.10/site-packages/prefect/tasks.py:206: UserWarning: A task named 'one' and defined at '/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py:1' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:
    
     `@task(name='my_unique_name', ...)`
      warnings.warn(
    /opt/miniconda3/envs/flywheel/lib/python3.10/site-packages/prefect/tasks.py:206: UserWarning: A task named 'two' and defined at '/var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py:5' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:
    And this makes sense, since I'm defining a task with the same name again. However, when I look at the path
    /var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/ipykernel_90535/3130328684.py
    , it doesn't exist. However, there are different folders stored at the path
    /var/folders/5c/hf9_blgj3jbbxxvkw_s9snsw0000gn/T/
    with names of the form
    ipykernel_<some_number>
    . If I look at the contents of these directories, they contain small snippets of code from my notebook but they aren't the task definitions themselves usually. So I think the ipykernel is storing this information temporarily but then it changes locations, contents, disappears, etc. I've also tested running the notebook end to end, having everything work fine, then restarting the conda kernel and re-running the notebook. This causes the error I'll share below. This strongly suggests to me that the way the kernel is managing the location of the notebook source code is the culprit here. Another point is that if I clear the orion database using the command
    prefect orion database reset -y
    then my issues are resolved. However, I would like to determine the issue and fix it in some better way.
    3 replies Β· 1 participant
  • t

    Tony

    12/09/2022, 3:04 PM
    Good morning Everyone. It's been a while but I decided to upgrade to Orion. I see that a lot has changed. I am perplexed by a few steps as i'm trying to get Orion started. I am using VENV into which I have installed prefect 2.7.0. I am having a lot of trouble setting the API addresses, using and setting a profile, and just getting things to work. AT first, the prefect config set PREFECT_API_URL command does not seem to be working. I then export PREFECT_API_URL, while showed correctly as I started prefect Orion, but in either case, once browsing to the dashboard, I get an error saying 127.0.0.1:4200/api is not available. So i seek help in ALL the steps necessary to get this working and available. I have scoured the prefect.io docs, but the section that deals with "Running Orion for the First time" seems to be completely missing. I am anxious to explore the new features that are available in 2.0(Orion). Thanks for any help you can provide.
    βœ… 1
    p
    r
    65 replies Β· 3 participants
Powered by Linen
Title
t

Tony

12/09/2022, 3:04 PM
Good morning Everyone. It's been a while but I decided to upgrade to Orion. I see that a lot has changed. I am perplexed by a few steps as i'm trying to get Orion started. I am using VENV into which I have installed prefect 2.7.0. I am having a lot of trouble setting the API addresses, using and setting a profile, and just getting things to work. AT first, the prefect config set PREFECT_API_URL command does not seem to be working. I then export PREFECT_API_URL, while showed correctly as I started prefect Orion, but in either case, once browsing to the dashboard, I get an error saying 127.0.0.1:4200/api is not available. So i seek help in ALL the steps necessary to get this working and available. I have scoured the prefect.io docs, but the section that deals with "Running Orion for the First time" seems to be completely missing. I am anxious to explore the new features that are available in 2.0(Orion). Thanks for any help you can provide.
βœ… 1
p

Peyton Runyan

12/09/2022, 3:04 PM
Hey there!
That URL is a bit tricky. If you look at the output when you start Orion, it specifies that the UI is at
127.0.0.1:4200
That's gotten me before too since the
/api
url is the most obvious one. I think it's worth us thinking about adjusting where we place the links. edit - looks like they been adjusted a bit already, but I definitely get the confusion there.
:thank-you: 1
t

Tony

12/09/2022, 3:12 PM
once i executed export PREFECT_API_URL = http://192.10.0.5:7/api, it shows correctly on the ORION startup screen. The problem is when going to the dashboard: It says can't connect to Orion api at http://127.0.0.1/api . I am curious why would the dashboard use localhost as the default address and how/where does the dashbord get that setting?
So I can get Orion to start on 192.10.0.5:7272. Docs, dashboard all seems to respond. However, the /ip show a real fancy 404 ; Page Not Found error, completely in the style of Prefect.
p

Peyton Runyan

12/09/2022, 3:15 PM
can you paste the results of
prefect config view
?
t

Tony

12/09/2022, 3:15 PM
At this point, i gotta say I am completely missing something obvious...
PREFECT_PROFILE='default' PREFECT_API_URL='192.10.0.5:7272' (from env) PREFECT_HOME='.prefect' (from profile)
this is on Ubuntu server
prefect profile ls shows *default however prefect profile inspect shows Profile 'default' is empty
p

Peyton Runyan

12/09/2022, 3:18 PM
hmmmm - give me just a moment
t

Tony

12/09/2022, 3:19 PM
Thank you, Sir.
p

Peyton Runyan

12/09/2022, 3:20 PM
Set your
PREFECT_UI_URL
to the same IP/Port combo, minus the
/api
and then try to restart Orion.
t

Tony

12/09/2022, 3:23 PM
I apologize, my paste did not paste correctly. Once I executed export PREFECT_UI_URL=192.168.0.5:7272 it starts. I have to start it this way: prefect orion start --host 192.10.0.5 --port 7272 Here is my startup screen Configure Prefect to communicate with the server with: prefect config set PREFECT_API_URL=http://192.10.0.5:7272/api View the API reference documentation at http://192.10.0.5:7272/docs Check out the dashboard at http://192.10.0.5:7272
p

Peyton Runyan

12/09/2022, 3:24 PM
Is it working now?
t

Tony

12/09/2022, 3:24 PM
that is weird, it is now working.
πŸ”₯ 1
although this is still showing 404: page not found: http://192.10.0.5:7272/api
p

Peyton Runyan

12/09/2022, 3:25 PM
Oh yeah - the API is just an API. No UI associated with the API
t

Tony

12/09/2022, 3:27 PM
I still don't feel very confident that I can get this to work. I feel like there are some initial steps needed with the profile. do you have official instructions on how to get orion started for the first time?
in 1.0, we had to edit a config.toml file which despite its clunky-ness worked really well.
Also, the /setting page is showing straight html, and the text editor has the same html as a single character vertical:
p

Peyton Runyan

12/09/2022, 3:30 PM
Oh interesting - I can't reproduce that one
What's your version number?
Regarding the docs, I think there's possibly a gap on self-hosted UI
t

Tony

12/09/2022, 3:30 PM
and, finally, I am worried that I am trying to do something unconventional running this in a virtual enviornment in python.
2.7.0 on python 3.8 on ubuntu
p

Peyton Runyan

12/09/2022, 3:31 PM
Here are the getting started docs, but I think there's a gap on the UI environment variable pieces: https://docs.prefect.io/getting-started/overview/#get-started-with-prefect-2
running this in a virtual enviornment in python
Running it in a virtual environment is pretty conventional. What's your usecase?
2.7.0
- can you try 2.7.1? I don't keep up with the UI releases, but this may have been a bug. It's not present on my machine on 2.7.1
and you still have a profiles toml
cat ~/.prefect/profiles.toml
t

Tony

12/09/2022, 3:36 PM
thanks again, Peyton. What's odd is I just installed this yesterday: pip install prefect. Why it chose 2.7.0 I don't know. I'll give that a try
p

Peyton Runyan

12/09/2022, 3:36 PM
No worries man! We do releases every Thursday, so you probably just missed it πŸ™‚
t

Tony

12/09/2022, 3:36 PM
oh, ok. that is good to know.
should I do a pip upgrade? to a complete reinstall
p

Peyton Runyan

12/09/2022, 3:37 PM
And keep me posted. The migration from 1 to 2 can have a bit of friction at first, but once you get moving it should feel pretty great
t

Tony

12/09/2022, 3:37 PM
or do a complete re-install
p

Peyton Runyan

12/09/2022, 3:37 PM
Just upgrading will be fine
t

Tony

12/09/2022, 3:37 PM
Thanks again, good sir!
πŸ‘ 1
verified i'm now on 2.7.1 In following your instructions, I run prefect orion start It defaults to 127.0.0.1:4200/api dashboard on 127.0.0.1:4200 even after i ran prefect config set PREFECT_API_URL=http://192.10.0.5:7272/api
it seems the only way to start orion correctly is by adding the --host and --port parameters at the prefect orion start command
p

Peyton Runyan

12/09/2022, 3:47 PM
Oh - I think I may have misunderstood you earlier then. This is expected behavior. It allows you to have profiles that can connect with prefect instances in multiple locations, including cloud, without having the profiles interfere with starting a new Orion instance.
t

Tony

12/09/2022, 3:52 PM
so here is all I am trying to do.
I have a 1.0 flow I want to upgrade. It is currently in production.
So I cannot upgrade the current env. I simply want to start Orion on 192.10.0.5, which has pelias running on port 4200. Then I want my agent to connect to it an run the 2.0 version of my flow.
it seems like you docs expect Orion to be the only thing to be running on the server...
Again, the seems to be the only thing that gets me anywhere: prefect orion start --host 0.0.0.0 --port 7272 But I get an "Unknown Error" when trying to create a work queue and the /settings page is jacked. I still don't understand the validity of defaulting the dashboard to use the loopback IP, this is ubuntu and the server has no gui to run a browser on...
p

Peyton Runyan

12/09/2022, 4:22 PM
where's your agent running?
t

Tony

12/09/2022, 4:57 PM
on a windows machine on a different subnet
the 1,0 agent, that is.
I don't have a 2,0 agent, yet.
p

Peyton Runyan

12/09/2022, 5:25 PM
In the near term I'd run through the getting started tutorials. You'll have a better feel for how each of the pieces interact with each other which will make this process a bit smoother: https://docs.prefect.io/tutorials/first-steps/
If you get started with them on your local machine it'll tighten the learning curve a bit, and it should be pretty easy to move forward from there
t

Tony

12/09/2022, 5:36 PM
thank you Peyton. But I am stuck on paragraph 2: Prerequisites: I can't go further becuause I can
't get Prefect 2 installed
in 1.0 all this was done with a docker setup.
p

Peyton Runyan

12/09/2022, 5:42 PM
even on a local machine?
t

Tony

12/12/2022, 2:51 PM
Still can't get it to work on this (ubuntu) server. Are there any startup logs and if so where are they? And can you add a debug flag to starting orion do send error messages to the putty terminal? Here is what is working: export PREFECT_API_URL=192.10.0.5:7272/api prefect orion start --host=0.0.0.0 --port=7272 Then the UI comes up: In chrome: http://192.10.0.5:7272/flow-runs However, I get a cryptic error when creating a work queue. And the settings page looks like an HTML string...
βœ… 1
r

Ryan Peden

12/12/2022, 3:12 PM
Hi Tony! A couple of things to try: β€’ I think
PREFECT_API_URL
needs to be a URL here, not just an IP address. Does setting it to
<http://192.10.0.5:7272/api>
help? β€’ If you still run into errors, does setting
PREFECT_ORION_UI_API_URL=<http://192.10.0.5:7272/api>
make a difference?
t

Tony

12/12/2022, 3:19 PM
Yes, setting export PREFECT_ORION_UI_API_URL=http://192.10.0.5:7272/api Allowed me to create my work queue and the settings page looks more like settings. I think that was it. I highly recommend this be added to the manual and move all this to a new topic called "Starting Orion UI for the first time" Thanks for all your help, Peyton and Ryan!
❀️ 1
πŸ”₯ 2
r

Ryan Peden

12/12/2022, 3:25 PM
Agreed! I'm adding it to our content tracker. And beyond that, I'll see if we could make some small adjustments
prefect orion
CLI command to make this scenario more straightforward. We sincerely appreciate your feedback! πŸ˜„
Since you mentioned Docker earlier, I just wanted to share this Docker Compose example I made in case it is useful: https://github.com/rpeden/prefect-docker-compose You'd need to add
PREFECT_ORION_UI_API_URL
to the
orion
container but aside from that, it should work for you.
t

Tony

12/12/2022, 4:16 PM
Thanks Ryan. This is definitely the direction I want to go.
r

Ryan Peden

12/12/2022, 4:19 PM
You're welcome! πŸ˜„ I've been planning a follow-up to that repo that shows how to add a production-ready web server like Nginx to make it easy to do things like add an SSL cert. Would that be useful to you as well?
t

Tony

12/12/2022, 4:34 PM
Absolutely. Thank you!
Good new, good news. I now have Orion running on my ubuntu server as a docker stack. I also have a windows agent running and connected. So here is my next question, one i have been asking since Prefect 1.0, what is the engineering intention for Docker Agents running as a service/Daemon? I was told that the best way to do this is to configure/schedule flows to deploy/execute as docker containers. There were 2 consequences to that: One, there were a lot of "dead" containers to clean up. Two, it was difficult for me to make sure the development dependencies were available AFTER the flow containers were spun up. Just picked the engineering team's collective brain...
View count: 6