• Scott Zelenka

    Scott Zelenka

    2 years ago
    I'm registering a Flow with two Schedules, my intent is to have the same Flow run twice but with different Parameters for each run:
    schedule=Schedule(
                clocks=[
                    IntervalClock(
                        start_date=pendulum.datetime(2020, 1, 1, tz='UTC'),
                        interval=datetime.timedelta(days=1),
                        parameter_defaults=dict(
                            sfdc_listener_name='CustomerService',
                            sfdc_report_url='...'
                        )
                    ),
                    IntervalClock(
                        start_date=pendulum.datetime(2020, 1, 1, tz='UTC'),
                        interval=datetime.timedelta(days=1),
                        parameter_defaults=dict(
                            sfdc_listener_name='OneSupport',
                            sfdc_report_url='...'
                        )
                    )
                ]
            ),
    However, it seems that when this registered with Prefect Cloud, only the first Schedule is setup. Do I need to create two different Flows to have the same logic execute with different parameters on a schedule?
    Scott Zelenka
    Kyle Moon-Wright
    +1
    6 replies
    Copy to Clipboard
  • b

    Ben Fogelson

    2 years ago
    Is there a way to have a
    Parameter
    whose default is the value of another
    Parameter
    ? Something like
    from prefect import task, Flow, Parameter
    
    @task
    def add(x, y):
        return x + y
    
    with Flow('flow') as flow:
        a = Parameter('a', default=0)
        b = Parameter('b', default=a)
        c = add(a, b)
    b
    Chris White
    +1
    8 replies
    Copy to Clipboard
  • c

    Christopher

    2 years ago
    Hi folks, I lead an ML and data science team, and I’m evaluating a bunch of tools in this space. Prefect looks very nice indeed, but I am having trouble understanding exactly what I get with a paid plan vs the open source core. I have looked at all the feature lists I can find, but I guess I’m missing one of those classic “feature comparison tables” that one often sees. Can someone point me in the direction of something like that?
    c
    Jeremiah
    +2
    10 replies
    Copy to Clipboard
  • d

    dherincx

    2 years ago
    Hello everyone! I'm using a
    switch
    . Is it possible to access the results of the switch in a subsequent task? For example, I have a list of data (some of which contain latitude/longitude and others don't). My switch consists of getting lat/long for the records that are missing it, while records that do have coordinates are simply returned. How can I access the final, joined list after the switch?
    d
    2 replies
    Copy to Clipboard
  • p

    Pierre CORBEL

    2 years ago
    Hello there 👋, I'm a new user of Prefect, coming from the Airflow world when I happily used it for 3+ years. 🎂 I want to begin by saying that the project, the quality of the code and the quality of the documentation are outstanding 🤩 But I need some help about finding the good way to use it and the good practice 🤓 I have a standard ELT flow with a big json file (1GB) as input. For my task to run successfully on my medium machine, I combine ijson and iterator to read and write the file on disk chunk by chunk and not overload the memory (I can't stuck a 1GB json dict in memory) Then I load the file directly into my DB, without passing via python. What is the prefect-way of handling a similar usecase here? 🤔 Prefect encourage passing data from task to task in-memory but here, I offload it to disk and only pass the path of the file between task. Is there a way to pass an iterator between task instead of a single object? One way I'm thinking of doing it in a industrialized way is maybe to share a file cache between tasks. What do you guys think about it?
    p
    1 replies
    Copy to Clipboard
  • Scott Zelenka

    Scott Zelenka

    2 years ago
    Has anyone had success using a large Docker base image? I'm trying to convert a Selenium script into a Prefect Flow .. for Selenium, it requires a full install of Google Chrome which makes the base image around 700MB with a ton of layers
    selenium/standalone-chrome
    . But when I try to build through the CLI, it's giving me trouble (essentially spun the CPU fan for an hour, and eventually gave up because
    Traceback (most recent call last):=============================>]  143.3MB/143.3MB
      File "example-selenium.py", line 473, in <module>
        parameters=dict(
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/core/flow.py", line 1419, in register
        no_url=no_url,
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/client/client.py", line 623, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/core/flow.py", line 1228, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 282, in build
        self._build_image(push=push)
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 312, in _build_image
        self.pull_image()
      File "/opt/anaconda3/envs/fastapi-async-sqlalchemy/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 520, in pull_image
        raise InterruptedError(line.get("error"))
    InterruptedError: write /var/lib/docker/tmp/GetImageBlob079036145: no space left on device
    Scott Zelenka
    Chris White
    +2
    6 replies
    Copy to Clipboard
  • p

    Pierre CORBEL

    2 years ago
    Hello there 👋, I got my first ELTC (Extract, Load, Transform, Cleanup) flow working on Prefect Core 🤩 But now I got a question: I would like to run the flow at the launch of the python app, but also to apply a schedule for every night at 3. So basically, I want a
    CronClock
    combine to a launch at startup. Is there an easy way to achieve this? 🧐 I think the fact that the scheduler wait for the next schedule raise the need for a "fire_at_start" parameter. In my case, if my schedule is programmed every day at 3 o'clock, I don't want to wait X hours before my DB is loaded up with data. 🕒 Does it make sense?
    p
    Scott Zelenka
    2 replies
    Copy to Clipboard
  • Scott Zelenka

    Scott Zelenka

    2 years ago
    Hi Cloud users, interested in your suggestions. I have a Flow that executes great on my local dev machine. However, some of the logic within the Flow leverages a different application (XVBF & Google Chrome) that's also installed on my local dev machine. When I attempt to serialize the Flow into a Docker image, it doesn't run because the required application(s) aren't installed in the Docker image the
    .register
    generates. I can get the dependent application(s) to run inside a Docker image with supervisord, but that would require the entrypoint of Docker to somehow trigger supervisord before it attempts to execute the Flow. I couldn't find an API in Prefect to allow for this type of functionality? Another option is to provide the dependent application(s) as a native cloud service, and communicate over the network to access the same way I do it on my local dev machine.. but that'd be overkill for this one Flow. Curious if anyone has been successful in wrapping another (non-Python) application inside a serialized Flow to deploy on Prefect Cloud.
    Scott Zelenka
    Chris White
    2 replies
    Copy to Clipboard
  • c

    Chris O'Brien

    2 years ago
    Hi Prefect Team, we have noticed that even without caching, that the results from tasks in a scheduled flow don’t fall out of scope (and therefore eligible for garbage collection) until the 3rd run. Is there a way to stop this behaviour? With large extract functions this can mean a lot of memory being held.
    c
    Chris White
    3 replies
    Copy to Clipboard
  • m

    Mohit Kumar Agarwal

    2 years ago
    Hi Prefect Team, I am new to Prefect. I am facing a very basic issue where in i am unable to start the prefect core server on my local machine. I am able to create/run basic prefect flow in a python IDE. But unable to start prefect core server and Web UI on local machine. is there any prerequisites i need to install to run prefect core server? i dont have docker installed on my machine.
    m
    1 replies
    Copy to Clipboard