https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Asif Imran

    02/11/2021, 5:27 PM
    Does
    <http://cloud.prefect.io|cloud.prefect.io>
    offer two-factor authentication?
    m
    3 replies · 2 participants
  • l

    Luis Gallegos

    02/11/2021, 5:34 PM
    Hi all, i want to know if this is possible. Run via dask executor different projects each one with respective virtual environment, in order to encapsulate its dependencies version. Thanks for ur help. A little representation
    j
    9 replies · 2 participants
  • m

    Milly gupta

    02/11/2021, 10:27 PM
    Hi, does anyone know if there are any free read only users available on the Prefect Cloud Team Plan?
    j
    1 reply · 2 participants
  • m

    Marwan Sarieddine

    02/11/2021, 11:58 PM
    Hi folks, I am trying to construct a graphql query to aggregate the total number of successful task runs that we had in the past month but I am noticing that it is not so trivial to form the query so I was wondering if someone else might have already done so and is willing to share the query they used. I imagine given the intended update in prefect’s pricing that others will also want to run similar queries (in case these analytics aren’t shown somewhere on the prefect cloud UI) to track their usage.
    c
    7 replies · 2 participants
  • s

    Severin Ryberg [sevberg]

    02/12/2021, 1:24 AM
    Any idea how to prevent prefect from building up large amounts of working memory? This is seen on a flow with 7 Tasks (6 of which are mapped over a list of ~40,000 elements), and running on a kubernetes agent
    a
    b
    +2
    13 replies · 5 participants
  • l

    Laura Vaida

    02/12/2021, 8:48 AM
    hi folks, getting this non-speaking error when i want to run a flow:
    INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
    does somebody know, what might be a reason for that?
    a
    a
    +1
    28 replies · 4 participants
  • n

    Nikola Milushev

    02/12/2021, 5:28 PM
    Hi guys, I have a scheduled prefect flow (runs once a day) that I register once. The flow has several Parameters, two of them being some dates which I would like to have default values with respect to the flow runs, not the flow registration. Now, the default values are set correctly upon flow registration, but on the next day, the same default dates appear, even though I expect them to be one day later. I've set the dates like this:
    schedule = Schedule(clocks=[IntervalClock(timedelta(days=1))])
    with Flow(name=name, schedule=schedule) as flow:
        start_date = Parameter(name="start_date",
                               required=False,
                               default=(datetime.now() - timedelta(days=2)).strftime("%Y-%m-%d"))
        end_date = Parameter(name="end_date",
                             required=False,
                             default=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d"))
    👍 1
    j
    7 replies · 2 participants
  • r

    Robin

    02/12/2021, 7:19 PM
    Hey community, we are getting the weird and low informative error message
    {'type': ['Missing data for required field.']}
    as single log directly in the beginning of a flow that directly fails. What could be possible reasons and did you encounter the same? 😮 PS: the flow runs locally but throws this error on Prefect Cloud 🤔
    ✅ 1
    m
    c
    5 replies · 3 participants
  • d

    Danny Vilela

    02/12/2021, 7:20 PM
    Hi Prefect! Quick question: PyCharm keeps telling me that the documentation-approved way of using the
    @task
    decorator is lacking a positional argument. For example:
    @task
    def add_one(x: int) -> int:
        return x + 1
    
    
    @task()
    def add_two(x: int) -> int:
        return x + 2
    The
    @task
    decorator over
    add_one
    is how I’ve seen it in the docs, but it seems like the decorator for
    add_two
    is how PyCharm likes it. Am I missing something?
    👍 1
    j
    6 replies · 2 participants
  • a

    Alex Welch

    02/12/2021, 7:45 PM
    Hello! I am running the hello-world flow and trying to get it running in the cloud. However when I do I get the following error
    {'type': ['Unsupported value: UniversalRun']}.
    It works locally. Here is my code:
    import prefect
    from prefect import task, Flow
    
    @task
    def say_hello():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Hello, Cloud!")
    
    with Flow("hello-flow") as flow:
        say_hello()
    
    flow.register(project_name="tutorial")
    m
    j
    62 replies · 3 participants
  • f

    Felipe Montefuscolo

    02/12/2021, 8:52 PM
    Hi guys, where is the best place to make license questions?
    n
    m
    +1
    16 replies · 4 participants
  • m

    Matthew Blau

    02/12/2021, 9:09 PM
    Hey all, I am running into a weird issue. We are trying to use Prefect to build a docker container that contains a module that depends on Pandas. We have been successfully able to build it and log "hello" to the UI. However, when we begin to use the module, we are unable to rebuild the docker container because it is looking for Pandas. The error it gives is: ModuleNotFoundError: No module named 'pandas' how can we resolve this?
    m
    13 replies · 2 participants
  • i

    itay livni

    02/14/2021, 11:22 PM
    Hi - I am trying to get some direction for the following error when a
    StartFlowRun
    is run from within another flow. Unexpected error: ClientError('400 Client Error: Bad Request for url: https://api.prefect.io/graphql\nThis is likely caused by a poorly formatted GraphQL query or mutation but the response could not be parsed for more details') Traceback (most recent call last): File "/home/ilivni/miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 361, in _send_request response.raise_for_status() File "/home/ilivni/miniconda3/envs/py38lmap/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.prefect.io/graphql The above exception was the direct cause of the following exception:
    Traceback (most recent call last):
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/utilities/executors.py", line 299, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 454, in method
        return run_method(self, *args, **kwargs)
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 177, in run
        flow_run_id = client.create_flow_run(
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 1094, in create_flow_run
        res = self.graphql(create_mutation, variables=dict(input=inputs))
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 298, in graphql
        result = <http://self.post|self.post>(
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 213, in post
        response = self._request(
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 445, in _request
        response = self._send_request(
      File "miniconda3/envs/py38lmap/lib/python3.8/site-packages/prefect/client/client.py", line 374, in _send_request
        raise ClientError(f"{exc}\n{graphql_msg}") from exc
    prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>
    This is likely caused by a poorly formatted GraphQL query or mutation but the response could not be parsed for more details
    Thanks in advance
    c
    2 replies · 2 participants
  • r

    Rob Fowler

    02/15/2021, 4:58 AM
    If I have a task (for example winrm) that uploads and executes some scripts, what is the best way with prefect server to get them into the flow. In prefect core, I load the scripts from the local filesystem into a dictionary and pass this into the flow as a parameter when doing flow.run() For prefect server I'd like to use a cron job so I don't have a local script and the scripts must already exist. Store them in bitbucket? S3?
  • s

    sark

    02/15/2021, 8:59 AM
    hi guys, how do i define a user friendly logging task? without making the user define a separate task just to concatenate strings?
    @prefect.task
    def log(item):
    	<http://prefect.context.logger.info|prefect.context.logger.info>(item)
    this works when you pass in the string output of an upstream task as the entirety of the input to the logging task
    log(output)
    but what if i want to do something like
    log("the output of the previous task was: " + output)
    here we have an issue because
    +
    is not a task and so we will actually be logging something like
    the output of the previous task was: Task<blah blah>
    k
    1 reply · 2 participants
  • r

    Ruben

    02/15/2021, 10:32 AM
    Hi, I want to limit my agent to running only a single flow / task. I thought I read about this functionality somewhere, but am unable to find it. Do you know where this is explained?
    a
    j
    6 replies · 3 participants
  • v

    Vitaly Shulgin

    02/15/2021, 12:17 PM
    Hi folks, we're using pyvirtualdisplay for one of the flow, running container in k8s, and got intermittent failure
    Failed to load and execute Flow's environment: StorageError("An error occurred while unpickling the flow:\n DisplayNameError('0.0')")
    2 replies · 1 participant
  • m

    Matthew Blau

    02/15/2021, 3:39 PM
    Hello all, I am looking through the documentation and I am struggling to find information on passing volume information to Docker Storage. My flow outputs some files to disk and I would like to mount a volume so that we can see the files that were created on the filesystem, outside of the docker container. How would I accomplish this?
    s
    k
    +1
    6 replies · 4 participants
  • d

    David Clements

    02/16/2021, 12:20 AM
    Hey There -- One of the reasons we are considering moving to prefect is to get better testing coverage. I am looking for examples how how to test Flows including any approach to mocking out Tasks. Anyone have any pointers?
    c
    2 replies · 2 participants
  • m

    Matthew Alhonte

    02/16/2021, 12:31 AM
    Hrm, can you turn a Prefect Flow into a Snowflake External Function? https://docs.snowflake.com/en/sql-reference/external-functions-introduction.html
    d
    c
    +1
    8 replies · 4 participants
  • j

    Jacob Blanco

    02/16/2021, 6:03 AM
    Hey folks, super excited for the new flow readme feature. Is there documentation anywhere for how to add the readme in code?
    👍 1
    j
    c
    +1
    6 replies · 4 participants
  • r

    rafaqat ali

    02/16/2021, 7:43 AM
    I want to deploy prefect UI and Backend on azure app service using docker and docker compose. is it possible? If yes, can someone guide me how I'll be able to deploy ? I'm totally new on prefect. Thanks,
    j
    5 replies · 2 participants
  • j

    Jonas Sølvsteen

    02/16/2021, 9:13 AM
    Dear Prefect people, can you please help me find out how to make Prefect (Cloud?) work for my use case? 🧑‍💻 We want to process thousands of satellite images 🛰️🌍 on a Kubernetes cluster, scaling to ~100 TB of RAM and thousands of jobs running in parallel. We already successfully did this with our own little event-based workflow manager (REST APIs, Postgres, Docker, Python K8s API), but it is lacking a lot of the bells and whistles that Prefect has, choking a bit on all the messages, and we would rather like to join the awesome community you have here! ❤️ Using your nice docs and the open source code, we already ran a single processing job on our K8s cluster via a Prefect flow 👍. Now we want to scale this up. Intuitively, I would have one flow process a single image (using 3-8 K8s tasks, ~10 min per task). But we want to scale to 1000s of images getting processed in parallel. Seeing that Prefect Cloud Teams plan only allows for 2️⃣(!) concurrent flows, I get in doubt whether Prefect is built for this number of concurrent flows and all the logs and messages generated from that. ❓ Can you please let me know whether Prefect is meant to scale to 10ths of 1000s of tasks running in parallel? And if so, whether I can run 1000 flows concurrently or should rather have one flow branch out into 1000 tasks with each their subtasks (if that is possible)? Please note that this is burst compute once a month or so, with lower (10s of images) daily loads. I am happy to work with Prefect Core, but Cloud might also be very nice if we can afford it.
    a
    j
    +1
    5 replies · 4 participants
  • k

    kromash

    02/16/2021, 9:49 AM
    Hi, is it possible to run LocalDaskExecutor or local DaskExecutor on Kubernetes agent? I tried it but it seems it doesn't use local Dask cluster at all.
    j
    1 reply · 2 participants
  • p

    Prashant Kumar

    02/16/2021, 1:07 PM
    Hi, Is it possible to restart a failed flow from a specific task ? Considering I've handled the failed state with a state handler for that task.
    j
    m
    5 replies · 3 participants
  • a

    Alex Welch

    02/16/2021, 2:17 PM
    Hi all, I am trying to run a Prefect agent on ECS and have been following this tutorial. I used the below configurations (along with exporting the runner token, aws creds, etc to environment variables). However I am continuously receiving a
    ValueError: Failed to infer default networkConfiguration, please explicitly configure using --run-task-kwargs
    error. I have tried all teh different combinations I can think of and been through the docs (both prefect and ECS) but cant find a solution that works. Has anyone else been able to solve this?
    export networkConfiguration="{'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['<EC2 instance subnet>'], 'securityGroups': ['<EC2 instance security group>']}}"
    prefect agent ecs start --token $RUNNER_TOKEN_FARGATE_AGENT \
     --task-role-arn=arn:aws:iam::<AWS ACCOUNT ID>:role/ECSTaskS3Role \
     --log-level INFO --label prod --label s3-flow-storage \
     --name prefect-prod
    a
    j
    +2
    51 replies · 5 participants
  • v

    Vincent

    02/16/2021, 2:36 PM
    Hi. I was wondering if anyone has noticed that some task-runs get resubmitted when running with prefect-cloud/dask. For example, in the screenshot below. I have a task that completed within 51 minutes. However, the reported completion time shows 9 hours. It seems that sometime during the flow, the task run was resubmitted and hit the cached state, which updated the logs. Does anyone have any ideas why the task resubmitted?
    j
    m
    6 replies · 3 participants
  • s

    Samuel Hinton

    02/16/2021, 3:36 PM
    Hi all! I was wondering how we can change the UI port from 8080 to something else. I have changed the
    PREFECT__SERVER__UI__PORT
    and
    PREFECT__SERVER__UI__HOST_PORT
    to try and follow the config.toml, but the UI service is still launching and saying “👾👾👾 UI running at localhost:8080 👾👾👾” and I dont know what else to change in the UI service 😞 As a follow up, what is the difference between the UI port and the UI host port? EDIT: I see echo
    UI running at localhost:8080
    is hardcoded in ./start_server.sh in the UI service, is this meant to be the case?
    m
    4 replies · 2 participants
  • d

    David Clements

    02/16/2021, 5:00 PM
    Does anyone have a recent example of getting DaskExecutor running on my local environment? It seems that most of the conversations in slack are all about using Environments instead of RunConfig.
    j
    v
    4 replies · 3 participants
  • j

    John Grubb

    02/16/2021, 5:33 PM
    Hi there, I was wondering if it's possible somehow to
    map
    on
    case
    ?
    with case.map(should_do_the_thing, True):
    does not agree...
    j
    4 replies · 2 participants
Powered by Linen
Title
j

John Grubb

02/16/2021, 5:33 PM
Hi there, I was wondering if it's possible somehow to
map
on
case
?
with case.map(should_do_the_thing, True):
does not agree...
j

Jim Crist-Harif

02/16/2021, 5:37 PM
You might find
apply_map
useful: https://docs.prefect.io/core/concepts/mapping.html#complex-mapped-pipelines
j

John Grubb

02/16/2021, 5:39 PM
Dangit, I promise I was almost there "for example, adding conditional tasks using `prefect.case`" Thanks
j

Jim Crist-Harif

02/16/2021, 5:41 PM
No worries! If there's a way we could have made this doc easier to find, please let us know. Glad to help.
j

John Grubb

02/16/2021, 5:47 PM
No, it's right there - "complex mapped pipelines" - I just always start reading at the top. Already done by the way, works perfectly..
View count: 1