• p

    Pierre CORBEL

    2 years ago
    Hello, Just to let you know that since
    prefect 0.11.0
    , you can't use the enviuronment variable
    PREFECT__CLOUD__AGENT__LABELS=["hello"]
    anymore but you have to wrap the values in single quote like
    PREFECT__CLOUD__AGENT__LABELS='["hello"]'
    I know it is documented with the right format in the doc but it was working with a "bad" format before and so it can broke your flow after upgrading to v0.11.0 👍
  • Will Milner

    Will Milner

    2 years ago
    I'm getting a weird error from the apollo service whenever I try and run a flow. The error I see in the logs is this
    apollo_1     | 2020-05-15T16:42:41.530Z {"message":"Cannot query field \"setFlowRunStates\" on type \"Mutation\". Did you mean \"set_flow_run_states\" or \"set_task_run_states\"?","locations":[{"line":2,"column":5}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
    
    apollo_1     | 2020-05-15T16:42:41.547Z {"message":"Unknown type \"writeRunLogsInput\". Did you mean \"write_run_logs_input\", \"write_run_log_input\", \"create_flow_input\", or \"archive_flow_input\"?","locations":[{"line":1,"column":18}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
    Not sure how I go about debugging this, any tips?
    Will Milner
    Tyler Wanner
    +1
    13 replies
    Copy to Clipboard
  • Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Hi all! @nicholas and I are queued up for a livestream today on how to contribute to Prefect Core’s open source UI in 15 minutes! Feel free to drop by, or go to this link after for the recording if you can’t make it!

    https://www.youtube.com/watch?v=YHqfJwFvTFY

    Laura Lorenz (she/her)
    d
    +1
    9 replies
    Copy to Clipboard
  • Brad

    Brad

    2 years ago
    Hi team, I’m just playing around with the new result class; it looks really nice so far! Would it be possible to pass the inputs to the read/write functions also? I’d like to parameterise my filenames/targets per input. And another question, according to the docs these results are persisted to the prefect database, should I expect to see this table in the GraphQL api ?
    Brad
    Chris White
    15 replies
    Copy to Clipboard
  • Pedro Machado

    Pedro Machado

    2 years ago
    Hi everyone. What is the recommended pattern to implement a groups of tasks that must be retried together"? Let's say I have a task that checks the status of a cluster (for example EMR) and starts it if the cluster is not running. There are some downstream tasks that run on this cluster. Is there a pattern that would ensure that before a given downstream task runs (including the case when the task needs to be retried), it will first run the task that checks/starts the cluster?
    Pedro Machado
    Chris White
    9 replies
    Copy to Clipboard
  • n

    Nicolas Michel

    2 years ago
    With prefect core installed on premise, do you also have an endpoint available for making graphql requests?
    n
    Chris White
    2 replies
    Copy to Clipboard
  • m

    Matthias Busch

    2 years ago
    Hi guys. I've just came over from @Laura Lorenz (she/her) PyCon Sprint-Stream. Looking forward to learn both how to use and how to contribute to prefect ! 😃
    m
    1 replies
    Copy to Clipboard
  • n

    Nate Atkins

    2 years ago
    I'd like to be able to pass a trigger to a task when I add the task to the flow. I have a trigger I'm working on that takes additional parameters based on its associated task. I can replace the default trigger directly, but this adds a bit of clutter when constructing the flow. Existing:
    with Flow("Existing") as flow:
        file_b = file_builder("b.data")
        file_b.trigger = upstream_dependency(src_fpaths=["a.data"])
    Proposed:
    with Flow("Proposed") as flow:
        file_b = file_builder("b.data", trigger=upstream_dependency(src_fpaths="a.data"))
    Example Trigger:
    @curry
    def upstream_dependency(
        upstream_states: Callable[[Dict["core.Edge", "state.State"]], bool],
        source_fpaths: List[Path] = None,
    ) -> bool:
        if not all(s.is_successful() for s in _get_all_states_as_set(upstream_states)):
            raise signals.TRIGGERFAIL(
                'Trigger was "all_successful" but some of the upstream tasks failed.'
            )
        # Everything skipped.   Check to see if any file dependencies require task to run.
        run_required = file_dependency_skipper(source_fpaths)
        if run_required:
            return True
        else:
            raise signals.SKIP("All dependencies are up to date.")
    I hacked this together with the following change. In
    prefect.core.task
    def bind(
        self,
        *args: Any,
        mapped: bool = False,
        upstream_tasks: Iterable[Any] = None,
        trigger: Callable[[Dict["core.Edge", "state.State"]], bool] = None,
        flow: "Flow" = None,
        **kwargs: Any
    ) -> "Task":
    self.trigger = trigger
    If I build the task by inheritance instead of the @task decorator, I can pass the trigger. I'm not sure if we want to dump all the parameters into the parameters for constructing a task as the list is pretty short now.
    upstream_tasks
    being the most similar one that is already in the signature of bind.
    n
    Jeremiah
    7 replies
    Copy to Clipboard
  • i

    itay livni

    2 years ago
    Hi - I am working with
    S3Result
    and receiving a
    botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist
    Which upon further research - it can be anything including a permission error. (I tried different buckets with settings) The credentials are stored as AWS_CREDENTIALS in prefect cloud. With the config.toml set to use cloud secrets
    [cloud]
    use_local_secrets = false
    Switching back to
    result_handler
    argument with
    S3Result
    subclass did work, . And combining
    result handler
    with
    target
    does not. Is there something different in the way that credentials are handled between
    result
    and
    result_handler
    ? The new prefect is really nice 🙂
    i
    Chris White
    +1
    17 replies
    Copy to Clipboard
  • Brad

    Brad

    2 years ago
    Hey guys - possible bug with
    log_stdout
    , it looks like maybe some references were changed from
    result
    to
    value
    here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/task_runner.py#L938
    Brad
    Chris White
    9 replies
    Copy to Clipboard