Hi everyone - I'm currently evaluating using Prefect/Dask for our ETL engine, running through our various use-cases - reading through the open-source docs I'm wondering if there is a simple way to trigger execution of a task from the outside, say, from an AWS lambda function written in Javascript...any ideas?

I am running into some issues deploying Docker Containers that I am not quite sure how to debug. I am using bitbucket as my flow storage and Docker as my run environment. I try to run the flow through my Docker Agent and it successfully pulls down the docker image and then does nothing else for a long time until it seems to fail silently and a Lazarus process restarts it. Any ideas on where to start looking to figure out the issue? The logs aren't any help and I can't tell why it is failing.

hello everyone, how do i flatten/join a task of tasks? let’s say i already have tasks a,b,c defined but i would like to group them as a task, perhaps because i want a conditional flow: for one branch i want to execute a,b,c in order, otherwise i want to execute d,e,f how do i go about this? without redefining a task which does what a,b,c/d,e,f already do?

I frequently can't load cloud.prefect.io. I get blinking dots forever, Doing a hard reload doesn't help. Has anyone else run into this?

Hello everyone, I am trying to write a state handler for my flow

when it is called the new_state's 'result' attribute is an empty dictionary

Its should ideally have all the tasks and whether the failed or succeeded

Can you guys suggested why the result attribute would be an empty dictionary?

Hello, I am wondering if anyone has an example of getting a flow’s ID using the flow name and project name to execute a pre-registered flow from within a web loop? I’m not entirely sure on the best approach and how to do this: Option 1: if there is a route that I can make a GET request too, how would I find that? (been searching the repo but no luck yet) Option 2: I would write this as a graphql query. How would I know what to write? Thanks - any help is much appreciated!

Hi All My CI process is failing health checks when building my flow with

prefect build -p

with the following error. Anyone experience this issue?

Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 150, in <module>
    flows = import_flow_from_script_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 63, in import_flow_from_script_check
    flows.append(extract_flow_from_file(file_path=flow_file_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 86, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 129, in <module>
NameError: name '__file__' is not defined

Is there any tips for using local agent and local run? I find it is really error prone. e.g I have a file

foo.py

where

flowA

is encoded. I can easily run into following cases: • register with

-p ./foo.py

, it doesn’t work while

-p foo.py

works. • Under same folder

/pipelines/

there are two files

flow.py, common_task.py

, I can also run into import error, sometimes

from .common_task import xxx

works sometimes not, depending on how PYTHONPATH is set. So far my experience is to make sure local agent run on the same level where my

resiger

command is called, no leading dot in register command etc. Curious about whether ppl have better ideas on how to manage the entire process.

Hi everyone, Can someone explain to me how I integrate Sentry with Prefect? Do I need to initialize it in each flow or? Just adding it to my .py files does not work unfortunately when the flow is being registered. Sentry: https://docs.sentry.io/platforms/python/

Hi every one,  I have this error when using SqlServerExecute `ImportError: Using

prefect.tasks.sql_server

requires Prefect to be installed with the "sql_server" extra`. I am using, docker for storage using a docker file as explained here. My code looks like this

from prefect.tasks.sql_server import SqlServerExecute

STORAGE = Docker(registry_url="<http://xxx.azurecr.io|xxx.azurecr.io>",
         image_name="prefect/test",
         dockerfile="Dockerfile",
         python_dependencies=["pyodbc"])

SQL = SqlServerExecute(db_name="myDb",
            user="user",
            host="<tcp:xxx.database.windows.net>",
            port=1433,
            driver="ODBC Driver 17 for SQL Server")

QUERY = """
  CREATE TABLE [dbo].[myTable] (
    [id] VARCHAR(22) COLLATE Latin1_General_100_BIN2 NOT NULL,
    [FirstName] VARCHAR(222) NULL,
    [LastName] VARCHAR(222) NULL
  )
"""

@task
def create_table():
  logger = prefect.context.get('logger')
  <http://logger.info|logger.info>("Create table in DW for loading the results")
  SQL.run(query=QUERY, password='myPass')

with Flow("candidates-flow", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
  create_table()

Can someone help? I do not understand the error. Many thanks already 🙏

Hi all 🙂 I am encountering an issue regards the new ECS agents and I will be glad for some advice, we are executing many flows nightly. For the last half year used the Fargate agents, which worked pretty fine, we tried to move to the new agent - ECS agent but now after deploying new flow and running the batches we receive -

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.

The ECS agent logs (verbose) isn’t very helpful and I can’t debug the error as it happens when the system at its peak. And I have couple of questions - 1. Why does the flow register itself on AWS only when its running and not when it was registered? 2. Can I ask prefect to just dockerize my flow instead of register a new task definition (because nothing changed in the task definition), I saw the parameter of

idempotency_key

but I don’t understand if this is helpful in my situation 3. Should I call

register

if my docker is the only thing changed? or should I call

flow,storage.build

instead? I can provide code if needed, Thank you in advance

Hi, I'm trying to set up the ECS agent without using dynamic flow task definitions (I already have a static task definition specified in the flow's

RunConfig/ECSRun

). When I am trying to start the agent, it tries to build the default parameters for a task definition and fails (since we don't have a default VPC). Is there a way to start the ECS agent without providing a task template and to only rely on the flow configuration instead?

Hi all, hopefully a quick question. Can

PREFECT__LOGGING__EXTRA_LOGGERS

accept wildcards? So instead of something like

flow.run_config = LocalRun(
    env={
        "PREFECT__LOGGING__EXTRA_LOGGERS": json.dumps([
            'foo.bar.aaa',
            'foo.bar.bbb',
        ])
    }
)

I could use

flow.run_config = LocalRun(
    env={
        "PREFECT__LOGGING__EXTRA_LOGGERS": json.dumps([
            'foo.bar.*'
        ])
    }
)

With larger projects it's quite time consuming to add each class individually.

Hi all I am trying to deploy prefect-agent on a local kubernetes cluster

prefect.utilities.exceptions.AuthorizationError: [{'path': ['auth_info'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]

I am unsure on how to set up the yaml variables:

- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
  value: ''
- name: PREFECT__CLOUD__API
  value: <https://api.prefect.io>
- name: NAMESPACE
  value: default
- name: IMAGE_PULL_SECRETS
  value: ''
- name: PREFECT__CLOUD__AGENT__LABELS
  value: '[]'
- name: JOB_MEM_REQUEST
  value: ''
- name: JOB_MEM_LIMIT
  value: ''
- name: JOB_CPU_REQUEST
  value: ''
- name: JOB_CPU_LIMIT
  value: ''
- name: IMAGE_PULL_POLICY
  value: ''
- name: SERVICE_ACCOUNT_NAME
  value: ''
- name: PREFECT__BACKEND
  value: cloud
- name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
  value: http://:8080
image: prefecthq/prefect:0.14

specifically the prefect__cloud___agent__address seem to be incorrect. Q: Is there a better documentation on how to set up prefect-agent ?

Hi 👋 I’m currently experimenting with the recent Youtube Demo of some data science prefect flows… We have an on-premise S3 server that has a custom endpoint_url. Can I use S3Result() to store my plots? How would I provide the endpoint_url and other custom settings?

I'm having some issue importing prefect.context.today. Was this changed from 14.16 to 14.17? I'm on .16, can't seem to use "import prefect.context" https://docs.prefect.io/api/latest/utilities/context.html

Hi all, I am trying to run a prefect flow on a dask executor. I am getting the following error

[2021-05-11 19:11:14+0200] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'prefect'")

it seems that the worker does not have the prefect installed. Dask is running in a kubernetes cluster so is there a smart way on how to install prefect module on all the dask workers? M

I’m trying to set up email task, and this page gives information about how to set up email, if you are using gmail. What I’m looking for is - similar level of details if I’m using Microsoft Outlook (my office email) - does anyone here work on such use case before? Any documentation regarding set up would be great help. Thank you.

Hello all 🙂 I am getting this error

TypeError: execute() takes no keyword arguments

When running sql query using SqlServerExecute. Here is my task code following the doc explained here https://docs.prefect.io/api/latest/tasks/sql_server.html#sqlserverexecute

from prefect.tasks.sql_server import SqlServerExecute

SQL = SqlServerExecute(db_name="my"Db",
            user="user",
            host="<tcp:xxxs.database.windows.net>",
            port=1433,
            driver="ODBC Driver 17 for SQL Server")

QUERY = "CREATE TABLE [dbo].[CandidateDetails] ([CandidateId] VARCHAR(22) COLLATE Latin1_General_100_BIN2 NOT NULL,[FirstName] VARCHAR(222) NULL,[LastName] VARCHAR(222) NULL)"

@task
def create_table():
  logger = prefect.context.get('logger')
  <http://logger.info|logger.info>("Create table in DW for loading the results")
  SQL.run(query=QUERY, password='myPass')

Can you help me on this? many thanks already 🙂

Hello all, does an easy way in the graphql api to cancel all submitted, but not currently running flows exist? If so, where in the docs can I read more about this?

Hi Team, I'm working to define a custom job spec to be referenced via a KubernetesRun config in my flow, but running into a few issues when the custom job spec gets kicked off via a Kubernetes Agent. 1. The job is not being tracked via Prefect Cloud. It gets scheduled and I can see the job execute to completion in Kubernetes but Prefect Cloud never reports beyond the scheduled status. 2. The Flow itself never gets executed and the only logs are the CLI printout of the main

prefect --help

results.

Hello! I was wondering if it is possible to map parameters in dictionary to their inputs of the task. Here’s the example:

from prefect import Task, Flow, task

@task()
def task_success():
    print("Success!")
    return {'p': 1, 'k': 2, 'j': 3}


@task()
def task_to_fail(p, k):
    print(f"p: {p}; k: {k}")
    raise NotImplementedError()


with Flow("catch-all-failed-flow", state_handlers=[STATE_HANDLER_CATCH_ALL_FAILED]) as flow:
    ts = task_success()
    tf = task_to_fail(ts) # how can I map p and k params to the task_to_fail task instead of passing the whole dict in p but nothing as k

flow.run()

I know I can do ts[‘p’] and ts[‘k’], but I’m looking for something like **ts Thank you!

Hi all. I was wondering, is this a correct way to set task defaults for an entire flow?

from prefect.configuration import config

config.tasks.defaults.max_retries = 2
config.tasks.defaults.retry_delay = datetime.timedelta(seconds=10)

[~] -> docker run -it prefecthq/<server:core-0.14.18> pip show prefect
Name: prefect
Version: 0.14.16+95.g377f2fe7a

holy crap! what is the purpose of the version number in the tag if not the prefect version?

hmm in https://github.com/PrefectHQ/prefect/blob/0.14.18/requirements.txt the dep

click >= 7.0.0

will install

click v8.0.0

. This is not compatible with

click v7.1.2

which is part of the prefect-server docker image (with tag

core-0.14.18

)

Hi all, hopefully an easy one.. I need to pass the flow run ID of any flow execution in to my tasks. This should be possible through the ‘Context’ object, but when I try and register my flow to Prefect Cloud, I get the following error:

AttributeError: ‘Context’ object has no attribute ‘flow_run_id’

Any ideas?