• s

    Sven Teresniak

    2 years ago
    A task's retry timeout is blocking other tasks from running. Is this on purpose? While the retry-timeout is blocking other runnable tasks, their task-timeout (running time) is running. This seems a bit odd. Or did I miss something? In other words: when a task is waiting for retry, other tasks cannot use that worker (dask-worker in my case) but the waiting task's clock is running towards runtime-timeout. okay, maybe i was wrong and the waiting task is in state "pending". i maybe misinterpreted the message
    /usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py:822: UserWarning: This task is running in a daemonic subprocess; consequently Prefect can only enforce a soft timeout limit, i.e., if your Task reaches its timeout limit it will enter a TimedOut state but continue running in the background.
    😞
  • Robin

    Robin

    2 years ago
    Hey all, I am having the
    Unexpected error: NoRegionError('You must specify a region.')
    , although I added
    AWS_DEFAULT_REGION
    (along with
    AWS_CREDENTIALS
    ) to the prefect secrets. Do I miss anything?
    Robin
    nicholas
    8 replies
    Copy to Clipboard
  • a

    as

    2 years ago
    Hey, I have a question on the use of the
    target
    parameter of tasks for checkpointing. If I'm correct it will not run the task if it can find a file with the name specified with the
    target
    parameter. correct? I'm wondering what happens if an upstream task has been triggered. Does it still looks for the target file or does task run nonetheless because the input has changed? If not, is there a way to achieve this behavior? Thanks
    a
    Jenny
    +1
    3 replies
    Copy to Clipboard
  • Nuno Silva

    Nuno Silva

    2 years ago
    Hi. quick question: starting the agent in the command line works fine when specifying the server url
    prefect agent start --api http://<url>:4200
    but in code doesn't:
    from prefect.agent.local import LocalAgent
    agent = LocalAgent(agent_address="http://<url>:4200")
    agent.start()
    fails with:
    /tornado/netutil.py", line 174, in bind_sockets
        sock.bind(sockaddr)
    OSError: [Errno 99] Cannot assign requested address
    Any idea? thanks
    Nuno Silva
    nicholas
    3 replies
    Copy to Clipboard
  • i

    itay livni

    2 years ago
    Hi - Is
    apply_map
    the only way to implement a mapped case statement? Thanks
    i
    Jenny
    +1
    7 replies
    Copy to Clipboard
  • t

    Thomas Hoeck

    2 years ago
    Hi. I keeping getting flows failing with
    HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)
    . I'm running a k8 agent. What is weird is that it is that it is not happening every time but in the same frequency (pictures shows how the same flow only fails every second time.
    t
    Jenny
    +1
    15 replies
    Copy to Clipboard
  • s

    Sven Teresniak

    2 years ago
    Whats the most elegant/preferred way to synchronize flow runs? Using/setting tags like locks and polling prefect for runs w/ or w/o tags to decide/delay flow runs? I have more than one trigger for a certain flow-run and I need every run to be "atomic", that is, running several times is okay, but never in parallel.
    s
    Jenny
    7 replies
    Copy to Clipboard
  • n

    Nate Joselson

    2 years ago
    Hey All! I am trying to run a
    DbtShellTask
    with a dynamic set of
    dbt_kwargs
    that are defined at run time. The task works when I define the
    dbt_kwargs
    in the task initialization, but not when they are provided at runtime.
    from prefect import task, Flow, Parameter
    from prefect.tasks.dbt import DbtShellTask
    
    @task
    def generate_dbt_kwargs_dict(project_id):
        dbt_kwargs_dict = {
            'type': 'bigquery',
            'method': 'service-account',
            'keyfile': 'key.json',
            'project': project_id,
            'dataset': 'derived_data',
            'location': 'EU',
        }
        return dbt_kwargs_dict
    
    dbt_run_task = DbtShellTask(
            name='dbt_run_task',
            command='dbt run',
            profile_name='default',
            environment='test',
            overwrite_profiles=True,
            profiles_dir=profiles_path,
        )
    
    with Flow(name="dbt_flow") as f:
        project_id = Parameter("project_id")
        dbt_kwargs_dict = generate_dbt_kwargs_dict(project_id)
        run_task = dbt_run_task(dbt_kwargs=dbt_kwargs_dict)
    
    out = f.run(project_id='bq-project')
    This returns a prefect error:
    ERROR - prefect.TaskRunner | Unexpected error: TypeError("'NoneType' object is not a mapping",)
    I assume this means from here that the
    dbt_kwargs
    dictionary is
    None
    but I can’t see why, since I am passing the dictionary from the previous task… Does anyone have an idea about this? Thanks!
    n
    Jenny
    5 replies
    Copy to Clipboard