• ale

    ale

    1 year ago
    Hi folks, I’ reading about custom logging here: https://docs.prefect.io/core/advanced_tutorials/custom-logs.html I see that log level and formatter can be specified in the Prefect user config file. Do these logging settings apply also to Prefect server if supplied in
    config.toml
    ?
  • Darragh

    Darragh

    1 year ago
    Hey guys, I’m upgrading from 0.12.2 to 0.13.12 and having some problems. Previously I was able to set env variables to override the graphql endpoint, and a bunch of other things, but it doesn’t seem to accept the format I had anymore. Previous variable:
    PREFECT__SERVER__UI__GRAPHQL_URL
    Not working with 0.13.12 Docs seem to suggest that it’s changed
    graphql_url
    to
    apollo_url
    , so I tried to update the env var to
    PREFECT__SERVER__UI__APOLLO_URL
    or even
    PREFECT__SERVER__SERVER__UI__APOLLO_URL
    but no luck. Any suggestions? Also tried
    PREFECT__TELEMETRY__SERVER__TELEMETRY__ENABLED=false
    to disable telemetry but no luck there either
    Darragh
    nicholas
    29 replies
    Copy to Clipboard
  • m

    Marley

    1 year ago
    I’d like my Cloud Flow to run and register from a Docker setup with a Prefect auth token that is at the team level. The setup right now is that I have a user token stored in a kube secret, but ideally that token wouldn’t be tied to a particular user. I cannot run a flow from a TENANT or RUNNER token, and I’d prefer not to pass a User token into the env variables of the image. Is there a way around using User tokens?
    m
    Chris White
    2 replies
    Copy to Clipboard
  • Anish Chhaparwal

    Anish Chhaparwal

    1 year ago
    I have a python program with large number on input parameters. say around 35+. We are currently using argparser to set default values, load some of them from config files and pass them all args to functions. I'd like to convert the program into a flow. Is there a way to use the existing args in the flow. I have illustrated a sample of what id like to achieve but this fails when i try to run this from UI (refer attach image too).
    import argparse
    from prefect import Flow, Parameter, task
    import prefect
    from prefect.environments import LocalEnvironment
    @task
    def testing(args):
        
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f"agrs value for name is : {args.name}")
    <http://logger.info|logger.info>(f"agrs value for worker is : {args.workers}")
    with Flow("args_trial") as flow:
        
    opts = Parameter("opts")
        
    testing(args=(opts))
    if __name__ == "__main__":
        
    parser = argparse.ArgumentParser(description="test pywinrm")
        
    parser.add_argument("--workers", type=int, default=10)
        
    parser.add_argument("--sleep_time", type=int, default=2)
        
    parser.add_argument("--name", type=str, default="args_tester")
        
    opts = parser.parse_args()
    #    flow.run(opts=opts)
        
    flow.environment = LocalEnvironment(labels=["qure9", "pipelines"])
        
    flow.register(project_name="Test")
    I can seem to figure out how to pass the default args while registering. Also, if i have to convert each of them into prefect parameters, i'd like to pass all parameters to a function using something like def func_1(args) instead of specifying 15-20 to each function. is that possible?
    Anish Chhaparwal
    Chris White
    3 replies
    Copy to Clipboard
  • Scott Asher

    Scott Asher

    1 year ago
    Running into problem - registering flow created in non
    __main__
    (moved code blocks into thread)
    Scott Asher
    Chris White
    7 replies
    Copy to Clipboard
  • Narasimhan Ramaswamy

    Narasimhan Ramaswamy

    1 year ago
    from prefect import task, case, apply_map,Flow,unmapped
    import prefect
    from prefect.tasks.control_flow import merge
    from prefect.tasks.shell import ShellTask
    from prefect.engine.results import LocalResult
    from prefect.tasks.control_flow import case
    from prefect.environments.storage import Docker
    import docker 
    
    def test2(name):
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>("Inside Task 2")
        a = ShellTask(name=name, command="ls", return_all=True, log_stderr=True)
        return a
    
    @task(log_stdout=True)
    def inc_if_even(x):
        logger = prefect.context.get("logger")
    
        <http://logger.info|logger.info>("----Start----")
        name = r'Shell' + str(x)
        b = test2(name).run()
        <http://logger.info|logger.info>(f"{b}")
        <http://logger.info|logger.info>("----End----")
        return 1
    
    @task
    def reduce(x):
        z = sum(x)
        if z > 1:
            return True
        else:
            return False
    
    @task
    def return_num(x):
        return x + 10
    
    @task
    def logger_st(status):
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(f"Task Failed - {status}")
        return 1
    
    from prefect.engine.executors import DaskExecutor,LocalDaskExecutor
    from prefect.environments import LocalEnvironment
    
    with Flow("test-flow",environment=LocalEnvironment(executor=DaskExecutor())) as flow:
        logger = prefect.context.get("logger")
        return_nums = return_num.map([1,2,3,4,5,6,7])
        result = inc_if_even.map(return_nums)
        reduced = reduce(result)
    Hi All, just a quick question on DaskExecutor() - when running using flow.run, the mapped tasks works in parallel. but when running on cloud, it runs in sequence. My setup is agent running on Kubernetes. can you please help on what i should be doing?
    Narasimhan Ramaswamy
    Jimmy Le
    2 replies
    Copy to Clipboard
  • Scott Asher

    Scott Asher

    1 year ago
    I’m still seeing this error, even with my flow entirely created in the main script:
    File "/usr/scratch/sasher/pyenvs/prefect/lib/python3.6/site-packages/prefect/utilities/storage.py", line 85, in extract_flow_from_file
        raise ValueError("No flow found in file.")
    ValueError: No flow found in file.
    Scott Asher
    Michael Adkins
    7 replies
    Copy to Clipboard
  • Scott Asher

    Scott Asher

    1 year ago
    Second question - and I assume this is in the docs somewhere, but i just can’t find it. In terms of using the Imperative API, you guys say:
    Prefect’s imperative API allows more fine-grained control. Its main advantage over the functional API is that it allows tasks to be set as upstream or downstream dependencies without passing their results. This allows you to create a strict ordering of tasks through state dependencies without also creating data dependencies.
    And then you give an example where you force a “PlusOne” task to run after a task that simply prints something. My question is - what if you WANT a data dependency? Can you do this through the imperative API? More code blocks to follow in thread so I don’t clutter up the main thread.
    Scott Asher
    Michael Adkins
    4 replies
    Copy to Clipboard
  • Leon Hao

    Leon Hao

    1 year ago
    Hi Team, In the why not Airflow blog it says More confusingly, the
    execution_date
    is not interpreted by Airflow as the start time of the DAG, but rather the end of an interval capped by the DAG’s start time. I believe execution_date in Airflow is the start time of the schedule interval?
    Leon Hao
    1 replies
    Copy to Clipboard
  • h

    Hagai Arad

    1 year ago
    Hello 👋 I’ve been trying to find the best way to set my prefect environment totally containerised and can’t manage to do it at the moment. I’m currently trying to use “docker in docker”: I’ve created an image that includes docker, docker-compose and prefect. I create a container of this image and run the prefect commands to initialise prefect server and ui. It all seems to work fine, but when launching localhost:8080 it shows me the home page and asks me to create a tenant. When running the create-tenant command (both in python and cli) from within the docker container I get an error (attached as a reply). When installing prefect on my host machine and running the create-tenant command the ui is working, but this is the exact thing I want to avoid (to install prefect on the host machine). Would love to get any tips how to solve it OR how to design it in another containerised way. Thanks!
    h
    Kyle Moon-Wright
    3 replies
    Copy to Clipboard