• Andy Dyer

    Andy Dyer

    1 year ago
    hey all. I am trying to register a flow but its having difficulty pickling a dependency of
    paramiko
    which I am using for sftp. See below stack trace. it makes sense and other people are having difficulties pickling the same object in different applications. Is the best bet to use ShellTasks to sftp?
    <https://github.com/paramiko/paramiko/issues/753>
    <https://stackoverflow.com/questions/39321606/cant-pickle-an-rsa-key-to-send-over-a-socket>
    stack trace : https://pastebin.com/8hyAdpx1
    Andy Dyer
    Jim Crist-Harif
    19 replies
    Copy to Clipboard
  • i

    itay livni

    1 year ago
    Hi - I am running into an error where a merge statement gets triggered in a case branch unnecessarily causing the flow to fail. Is this expected behavior? https://github.com/PrefectHQ/prefect/issues/3399
    i
    Chris White
    +1
    3 replies
    Copy to Clipboard
  • m

    Matt

    1 year ago
    Hi! I'm new to Prefect, working on my first flow, and struggling a bit with config and context in Prefect Core. I'm calling this from the command line and have the parameters going in fine and am grabbing secrets from env settings. I also understand that Prefect returns these as Tasks and not strings. Are both of these (Parameters and EnvVarSecrets) only available to the Flow? I don't seem to be able to access them from the Tasks themselves and it doesn't seem like they get automatically added to prefect.context or prefect.config. Is the only way to do this to pass them to the Task (socrata.download_dataset in this case) as parameters? I've tried adding the Secrets to both context and config programmatically but they don't seem to appear when I do this although the docs suggest this should work. My first attempt at a flow had parameters getting passed every which way and I'm trying to clean this up. Would love some guidance from the community. Thanks!
    m
    nicholas
    +1
    10 replies
    Copy to Clipboard
  • JC Garcia

    JC Garcia

    1 year ago
    Hey guys, got a noob question here. When running:
    prefect create project "Test Project X"
    I assume that under the hood prefect is hitting localhost:4200, is there any way to change that url?
    JC Garcia
    Chris White
    2 replies
    Copy to Clipboard
  • j

    John Song

    1 year ago
    A dumb question, how can I redirect or config prefect logging to a local file? I read to doc but didn't find any example to do it
    j
    Chris White
    7 replies
    Copy to Clipboard
  • ms16

    ms16

    1 year ago
    Greetings ! Prefect Community Im using the Prefect for the first time and trying to figure out how to get the flow to execute on the Prefect Cloud . Here is what I have done.1. Create RUNNER Token as an Admin user 2. Build a docker container to install the prefect pip pkg and run prefect agent start at the ENTRYPOINT
    ENTRYPOINT prefect agent start --token  $PREFECT_RUNNER_TOKEN -l fargate-dev
    3. When I run the docker container locally - The Precct agent starts inside the container and can see the output
    [2020-09-30 02:53:34,447] INFO - agent | Waiting for flow runs...
    4. But the flows registered in the project never start to execute I have RTFM but still missing a link here 😕 Here is the code of my registered flow
    import prefect
    from prefect import task, Flow
    @task
    def hello_task():
        print("hello")
    flow = Flow("hello-flow", tasks=[hello_task])
    flow.register(project_name="Demo",labels = "fargate-dev")
    ms16
    Jacob Blanco
    4 replies
    Copy to Clipboard
  • ms16

    ms16

    1 year ago
    tldr; registered the agent , cant run flow
  • Prathamesh

    Prathamesh

    1 year ago
    Hi all, I am new to Prefect. I have a simple Hello World python script that I'm orchestrating via Prefect. In my local Prefect Core server it takes 1 sec to run from submission to execution. But from Prefect Cloud, it takes about 6-9 minutes to complete. Any help/pointers will be helpful, thanks in advance.
    Prathamesh
    nicholas
    9 replies
    Copy to Clipboard
  • Robin

    Robin

    1 year ago
    Hey all, is there a way to set the
    min_workers
    and
    max_workers
    as flow parameters, such that they can be changed from the cloud ui?
    flow.environment = DaskKubernetesEnvironment(
            min_workers=20, max_workers=30, labels=["k8s"]
        )
    Robin
    j
    5 replies
    Copy to Clipboard
  • Nuno Silva

    Nuno Silva

    1 year ago
    Hello. When using mapped tasks, is it possible provide a custom name to each child so that in prefect server instead of showing e.g.
    table_copy 0....table_copy n
    would show
    table_copy table_name_1....table_copy table_name_n
    ? Thank you
    Nuno Silva
    nicholas
    +2
    8 replies
    Copy to Clipboard