I have 4 tasks: t1, t2, t3, t4 How to start t3 and t4 after t1 and t2 successfully finished?

Hey 🙂 Prefect so far looks to be quite interesting and stream lined! I'm just wondering if there is any good way to set up an CI/CD process? Let's say I have a repo, where I store my ETLs. How to I setup a way that prefect loads changes from the master branch and registers the changes? Is this even a problem prefect wants to solve?

why flows doesn’t work in parallel? I have one local agent. anyway, why? cloud

Hello, how should I pass defined parameter (prefect input) to a MySQLExecute query param? query expects string 😕

Hi! I work at a large corp and will not be able to use Prefect Cloud. Is there a good tutorial for installing prefect core server on a host? Preferably will run on a long running kubernetes container, but EC2 also OK.

Hello everybody. I’m running a local agent inside a docker container and triggering a flow from Prefect Cloud and getting this stacktrace:

[2021-01-20 18:57:29,015] INFO - agent | Deploying flow run 0ed795b0-8eb3-4086-bf40-c19a54792a7b
'NoneType' object has no attribute 'setup'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/prefect/__main__.py", line 4, in <module>
    cli()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 90, in flow_run
    raise exc
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 75, in flow_run
    environment.setup(flow)
AttributeError: 'NoneType' object has no attribute 'setup'
[2021-01-20 18:58:11,469] INFO - agent | Process PID 188 returned non-zero exit code

However, I can run it locally using flow.run() without issue. I can see in the code that environment is not set, but I don’t understand what I’m doing wrong… Thanks in advance!

Hi everyone! I have the below Prefect flow with ECSRun run configration. On registering the flow, I receive the below error. How can I override the task def parameters? Error

Parameter validation failed: Invalid type for parameter containerDefinitions[0].logConfiguration.options.awslogs-create-group, value: True, type: <class 'bool'>, valid types: <class 'str'>

Hi, I am trying to access the registered flow from aws ec2 public dns...something like this http://ec2-xxxxxxxxxxxx.com:8080/default/flow/4dfbfb93-e00a-4ad4-a34d-fdd813f9a123, but the url reverts back to http://ec2-xxxxxxxxxxxx.com:8080 and the dashboard is not showing the flows, but the flow is running and I can see the output being stored in my database...Am I missing something in the setup when the host is not the localhost for the prefect server?

Hey! Is there a way to create the default tenant at deployment time using helm chart, so we would avoid typing "prefect create tenant"?

[Resolved] How do I remove default labels from an agent? There are several labels on the agent that I'm running, and I'm not sure how they got set. There are no labels set in ~/.prefect/config.toml for the user running the agent. I also don't see any environment variables set in the script that starts the agent. Where else would I look?

I am new to Prefect and working my way through the docs. I noticed something weird in the

flow.storage

assignment in the file based storage discussion:

flow.storage = GitHub(
    repo="org/repo",                 # name of repo
    path="flows/my_flow.py",        # location of flow

Hi wonder if anyone can help me - I have got the server running on kubernetes using the helm chart in the server repo. I am trying to add a new tenant using the command

prefect server create-tenant ...

it errors every time because its trying to connect to

localhost:4200

rather than

myserver_ip:4200

- how do I configure it to use the correct hostname?

Can someone help me here....I have 1 agent locally running and upto 3 flows are running fine...when I create and schedule the 4th flow...the agent is stopping.....any limitations to number of flow or am I missing something????

Can anyone point me to some detailed notes on setting up prefect server and agent on k8s? I have managed to deploy the UI and the agent all pods running green - however the agent does not know where it needs to query the webserver - how do you set the location of the webserver in the agent?

Hey, I’m trying to start a docker agent on CentOS7 (via prefect agent docker start), and get this error: “urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=‘host.docker.internal’, port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f2881d77350>: Failed to establish a new connection: [Errno 113] No route to host’))” I thought maybe it was this bug (https://github.com/PrefectHQ/server/issues/25) but I still can’t figure out how to work around it. Any suggestions on how to move forward? I’m stuck

Hi - are there any updates on a multi-node server deployment?

I can't seem to find docs on access to the graphql interactive API outside of the UI? Is the idea to use the lib for interactions?

Hi, Is it possible to upload a file to a Prefect Flow, say while initiating a Flow run, either as a parameter or context object, which can then be further read in a task?

Hi I am still tying to get the server working robustly on k8s using the helm chart in the repo - my current problem if anyone could help? I don't want the UI to have to talk to the apollo server via the open internet on the public ip so I have tried to set the following overrides;

agent:

  # enabled determines if the Prefect Kubernetes agent is deployed
  enabled: true
  prefectLabels: ['prefect-agent']

ui:
  apolloApiUrl: <http://prefect-apollo.prefect:4200/graphql>

The UI just gives me the following error;

Oops! It looks like something went wrong when trying to connect; make sure Prefect Server is running at the URL above and try again.

However if I start a shell on the UI pod and run curl against that URL it connects as expected to the api;

# curl <http://prefect-apollo.prefect:4200/graphql>
GET query missing.#

The reason I am even trying this is because if I use the public IP of the apollo server I always end up getting very slow UI and occasionally complete disconnects from the graphql server.

Can anyone point me to the deafult job spec for kubernetes without dask the docs point here -> https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/dask/job.yaml

hello all! i'm experiencing a strange error when trying to set a schedule for a flow: i'm setting up my workflow as follows:

def main():
    # Schedule for every day @ 4AM ET / 9AM UTC
    today = datetime.now(timezone.utc)
    start = datetime(
        int(today.year),
        int(today.month),
        int(today.day+1),
        hour=int(START_TIME.hour),
        minute=int(START_TIME.minute),
        tzinfo=timezone.utc
    )
    schedule = IntervalClock(
        start_date=start,
        interval=timedelta(days=1),
    )

    with Flow(schedule=schedule):
.........

when i try to register this flow, i get the following:

<snip>
TypeError: not all arguments converted during string formatting

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "register_flows.py", line 7, in <module>
    dev_flow.register()
  File "/usr/local/bin/ddiqa-aws/ddiqa_aws/flows/dev_flow.py", line 234, in register
    main()
  File "/usr/local/bin/ddiqa-aws/ddiqa_aws/flows/dev_flow.py", line 227, in main
    flow.register(
  File "/home/nlambeth/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1668, in register
    registered_flow = client.register(
  File "/home/nlambeth/.local/lib/python3.8/site-packages/prefect/client/client.py", line 802, in register
    raise ValueError(
ValueError: Flow could not be deserialized successfully. Error was: TypeError('not all arguments converted during string formatting')

this flow registers and runs 100% ok if i don't pass

schedule

in to Flow(). i can't seem to find any relevant bug reports out there.

I am having a lot of trouble with liveness and readiness probes of the graphql and hasura deployments. This is the current state of my pods;

prefect-agent-55775d5684-qml7k     1/1     Running   0          5h21m
prefect-apollo-b5d9c4cd8-82txc     1/1     Running   1          28h
prefect-graphql-5f77cd4674-wmfbj   0/1     Running   21         22h
prefect-hasura-bf4dd5d95-z2j8x     0/1     Running   24         28h
prefect-job-e63a7de9-nwrxg         1/1     Running   0          20m
prefect-postgresql-0               1/1     Running   0          28h
prefect-towel-5dc84cb477-zx2rg     1/1     Running   0          28h
prefect-ui-8bd5ff9f8-2scsz         1/1     Running   0          22h

They keep on restarting and going into

CrashLoopBackOff

with errors like;

Warning  Unhealthy  9m39s (x45 over 22h)   kubelet  Readiness probe failed: Get <http://10.40.14.208:4201/health>: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  9m37s (x41 over 22h)   kubelet  Liveness probe failed: Get <http://10.40.14.208:4201/health>: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  5m18s (x35 over 109m)  kubelet  Liveness probe failed: Get <http://10.40.14.208:4201/health>: dial tcp 10.40.14.208:4201: connect: connection refused

Has anybody had similar problems and did they manage to solve them?

Something about prefect server start?

ERROR:
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

Exception caught; killing services (press ctrl-C to force)
ERROR:
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

Traceback (most recent call last):
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/prefect/cli/server.py", line 346, in start
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/prefect/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/prefect/lib/python3.8/site-packages/prefect/cli/server.py", line 384, in start
    subprocess.check_output(
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.

Seems like docker-compose.yml is missing

Hello everyone, we have been testing Prefect since December, and so far we really like it and are excited to test it on live projects. I am by now quite familiar with the docs and with orchestration, and I would like to ask for some help on one thing which is not entirely clear yet to me. I want to give an example: Let’s say we have a flow that does SQL stuff on Postgres, for example

1) create table a

,

2) create table b

and

3) join data from tables a and b

. We want to use the dask executor, because there are numerous more tasks. What is the recommended way to have

task 1

and

task 2

run simultaneously, but have

task 3

wait for tasks 1 and 2? The dask executor would create a DAG where it would run tasks 1, 2 and 3 together, but that will fail, since task 3 is depending on tasks 1 and 2. The Python code however, has no dependency. I have few ideas, and I am excited to see your opinion on that question. I hope that I could express myself clearly. Best regards from Berlin, David.

Hi all! I'm fairly new to devops and data engineering, quite new to prefect, and brand new to graphql. I'm trying run prefect locally and submit a secret via the graphql ui. I can

prefect server start --use-volume

then navigate to

<http://0.0.0.0:4200/>

to access the graphql gui, and then input

mutation {
  set_secret(input: { name: "app_pw", value: "values_of_password" }) {
    success
  }
}

per this documentation https://docs.prefect.io/orchestration/concepts/secrets.html#graphql and I get back an error

{
  "error": {
    "errors": [
      {
        "message": "Cannot query field \"set_secret\" on type \"Mutation\".",
        "locations": [
          {
            "line": 2,
            "column": 3
          }
        ],
        "extensions": {
          "code": "GRAPHQL_VALIDATION_FAILED"
        }
      }
    ]
  }
}

I get a similar error trying

query {
  secret(order_by: { name: asc }) {
    name
  }
}

Is there an example of how to create a markdown artifact?

Hi there, is there a way how to add Secrets to already started prefect server (prefect core)? or to check somehow that the secrets are present there?

Another thing is that I am getting a strange error for this simple Flow:

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'C'")

Can anybody tell me some hint where to look for a cause? googling did not help... 😞 More details in thread.