https://prefect.io
Join Slack
hi folks, getting this non-speaking error when i want to run a flow: `INFO - prefect.FlowRunner | F...
# ask-community
l
hi folks, getting this non-speaking error when i want to run a flow: 
INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
does somebody know, what might be a reason for that?
a
Hey @Laura Vaida seems like one of your tasks failed. I see that you’re using a FlowRunner, which I never used tbh 😅 However, you should be able to increase logging and see what’s happening
l
heey, thanks for your reply! how does this logging increase work?
a
Can you share your code, maybe it will be easier for me to understand how logging can be increased
l
Copy code
with Flow("billwerk-contracts") as flow:
    BW_Client_Id = PrefectSecret("BW_Client_id")
    BW_Username= PrefectSecret("BW_Username")
    BW_Password = PrefectSecret("BW_Password")
    BW_Client_Secret = PrefectSecret("BW_Client_Secret")
    oauth(client_id = BW_Client_Id, username = BW_Username, password = BW_Password, client_secret = BW_Client_Secret)



flow.run(executor=executor)
flow.storage= Docker(registry_url="<http://gcr.io/keller-steering-enabling/flows|gcr.io/keller-steering-enabling/flows>")
flow.run_config = KubernetesRun()
flow.register('Billwerk')
a
How are you running this flow? Using a Prefect Agent?
l
yes i want to use a kubernets agent
a
Are you triggering the flow run from within a Prefect Server?
l
do i have to create them first? i already have one in gcp
no kubernetes agent on gcp
a
Ok, but since you’re calling 
flow.register
you are register them either in Prefect Server or Prefect Cloud 🙂 Both of them expose logs of flow runs, you should be able to get them from the UI
l
prefect cloud
a
I’m not very familiar with Prefect Cloud, but when you run a flow, there should be logs somewhere. If the only log you get is the one above, then I suggest checking the pod logs on k8s
l
i guess the problem is that PrefectSecret cant read the secret from the cloud
yes, checked that and it says
a
For Prefect Cloud I guess you should configure some sort of auth token to be able to read/access secrets
l
yes i already have
a
Mmmh...I've run out of ideas 😅 Do you have the full log from the pod?
a
One possible problem is that 
use_local_secrets = true
by default: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml You have to set this to be 
false
say using a user-defined config file or an env var in order for the secret to be searched for in Prefect Cloud if it is not available locally.
l
okey, thanks do you know how to specify that?
a
You can refer to the docs on configuration: https://docs.prefect.io/core/concepts/configuration.html
It would probably suffice to create a file 
$HOME/.prefect/config.toml
with the content:
Copy code
[cloud]
use_local_secrets = false
✅ 1
l
how does config.toml differentiate from backend.toml?
z
The 
backend.toml
file is used by the Prefect library for automated config changes e.g. 
prefect backend cloud
will write to the 
backend.toml
whereas the 
config.toml
is for users to configure the library directly.
They're merged when loaded.
I'm not sure that this error is related to secrets though, can you run a flow in K8s that just executes a "Hello world" task?
l
i just ran another flow in the kubernets engine, that works fine
this works:
Copy code
@task
def inc(x):
    return x + 1


@task
def dec(x):
    return x - 1


@task
def add(x, y):
    return x + y


@task
def list_sum(arr):
    return sum(arr)


with Flow("dask-example")  as flow:
    incs = inc.map(x=range(100))
    decs = dec.map(x=range(100))
    adds = add.map(x=incs, y=decs)
    total = list_sum(adds)


flow.run(executor=executor)
flow.storage= Docker(registry_url="<http://gcr.io/keller-steering-enabling/flows|gcr.io/keller-steering-enabling/flows>")
flow.run_config = KubernetesRun()
flow.register('tutorial')
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by