Scarlett King
06/23/2021, 10:44 AMLawrence Finn
06/23/2021, 4:25 PMLawrence Finn
06/23/2021, 7:48 PMRobert Hales
06/24/2021, 10:10 AMAlderson
06/24/2021, 12:56 PMKarim Benjelloun
06/24/2021, 1:48 PMLawrence Finn
06/24/2021, 7:28 PMRunNamespacedJob
and use it as an input for another task?Hugo Kitano
06/24/2021, 11:03 PMprefect server start
and running into an error:
{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
There’s a lot more to the error I can supply. I’ve done prefect backend server
before trying to start the serverAlexander van Eck
06/25/2021, 11:16 AMjcozar
06/27/2021, 9:01 AM...
07:17:34 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
07:17:35 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
...
It seems the tasks started but finished in Queued state again… What could be the problem here? I am using task concurrency for those tasks if that helps.
Thank you!Aiden Price
06/28/2021, 8:07 AMFailed to load and execute Flow's environment
error when the flow starts though. I can see that it's missing the Azure storage connection string in the run config. In a KubernetesRun I handled this with a custom job definition but I don't know what to do with a LocalRun. I have the necessary secrets in the LocalAgent's container's environment, but they're coming up null.
Am I missing something simple here? Thanks everyone.Ruslan
06/28/2021, 2:23 PMflow.storage = GitHub(secrets=["github"],
repo="corp/etl",
path="flows/flow.py")
and error:
Repo 'corp/etl' not found. Check that it exists (and is spelled correctly), and that you have configured the proper credentials for accessing it.
Daniel Davee
06/28/2021, 6:11 PMDocker(
files={
# absolute path source -> destination in image
"/Users/me/code/my_mods/": "/my_mods",
},
env_vars={
# append modules directory to PYTHONPATH
"PYTHONPATH": "$PYTHONPATH:modules/"
},
)
Casey Green
06/28/2021, 7:34 PMKen Farr
06/28/2021, 10:17 PM...
cft_container = CreateContainer(image_name="twodub:v0.4")
cft_start = StartContainer()
cft_logs = GetContainerLogs()
cft_status_code = WaitOnContainer()
@task
def set_cft_container_command(cft_name: str, account_id: str):
logger = prefect.context.get("logger")
ret = f"python -m twodub cft install --cft-name {cft_name} --account-id {account_id}"
new_container.command = ret # <--- Referencing new_container before declared
<http://logger.info|logger.info>(f"command is: '{ret}'")
return ret
with Flow("install-cft-flow") as flow:
cft_name = Parameter("CFT Name")
account_id = Parameter("Account ID")
scc = set_cft_container_command(cft_name, account_id)
# I'm creating a new version of this Task here with bind
# I suspect there is a cleaner way than this, but it wasn't
# apparent to me in the documentation
#
# If I did not do this, then the container would be created
# before the set_cft_container_command was called and the
# correct command argument would not be set
new_container = cft_container().bind(upstream_tasks=[scc])
start_container = cft_start(container_id=new_container)
...
Bruno Murino
06/29/2021, 2:24 PMBruno Murino
06/29/2021, 2:26 PMBruno Murino
06/29/2021, 2:39 PMKrapi Shah
06/29/2021, 6:12 PMDeckel de Lange
06/30/2021, 11:59 AMprefect server start
where I get the following error: ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
any help would be appreciated thanks!Arkady K.
06/30/2021, 3:25 PMMadison Schott
06/30/2021, 5:15 PMazure_sql_task = FivetranSyncTask.run(
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 453, in method
kwargs.setdefault(attr, getattr(self, attr))
AttributeError: 'str' object has no attribute 'connector_id'
Thomas Opsomer
06/30/2021, 5:49 PMSerdar Tumgoren
07/01/2021, 12:31 AMnew_state.result
inside the state handler is the problem locale. Locally, it is a dict populated with (<task>, <state>) items; on Server or Cloud, new_state.result
appears to return an empty dict.
I’ve created two basic flow scripts to demonstrate this behavior (one for a local manual run, the other for Server or Cloud) and will attach them below. Each contains a NOTE
marking the divergent status of new_state.result
. Would be indebted if someone could advise on what I’m doing wrong!Dmitry Lyfar
07/01/2021, 2:31 AMBruno Murino
07/01/2021, 8:38 AMMichael Wheeler
07/01/2021, 6:05 PMservices/apollo/src/executors.js
it reads the Hasura URL from an env var, but not the secret.Chris Leber
07/02/2021, 11:00 PMdb_kwargs
account, etc
dbt = DbtShellTask(
return_all=True,
profile_name="test",
environment="dev",
overwrite_profiles=True,
log_stdout=True,
helper_script="cd dbtProject",
log_stderr=True,
dbt_kwargs={
"type": "snowflake",
"account": "${ACCOUNT}",
...
I then retrieve the account name as a prefect secret, and set it as an env variable called ACCOUNT when I run the flow:
with Flow() as flow:
account = PrefectSecret("snowflake-account")
parse = dbt(
command=f"dbt run --models ./models/{database}",
env={"ACCOUNT": account}
)
I have used this approach with ShellTasks, and it works just fine. However, it is not working with DBTShellTasks. Instead of using the value of the Prefect Secret for the account name, it is trying to connect me to snowflake with "${ACCOUNT}" as the account name.
Any ideas?? Thanks in advance for the help 😁wiretrack
07/03/2021, 6:31 PMconnection refused
in the liveness and readiness probes. does anybody have any idea on what I might be doing wrong?wiretrack
07/03/2021, 11:07 PMprefect agent kubernetes install
but I keep getting a ClientError
with Name os service not known
. what I can think of is the lack of a tenant
, so I created one with slug=default
and name=default
. I’m using PREFECT__CLOUD___API=<http://prefect-apollo-service:4200/graphql>
. RBAC
is in place, so running out of ideas on what I might be doing wrong. the UI also can’t connect to apollo-service
, but health checks for apollo are ok, and curl localhost:4200/graphql
also works inside apollo
pod.
Also tried querying apollo-service from within the agent pod with:
curl \
-X POST \
-H "Content-Type: application/json" \
--data '{ "query": "{ tenant { id } }" }' \
<http://prefect-apollo-service:4200/graphql>
And it works fine, I get the tenant Id.
Any inputs? Thanks!wiretrack
07/03/2021, 11:07 PMprefect agent kubernetes install
but I keep getting a ClientError
with Name os service not known
. what I can think of is the lack of a tenant
, so I created one with slug=default
and name=default
. I’m using PREFECT__CLOUD___API=<http://prefect-apollo-service:4200/graphql>
. RBAC
is in place, so running out of ideas on what I might be doing wrong. the UI also can’t connect to apollo-service
, but health checks for apollo are ok, and curl localhost:4200/graphql
also works inside apollo
pod.
Also tried querying apollo-service from within the agent pod with:
curl \
-X POST \
-H "Content-Type: application/json" \
--data '{ "query": "{ tenant { id } }" }' \
<http://prefect-apollo-service:4200/graphql>
And it works fine, I get the tenant Id.
Any inputs? Thanks!env_vars
: basically even if the health checks are ok, communication between services might not be working because of the way environment variables was passed, a more detailed explanation in this issue: https://github.com/PrefectHQ/prefect/issues/4738