Ajith Kumara Beragala Acharige Lal
02/23/2021, 5:50 PMGITLab repo
in my prefect-server, can someone help me to figure-out what is the mistake in my code? the error Failed to load and execute Flow's environment: GitlabGetError('404 Project Not Found')
f.storage = GitLab( host="<https://hakko.sekai.dev>",
repo="sekai-backend/spark-runner",
path="kikai/charts/kikai/DailySummariesJob/kikai_dailySummaries.py" ,
ref="dev-prefect")
h*<ttps://hakko.sekai.dev/sekai-backend/spark-runner/blob/dev-prefect/kikai/charts/kikai/DailySummariesJob/kikai_dailySummaries.py>*
Jim Crist-Harif
02/23/2021, 5:55 PMaccess_token_secret
kwarg to `GitLab`: https://docs.prefect.io/orchestration/flow_config/storage.html#gitlabAjith Kumara Beragala Acharige Lal
02/23/2021, 5:58 PMFailed to load and execute Flow's environment: ValueError('Local Secret "XXXXXXXXX" was not found.')
, in the doc it says
# name of personal access token secret
( but I passed the access token) where should I specify( configure) the *name* of personal access token
?Jim Crist-Harif
02/23/2021, 6:08 PMGITLAB_ACCESS_TOKEN
somewhere (either in cloud, or as a local secret), then pass in access_token_secret="GITLAB_ACCESS_TOKEN"
to GitLab
storage.Ajith Kumara Beragala Acharige Lal
02/23/2021, 6:11 PMprefect-server
runs on kubernetes cluster , in that case , should i configure it on the agent? or is there any other place?Jim Crist-Harif
02/23/2021, 6:13 PMAjith Kumara Beragala Acharige Lal
02/23/2021, 6:36 PMFailed to load and execute Flow's environment: ValueError('No flow found in file.')
prefect register flow -f flow.py -n flow_dailySummaries
Jim Crist-Harif
02/23/2021, 6:38 PMh*<ttps://hakko.sekai.dev/sekai-backend/spark-runner/blob/dev-prefect/kikai/charts/kikai/DailySummariesJob/kikai_dailySummaries.py>*
contain a flow? Prefect will look for prefect.Flow
objects in the file that match the name of the flow provided.Ajith Kumara Beragala Acharige Lal
02/23/2021, 6:38 PMTypeError: 'project_name' is a required field when registering a flow.
Jim Crist-Harif
02/23/2021, 6:40 PM--project
flag in the CLI.--name
flag, that will be inferred automatically from the flow.ref
in the GitLab
storage - is the file up to date on that branch?Ajith Kumara Beragala Acharige Lal
02/23/2021, 6:53 PMNo flow found in file.
<string>:16: UserWarning: Attempting to call `flow.register` during execution of flow file will lead to unexpected results.
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 90, in flow_run
raise exc
File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 67, in flow_run
flow = storage.get_flow(flow_data.name)
File "/usr/local/lib/python3.7/site-packages/prefect/storage/gitlab.py", line 106, in get_flow
file_contents=contents.decode(), flow_name=flow_name
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 98, in extract_flow_from_file
raise ValueError("No flow found in file.")
ValueError: No flow found in file.
Jim Crist-Harif
02/23/2021, 7:04 PMflow_dailySummaries
as registered in cloud? Or does the file in cloud not match that name?Ajith Kumara Beragala Acharige Lal
02/23/2021, 7:09 PMJim Crist-Harif
02/23/2021, 7:13 PMAjith Kumara Beragala Acharige Lal
02/23/2021, 7:20 PMJim Crist-Harif
02/23/2021, 7:34 PMcontents
and flow_name
from here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/gitlab.py#L105. Adding
<http://self.logger.info|self.logger.info>("File contents: %s", contents.decode())
<http://self.logger.info|self.logger.info>("Flow name: %s", flow_name)
before that line should get the info we want.Ajith Kumara Beragala Acharige Lal
02/23/2021, 7:38 PMimport prefect
from prefect.tasks.shell import ShellTask
from prefect import task, Flow, Task
from prefect.run_configs import LocalRun
tasking = ShellTask(helper_script="ls -l",return_all=True,log_stdout=True,log_stderr=True)
#with Flow("flow_punctualityDailySummaries",schedule=schedule)
with Flow("flow_dailySummaries") as f:
def run(self):
<http://self.logger.info|self.logger.info>("File contents: %s", contents.decode())
<http://self.logger.info|self.logger.info>("Flow name: %s", f)
execute_command = tasking(command='./deploy-dev.sh ')
f.run_config = LocalRun()
f.register(project_name="project_dailySummaries")
Jim Crist-Harif
02/23/2021, 8:23 PMAjith Kumara Beragala Acharige Lal
02/23/2021, 8:24 PMJim Crist-Harif
02/23/2021, 8:24 PMAjith Kumara Beragala Acharige Lal
02/23/2021, 8:26 PMJim Crist-Harif
02/23/2021, 8:26 PMAjith Kumara Beragala Acharige Lal
02/23/2021, 8:26 PMJim Crist-Harif
02/23/2021, 8:56 PMfrom urllib.parse import quote_plus
import prefect
from prefect.utilities.graphql import with_args
from prefect.serialization.storage import StorageSchema
from gitlab import Gitlab
# Put your flow id here. This should be a UUID:
FLOW_ID = "..."
GITLAB_HOST = "<https://hakko.sekai.dev>"
# Your gitlab access token goes here
GITLAB_ACCESS_TOKEN = "..."
# Attempt to load the flow
client = prefect.Client()
info = client.graphql(
{"query": {with_args("flow", {"where": {"id": {"_eq": FLOW_ID}}}): {"name", "storage"}}}
)
flow_name = info.data.flow[0].name
flow_storage = info.data.flow[0].storage
print(f"Flow name: {flow_name}")
print(f"Flow storage: {flow_storage}")
# Attempt to load the flow from storage
storage = StorageSchema().load(flow_storage)
print(f"Storage type: {storage}")
try:
storage.get_flow(flow_name)
except Exception as exc:
print(f"Failed to load flow: {exc}")
gitlab = Gitlab(GITLAB_HOST, private_token=GITLAB_ACCESS_TOKEN)
print(f"Accessing repo {storage.repo}")
project = gitlab.projects.get(quote_plus(storage.repo))
flow_location = storage.flows[flow_name]
print(f"Loading flow from {flow_location} at ref {storage.ref}")
contents = project.files.get(file_path=flow_location, ref=storage.ref)
print(f"Flow contents: {contents.decode()}")
GITLAB_HOST
: the address of your gitlab host. I believe I've entered this correctly from your code above.
• `GITLAB_ACCESS_TOKEN`: the private access token for your gitlab account (the actual token value, not the name of the secret like before).Ajith Kumara Beragala Acharige Lal
02/23/2021, 9:18 PMstorage = StorageSchema().load(flow_storage)
print(f"Storage type: {storage}")
try:
storage.get_flow(flow_name)
except Exception as exc:
print(f"Failed to load flow: {exc}")
gitlab = GitLab( host="<https://hakko.sekai.dev>",
repo="sekai-backend/spark-runner",
ref="dev-prefect",
access_token_secret=GITLAB_ACCESS_TOKEN)
print(f"Accessing repo {storage.repo}")
print(f"Accessing repo {storage.repo}")
Jim Crist-Harif
02/23/2021, 9:36 PMGitLab
storage. You can see that in the first line of the output (<Storage: Local>
). So the flow you ran the script for either isn't the same one as the one you were executing before, or was re-registered with a different storage type.Ajith Kumara Beragala Acharige Lal
02/23/2021, 9:41 PMJim Crist-Harif
02/23/2021, 9:43 PMFLOW_ID
parameter with the flow id of the flow that was failing above. From the output you've given, it looks like the flow you ran this for is using local storage (not gitlab storage), which doesn't match your flow from before. So either that flow changed, or you have the wrong flow id.Ajith Kumara Beragala Acharige Lal
02/23/2021, 9:46 PMJim Crist-Harif
02/23/2021, 9:47 PMrun_config
. Your flow script there doesn't set flow.storage
to GitLab
though, so it will use Local
storage by default.Ajith Kumara Beragala Acharige Lal
02/23/2021, 10:24 PMFailed to load flow: Unable to import gitlab, please ensure you have installed the gitlab extra
Flow name: KubeTestFlow
Flow storage: {'ref': 'dev-prefect', 'host': '<https://hakko.sekai.dev>', 'path': 'kikai/charts/kikai/DailySummariesJob/kikai_dailySummaries.py', 'repo': 'sekai-backend/spark-runner', 'type': 'GitLab', 'flows': {'KubeTestFlow': 'kikai/charts/kikai/DailySummariesJob/kikai_dailySummaries.py'}, 'secrets': [], '__version__': '0.14.7', 'access_token_secret': 'GITLAB_ACCESS_TOKEN'}
Storage type: <Storage: GitLab>
Failed to load flow: Unable to import gitlab, please ensure you have installed the gitlab extra
Accessing repo sekai-backend/spark-runner
Traceback (most recent call last):
File "prefecTest.py", line 33, in <module>
project = gitlab.projects.get(quote_plus(storage.repo))
AttributeError: 'GitLab' object has no attribute 'projects'
ValueError('No flow found in file.')
Jim Crist-Harif
02/23/2021, 10:32 PMfrom gitlab import Gitlab
to from prefect.storage import GitLab
. The original import was correct. You'll need to install the prefect gitlab extra, as the exception said. pip install prefect[gitlab]
.Ajith Kumara Beragala Acharige Lal
02/23/2021, 10:36 PMpip install prefect[gitlab]
would install gitlab locally, but how can I install it on kubernetes pod? I have installed prefect-server by using helm chart - do I have to re-install prefect-server again?Jim Crist-Harif
02/23/2021, 10:57 PMgitlab
in the pod running your actual flow. By default the prefecthq/prefect
images already have that installed, and from the error messages you see in the pods, this is not your issue (the original error message you sent had no issues importing gitlab).gitlab
locally and run the script I sent you to get some more debug logs on the flow?Ajith Kumara Beragala Acharige Lal
02/23/2021, 11:04 PMfrom gitlab import Gitlab
, but then again i faced bellow errorJim Crist-Harif
02/23/2021, 11:05 PMgitlab
installed locally. The from gitlab import Gitlab
import is the correct one.pip install python-gitlab
Ajith Kumara Beragala Acharige Lal
02/23/2021, 11:07 PMJim Crist-Harif
02/23/2021, 11:08 PMpip3
and python3
are using different python installations.Ajith Kumara Beragala Acharige Lal
02/24/2021, 1:53 PMJim Crist-Harif
02/24/2021, 3:04 PMAjith Kumara Beragala Acharige Lal
02/24/2021, 3:11 PMJim Crist-Harif
02/24/2021, 3:12 PMAjith Kumara Beragala Acharige Lal
02/24/2021, 3:12 PM