https://prefect.io
Join Slack
Hey team! When using Prefect cloud for deployment with a GitHub storage block, I keep getting this e...
# data-tricks-and-tips
s
Hey team! When using Prefect cloud for deployment with a GitHub storage block, I keep getting this error and it seems to be causing the flow to be unable to run. Does anyone know what the issue may be? Block document has schema checksum sha256:196e3675d678bda766b96c223c5dd5d87223f556150ce5e6 3f55f0775feec972 which does not match the schema checksum for class 'GitHub'. This indicates the schema has changed and this block may not load.
j
Usually this warning can be safely ignored. Can you share the error traceback?
s
Here is the traceback I get when running the flow from the cloud ui. Everything looks correct in the block itself. Agent started! Looking for work from queue(s): default... 161538.875 | INFO | prefect.agent - Submitting flow run '88af556b-45c0-467e-9d52-5b5ac38e38a5' 161539.132 | INFO | prefect.infrastructure.process - Opening process 'unique-bustard'... 161539.142 | INFO | prefect.agent - Completed submission of flow run '88af556b-45c0-467e-9d52-5b5ac38e38a5' C\Users\sthibeault\PycharmProjects\prefect\venv\lib\site packages\prefect\deployments.py153: UserWarning: Block document has schema checksum sha256:196e3675d678bda766b96c223c5dd5d87223f556150ce5e6 3f55f0775feec972 which does not match the schema checksum for class 'GitHub'. This indicates the schema has changed and this block may not load. storage_block = Block._from_block_document(storage_document) 161541.857 | ERROR | Flow run 'unique-bustard' - Flow could not be retrieved from deployment. Traceback (most recent call last): File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run await storage_block.get_directory(from_path=deployment.path, local_path=".") File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\filesystems.py", line 837, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: & was unexpected at this time.
🙏 1
j
Thank you, Stephen. Interesting. Not sure why that’s happening. Recreating the block and making a new deployment with that block should work in the short term.
gratitude thank you 1
s
Hey Jeff! I have created a new block through the cloud ui a few times and keep getting the same error.
j
Hmm. And you made a new deployment with the new block? Is the GitHub repo publicly accessible?
s
I did, and the repo is private but I have an access token in the block with repo scope.
j
What’s the output from 
prefect version
in the CLI? Can you share your deployment creation command or contents of the deployment Python file?
s
below are the results of prefect version. The deploy command I am using is "prefect deployment build -n gh1 -sb github/github-test --apply prefect_flows\Livevox-snowflake-upload.py:get_livevox_users"
j
I think we found it
Upgrade prefect with 
pip install -U prefect
2.6.3 added support for private Github repos
s
Gotcha, is there anything else I would need to reinstall? I am now getting this error when running this code (I have removed the access token from the code)
Copy code
from prefect.filesystems import GitHub

github_block = GitHub(repository = '<https://github.com/sthibeaultgateway/prefect-test-4.git>',
                       access_token = 'access_token'
github_block.get_directory()
I also get the same4 error when running a deploy
j
ok
did you save the block - I don’t see that in the code above? - maybe just for now - can you make a new block in the GUI and save it and then use it to build the deployment (and give the deployment a new name just to keep things straight)?
s
Alright, I created a new block from the UI, created a new deployment, and am still getting that error when the deployment is ran in the next post I will show the CLI output
👍 1
(venv) PS C:\Users\sthibeault\PycharmProjects\prefect> prefect deployment build -n gh-new -sb github/github-block-new --apply prefect_flows\Livevox-snowflake-upload.py:get_livevox_users 095933.049 | INFO | prefect.engine - Created flow run 'steel-coyote' for flow 'get-livevox-users' 095934.069 | INFO | Flow run 'steel-coyote' - Created task run 'get_session_id-f5b5b56d-0' for task 'get_session_id' 095934.070 | INFO | Flow run 'steel-coyote' - Executing 'get_session_id-f5b5b56d-0' immediately... 095934.816 | INFO | Task run 'get_session_id-f5b5b56d-0' - Finished in state Completed() 095934.958 | INFO | Flow run 'steel-coyote' - Created task run 'get_user_list-875e7d0f-0' for task 'get_user_list' 095934.959 | INFO | Flow run 'steel-coyote' - Executing 'get_user_list-875e7d0f-0' immediately... 095935.529 | INFO | Task run 'get_user_list-875e7d0f-0' - Finished in state Completed() 095935.688 | INFO | Flow run 'steel-coyote' - Created task run 'convert_results_to_df-1b2e3f5a-0' for task 'convert_results_to_df' 095935.688 | INFO | Flow run 'steel-coyote' - Executing 'convert_results_to_df-1b2e3f5a-0' immediately... 095936.049 | INFO | Task run 'convert_results_to_df-1b2e3f5a-0' - Finished in state Completed() 095936.136 | INFO | Flow run 'steel-coyote' - Created task run 'load_dataframe_to_snowflake-b5b7c969-0' for task 'load_dataframe_to_snowflake' 095936.137 | INFO | Flow run 'steel-coyote' - Executing 'load_dataframe_to_snowflake-b5b7c969-0' immediately... 095940.215 | INFO | Flow run 'steel-coyote' - Finished in state Completed('All states completed.') Found flow 'get-livevox-users' Deployment YAML created at 'C:\Users\sthibeault\PycharmProjects\prefect\get_livevox_users-deployment.yaml'. Deployment storage GitHub(repository='https://github.com/sthibeaultgateway/prefect-test-4.git', reference=None, access_token=SecretStr('**********')) does not have upload capabilities; no files uploaded. Pass --skip-upload to suppress this warning. Deployment 'get-livevox-users/gh-new' successfully created with id '2ce90d3f-57d5-4fe9-aa04-0b4bb3e07ccc'. To execute flow runs from this deployment, start an agent that pulls work from the 'default' work queue: $ prefect agent start -q 'default' (venv) PS C:\Users\sthibeault\PycharmProjects\prefect> prefect agent start -q 'default' Starting v2.6.5 agent connected to https://api.prefect.cloud/api/accounts/8e48fde3-9f63-4f51-9931-648e3f621b03/workspaces/dc9faed8-bc98-4ccd-a4f4-fe5a9efc60c7... _ _ _ _ _ _ _____ _ _ _ _ _ _____ | _ \ _ \ __| __| __/ _| _| /_\ / __| _| \| | _| | _/ / _|| _|| | (_ | | / _ \ (_ | _|| .` | | | |_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_| Agent started! Looking for work from queue(s): default... 100044.068 | INFO | prefect.agent - Submitting flow run 'e0a15f12-2239-4e46-bfe6-3475e01f08df' 100044.374 | INFO | prefect.infrastructure.process - Opening process 'delightful-magpie'... 100044.387 | INFO | prefect.agent - Completed submission of flow run 'e0a15f12-2239-4e46-bfe6-3475e01f08df' 100047.240 | ERROR | Flow run 'delightful-magpie' - Flow could not be retrieved from deployment. Traceback (most recent call last): File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run await storage_block.get_directory(from_path=deployment.path, local_path=".") File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\filesystems.py", line 913, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: 'g' is not recognized as an internal or external command, operable program or batch file. 100047.889 | INFO | prefect.infrastructure.process - Process 'delightful-magpie' exited cleanly.
j
It looks like deployment build line is running your flow code, too. Can you put that in a name=main block before creating the deployment? Not sure that’s the issue, but I’ve seen it mess up things.
s
Sorry, I am very very new to Prefect, what exactly do you mean by a name=main block?
j
No sweat. It looks like your flow code is being called when you create your deployment - like a function was being called that then ran all your flows and tasks. Was I reading that right? If so, can you make sure any calls to functions are in a guarded code block (not a Prefect block, sorry for any confusion).
Copy code
if __name__ == "__main__":
    call_my_function_here(any_params)
s
Gotcha! I still had the call to my script to run it locally, I have removed it from both my local code and github, so function calls shouldn't be an issue. I redeployed and still get that "g is not recognized" error
👍 1
j
ok. I think the issue is with the path. Is the flow code in the top level of the github repo? if not, you can specify it with the 
path
flag. Also, to cut out a variable with slash (forward and backward with windows vs linux) directions, can you move into the folder where the flow code is to build the deployment?
s
I tried using path, then I also tried to create a new repo with only the flow file and a readme, created a new PAT for my account with complete access, creat3ed a new block using that PAT and repo, created a new deploy, and still got that same error. I was able to run a deployment that I set up with my local filesystem, github just seems to be having issues.
p
Howdy! Prefect dev here. There error 
'g' is not recognized as an internal or external command, operable program or batch file.
is possibly a windows-specific bug due to how we're executing the command with the GitHub block.
I'll be able to look at this a little closer in about an hour, but I think I have an idea of what's happening.
🙏 2
j
I asked for backup b/c I have to run @Stephen Thibeault but optimistic a fix will be on the way soon. Thanks for your patience digging into this!
s
Thank you! I appreciate all of the help!
👍 2
p
Alright - reproduced the problem
🙌 1
and fixed on Windows. Need to double check that it also works on unix
I'll try to have this fix out bright and early next week. Thanks a ton for catching this @Stephen Thibeault
s
Thank you so much for all of your help! Please let me know when I can give it another try early nest week!
✔️ 1
p
Here's the PR: https://github.com/PrefectHQ/prefect/pull/7372 . It will be in the next release. Once it's merged, you'll still have to wait for a release unless you want to pip install from main. Releases are announced in the #CKNSX5WG3 channel. So if you see this PR is merged and a new release announcement, you should be good to go!
🎉 2
🙏 1
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by