https://prefect.io logo
Powered by
s

Stephen Thibeault

10/27/2022, 8:06 PM
Hey team! When using Prefect cloud for deployment with a GitHub storage block, I keep getting this error and it seems to be causing the flow to be unable to run. Does anyone know what the issue may be? Block document has schema checksum sha256:196e3675d678bda766b96c223c5dd5d87223f556150ce5e6 3f55f0775feec972 which does not match the schema checksum for class 'GitHub'. This indicates the schema has changed and this block may not load.
j

Jeff Hale

10/27/2022, 8:12 PM
Usually this warning can be safely ignored. Can you share the error traceback?
s

Stephen Thibeault

10/27/2022, 8:18 PM
Here is the traceback I get when running the flow from the cloud ui. Everything looks correct in the block itself. Agent started! Looking for work from queue(s): default... 161538.875 | INFO | prefect.agent - Submitting flow run '88af556b-45c0-467e-9d52-5b5ac38e38a5' 161539.132 | INFO | prefect.infrastructure.process - Opening process 'unique-bustard'... 161539.142 | INFO | prefect.agent - Completed submission of flow run '88af556b-45c0-467e-9d52-5b5ac38e38a5' C\Users\sthibeault\PycharmProjects\prefect\venv\lib\site packages\prefect\deployments.py153: UserWarning: Block document has schema checksum sha256:196e3675d678bda766b96c223c5dd5d87223f556150ce5e6 3f55f0775feec972 which does not match the schema checksum for class 'GitHub'. This indicates the schema has changed and this block may not load. storage_block = Block._from_block_document(storage_document) 161541.857 | ERROR | Flow run 'unique-bustard' - Flow could not be retrieved from deployment. Traceback (most recent call last): File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run await storage_block.get_directory(from_path=deployment.path, local_path=".") File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\filesystems.py", line 837, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: & was unexpected at this time.
🙏 1
j

Jeff Hale

10/27/2022, 8:28 PM
Thank you, Stephen. Interesting. Not sure why that’s happening. Recreating the block and making a new deployment with that block should work in the short term.
gratitude thank you 1
s

Stephen Thibeault

10/28/2022, 12:10 PM
Hey Jeff! I have created a new block through the cloud ui a few times and keep getting the same error.
j

Jeff Hale

10/28/2022, 12:14 PM
Hmm. And you made a new deployment with the new block? Is the GitHub repo publicly accessible?
s

Stephen Thibeault

10/28/2022, 12:18 PM
I did, and the repo is private but I have an access token in the block with repo scope.
j

Jeff Hale

10/28/2022, 12:59 PM
What’s the output from 
prefect version
in the CLI? Can you share your deployment creation command or contents of the deployment Python file?
s

Stephen Thibeault

10/28/2022, 1:09 PM
below are the results of prefect version. The deploy command I am using is "prefect deployment build -n gh1 -sb github/github-test --apply prefect_flows\Livevox-snowflake-upload.py:get_livevox_users"
j

Jeff Hale

10/28/2022, 1:12 PM
I think we found it
Upgrade prefect with 
pip install -U prefect
2.6.3 added support for private Github repos
s

Stephen Thibeault

10/28/2022, 1:28 PM
Gotcha, is there anything else I would need to reinstall? I am now getting this error when running this code (I have removed the access token from the code)
Copy code
from prefect.filesystems import GitHub

github_block = GitHub(repository = '<https://github.com/sthibeaultgateway/prefect-test-4.git>',
                       access_token = 'access_token'
github_block.get_directory()
I also get the same4 error when running a deploy
j

Jeff Hale

10/28/2022, 1:52 PM
ok
did you save the block - I don’t see that in the code above? - maybe just for now - can you make a new block in the GUI and save it and then use it to build the deployment (and give the deployment a new name just to keep things straight)?
s

Stephen Thibeault

10/28/2022, 2:01 PM
Alright, I created a new block from the UI, created a new deployment, and am still getting that error when the deployment is ran in the next post I will show the CLI output
👍 1
(venv) PS C:\Users\sthibeault\PycharmProjects\prefect> prefect deployment build -n gh-new -sb github/github-block-new --apply prefect_flows\Livevox-snowflake-upload.py:get_livevox_users 095933.049 | INFO | prefect.engine - Created flow run 'steel-coyote' for flow 'get-livevox-users' 095934.069 | INFO | Flow run 'steel-coyote' - Created task run 'get_session_id-f5b5b56d-0' for task 'get_session_id' 095934.070 | INFO | Flow run 'steel-coyote' - Executing 'get_session_id-f5b5b56d-0' immediately... 095934.816 | INFO | Task run 'get_session_id-f5b5b56d-0' - Finished in state Completed() 095934.958 | INFO | Flow run 'steel-coyote' - Created task run 'get_user_list-875e7d0f-0' for task 'get_user_list' 095934.959 | INFO | Flow run 'steel-coyote' - Executing 'get_user_list-875e7d0f-0' immediately... 095935.529 | INFO | Task run 'get_user_list-875e7d0f-0' - Finished in state Completed() 095935.688 | INFO | Flow run 'steel-coyote' - Created task run 'convert_results_to_df-1b2e3f5a-0' for task 'convert_results_to_df' 095935.688 | INFO | Flow run 'steel-coyote' - Executing 'convert_results_to_df-1b2e3f5a-0' immediately... 095936.049 | INFO | Task run 'convert_results_to_df-1b2e3f5a-0' - Finished in state Completed() 095936.136 | INFO | Flow run 'steel-coyote' - Created task run 'load_dataframe_to_snowflake-b5b7c969-0' for task 'load_dataframe_to_snowflake' 095936.137 | INFO | Flow run 'steel-coyote' - Executing 'load_dataframe_to_snowflake-b5b7c969-0' immediately... 095940.215 | INFO | Flow run 'steel-coyote' - Finished in state Completed('All states completed.') Found flow 'get-livevox-users' Deployment YAML created at 'C:\Users\sthibeault\PycharmProjects\prefect\get_livevox_users-deployment.yaml'. Deployment storage GitHub(repository='https://github.com/sthibeaultgateway/prefect-test-4.git', reference=None, access_token=SecretStr('**********')) does not have upload capabilities; no files uploaded. Pass --skip-upload to suppress this warning. Deployment 'get-livevox-users/gh-new' successfully created with id '2ce90d3f-57d5-4fe9-aa04-0b4bb3e07ccc'. To execute flow runs from this deployment, start an agent that pulls work from the 'default' work queue: $ prefect agent start -q 'default' (venv) PS C:\Users\sthibeault\PycharmProjects\prefect> prefect agent start -q 'default' Starting v2.6.5 agent connected to https://api.prefect.cloud/api/accounts/8e48fde3-9f63-4f51-9931-648e3f621b03/workspaces/dc9faed8-bc98-4ccd-a4f4-fe5a9efc60c7... _ _ _ _ _ _ _____ _ _ _ _ _ _____ | _ \ _ \ __| __| __/ _| _| /_\ / __| _| \| | _| | _/ / _|| _|| | (_ | | / _ \ (_ | _|| .` | | | |_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_| Agent started! Looking for work from queue(s): default... 100044.068 | INFO | prefect.agent - Submitting flow run 'e0a15f12-2239-4e46-bfe6-3475e01f08df' 100044.374 | INFO | prefect.infrastructure.process - Opening process 'delightful-magpie'... 100044.387 | INFO | prefect.agent - Completed submission of flow run 'e0a15f12-2239-4e46-bfe6-3475e01f08df' 100047.240 | ERROR | Flow run 'delightful-magpie' - Flow could not be retrieved from deployment. Traceback (most recent call last): File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run flow = await load_flow_from_flow_run(flow_run, client=client) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client return await fn(*args, **kwargs) File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\deployments.py", line 159, in load_flow_from_flow_run await storage_block.get_directory(from_path=deployment.path, local_path=".") File "C:\Users\sthibeault\PycharmProjects\prefect\venv\lib\site-packages\prefect\filesystems.py", line 913, in get_directory raise OSError(f"Failed to pull from remote:\n {err_stream.read()}") OSError: Failed to pull from remote: 'g' is not recognized as an internal or external command, operable program or batch file. 100047.889 | INFO | prefect.infrastructure.process - Process 'delightful-magpie' exited cleanly.
j

Jeff Hale

10/28/2022, 2:10 PM
It looks like deployment build line is running your flow code, too. Can you put that in a name=main block before creating the deployment? Not sure that’s the issue, but I’ve seen it mess up things.
s

Stephen Thibeault

10/28/2022, 2:13 PM
Sorry, I am very very new to Prefect, what exactly do you mean by a name=main block?
j

Jeff Hale

10/28/2022, 2:24 PM
No sweat. It looks like your flow code is being called when you create your deployment - like a function was being called that then ran all your flows and tasks. Was I reading that right? If so, can you make sure any calls to functions are in a guarded code block (not a Prefect block, sorry for any confusion).
Copy code
if __name__ == "__main__":
    call_my_function_here(any_params)
s

Stephen Thibeault

10/28/2022, 2:30 PM
Gotcha! I still had the call to my script to run it locally, I have removed it from both my local code and github, so function calls shouldn't be an issue. I redeployed and still get that "g is not recognized" error
👍 1
j

Jeff Hale

10/28/2022, 2:55 PM
ok. I think the issue is with the path. Is the flow code in the top level of the github repo? if not, you can specify it with the 
path
flag. Also, to cut out a variable with slash (forward and backward with windows vs linux) directions, can you move into the folder where the flow code is to build the deployment?
s

Stephen Thibeault

10/28/2022, 4:01 PM
I tried using path, then I also tried to create a new repo with only the flow file and a readme, created a new PAT for my account with complete access, creat3ed a new block using that PAT and repo, created a new deploy, and still got that same error. I was able to run a deployment that I set up with my local filesystem, github just seems to be having issues.
p

Peyton Runyan

10/28/2022, 4:14 PM
Howdy! Prefect dev here. There error 
'g' is not recognized as an internal or external command, operable program or batch file.
is possibly a windows-specific bug due to how we're executing the command with the GitHub block.
I'll be able to look at this a little closer in about an hour, but I think I have an idea of what's happening.
🙏 2
j

Jeff Hale

10/28/2022, 4:27 PM
I asked for backup b/c I have to run @Stephen Thibeault but optimistic a fix will be on the way soon. Thanks for your patience digging into this!
s

Stephen Thibeault

10/28/2022, 4:36 PM
Thank you! I appreciate all of the help!
👍 2
p

Peyton Runyan

10/28/2022, 7:47 PM
Alright - reproduced the problem
🙌 1
and fixed on Windows. Need to double check that it also works on unix
I'll try to have this fix out bright and early next week. Thanks a ton for catching this @Stephen Thibeault
s

Stephen Thibeault

10/28/2022, 7:54 PM
Thank you so much for all of your help! Please let me know when I can give it another try early nest week!
✔️ 1
p

Peyton Runyan

10/28/2022, 8:19 PM
Here's the PR: https://github.com/PrefectHQ/prefect/pull/7372 . It will be in the next release. Once it's merged, you'll still have to wait for a release unless you want to pip install from main. Releases are announced in the #announcements channel. So if you see this PR is merged and a new release announcement, you should be good to go!
🎉 2
🙏 1
5 Views
Powered by