This class just SELECTs data from a big SQL table. I want to perform this SELECT in parallel with say n sub-Tasks. So the
object should spawn n sub-Tasks when it runs. How would this work within Prefect's OOP paradigm, where
. This seems to work fine when I run the flow locally with flow.run(), but not when running it from the Prefect Cloud/Server. (The error says the module can’t be found) Does it sound right? If so, why would it be the case?
method for flow registration makes this really nice, but I'll still get an error for the project creation when the ECS task restarts. I could ignore this error, but then that could be problem if somehow the error is not due to the project already existing, but a genuine failure to create a new project.
any ideas on why this might be happening? it looks like it cannot connect to pypi for some odd reason. ideas/thoughts welcome. thanks!
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f74c98d0710>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/cryptography/ ERROR: Could not find a version that satisfies the requirement cryptography (from versions: none) ERROR: No matching distribution found for cryptography Removing intermediate container f73d8367b237 The command '/bin/sh -c pip install requests cryptography snowflake-connector-python wheel' returned a non-zero code: 1 Traceback (most recent call last): File "./flows/frieze-viewing-room-flow/flow.py", line 113, in <module> flow.storage.build() File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 351, in build self._build_image(push=push) File "/usr/local/lib/python3.6/dist-packages/prefect/environments/storage/docker.py", line 418, in _build_image "Your docker image failed to build! Your flow might have " ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included. ##[error]Bash exited with code '1'.
. I originally started with a kubernetes agent and thought it was a configuration issue between prefect and kubernetes, but I get the same behavior with a docker agent. The agents pull the correct docker image and submit it, but the flows don't enter a running state. I'd be grateful for any ideas or pointers! Cheers