Apoorva Desai04/14/2022, 12:20 AM
Alexander Butler04/14/2022, 1:57 AM
Assuming interval is seconds? Can I specify another grain? Can schedule take a dict? If it takes cron, does that take a dict? Honestly schedule is the primary question point. Everything else is straightforward enough.
- name: elt-salesforce flow_location: ./salesforce_flows.py flow_name: elt-salesforce tags: - salesforce - core parameters: destination: "gcp" schedule: interval: 3600
Salim Doost04/14/2022, 2:53 AM
However, we’re able to confirm that the image with this tag exists on EMR. Updating an existing flow by overriding an existing image-tag leads to the following error:
404 Client Error for <http+docker://localhost/v1.41/containers/create?name=quantum-squid>: Not Found ("No such image: <account-id>.<http://dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>%22|dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>">)
Again, we’re able to confirm in AWS ECR that the image got pushed and updated successfully. Our deployment job didn’t throw any error messages as well. Any idea what we can do to resolve this issue?
KeyError: 'Task slug <task-name> is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage. - Did you change the flow without re-registering it? - Did you register the flow without updating it in your storage location (if applicable)?'
Carlos Cueto04/14/2022, 4:27 AM
parameter. Whenever I specify the following:
Whenever I register the flow (I'm using Prefect 1.2.0 on MacOS python 3.10) I get the following working_dir on the UI of Prefect Cloud:
flow.run_config = LocalRun(_working_dir_='C:/scripts/GetADUsers', _labels_=["SVRBIPTH01"])
It seems to be adding the path from where I register the script from (on the local machine) to the working_dir string I specified on the run_config. Has anybody encountered this before? Everything works fine when I register the flow from a Windows computer.
Alexander Butler04/14/2022, 6:13 AM
prefect orion start
prefect deployment create ...
HERE IS THE GAP -- the response to the above command is something like
prefect work-queue create -t etl -- etl-queue
, which is useless when setting something up from the CLI without sed/awk?
The less appealing part afterwards is that
prefect agent start 'no simple headless way to derive id...'
renders a table which is pretty in a CLI but useless again to simply get an ID. Has anyone set up Prefect 2.0 to self deploy in an image along with all their code? The ephemeral nature makes this very advantageous with what seems to be a tiny unconsidered gap. I am pretty sure a more reliable consistent way to get work queue is all thats needed basically, but if I am totally missing it just lmk. I am a big fan of the package for the record but now its crunch time production use attempts 🙂
prefect work-queue ls
Jacob Blanco04/14/2022, 6:32 AM
Stéphanie Cérulli04/14/2022, 6:43 AM
Stéphanie Cérulli04/14/2022, 6:44 AM
2022-04-14T06:42:11.384990+00:00 - - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
Luuk04/14/2022, 6:45 AM
Jacob Blanco04/14/2022, 7:02 AM
Jars04/14/2022, 7:14 AM
Evan04/14/2022, 7:15 AM
Stephen Lloyd04/14/2022, 7:15 AM
Jacob Blanco04/14/2022, 7:16 AM
Jars04/14/2022, 7:22 AM
Gaurav kumar04/14/2022, 7:31 AM
Jars04/14/2022, 7:36 AM
Chris White04/14/2022, 7:39 AM
Jacob Blanco04/14/2022, 7:39 AM
Stephen Lloyd04/14/2022, 7:48 AM
creds is passed in from a PrefectSecret task.
Traceback (most recent call last): File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state value = prefect.utilities.executors.run_task_with_timeout( File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout return task.run(*args, **kwargs) # type: ignore File "workable/src/flow.py", line 64, in fivetran_sync status = FivetranSyncTask.run( TypeError: method() missing 1 required positional argument: 'self'
@task def fivetran_sync(connector: str, creds: dict) -> dict: status = FivetranSyncTask.run( api_key=creds['api_key'], api_secret=creds['api_secret'], connector_id=connector ) return status
Chris White04/14/2022, 7:56 AM
John Muddle04/14/2022, 12:48 PM
Geert-Jan Van den Bogaerde04/14/2022, 3:13 PM
Mike04/14/2022, 4:13 PM
Alexander Gorokhov04/14/2022, 4:24 PM
Max Kolasinski04/14/2022, 4:36 PM
Jason04/14/2022, 4:39 PM
Has anyone run into a similar exception? I'm successfully authed to ECR from my local docker --login as well.
Jason04/14/2022, 5:22 PM
Alexander Belikov04/14/2022, 5:39 PM
Alexander Belikov04/14/2022, 5:39 PM
but then to register it without parameters... This example essentially emulates my case
Anna Geller04/14/2022, 5:47 PM
Kevin Kho04/14/2022, 5:47 PM
task. On Server, you need to store local secrets. You can also see more on that here 2. On the parameters, you can also do
so that you can register without a default 3. Parameters defaults on clocks/schedule don’t fulfill the requirement of having a default parameter because you can still trigger ad-hoc runs that would then not have any filled value.
Alexander Belikov04/15/2022, 10:19 AM
- I want to be able to register and schedule flows with parameters that are required
Anna Geller04/15/2022, 10:25 AM
sounds good - I'll keep posting on StackOverflow since it might help wider audience.Actually, could you post to Discourse instead? This is our preferred platform
Alexander Belikov04/15/2022, 1:14 PM
Kevin Kho04/15/2022, 1:39 PM