Apoorva Desai
04/14/2022, 12:20 AMAlexander Butler
04/14/2022, 1:57 AM- name: elt-salesforce
flow_location: ./salesforce_flows.py
flow_name: elt-salesforce
tags:
- salesforce
- core
parameters:
destination: "gcp"
schedule:
interval: 3600
Assuming interval is seconds? Can I specify another grain? Can schedule take a dict? If it takes cron, does that take a dict?
Honestly schedule is the primary question point. Everything else is straightforward enough.Salim Doost
04/14/2022, 2:53 AM404 Client Error for <http+docker://localhost/v1.41/containers/create?name=quantum-squid>: Not Found ("No such image: <account-id>.<http://dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>%22|dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>">)
However, we’re able to confirm that the image with this tag exists on EMR.
Updating an existing flow by overriding an existing image-tag leads to the following error:
KeyError: 'Task slug <task-name> is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.
- Did you change the flow without re-registering it?
- Did you register the flow without updating it in your storage location (if applicable)?'
Again, we’re able to confirm in AWS ECR that the image got pushed and updated successfully. Our deployment job didn’t throw any error messages as well.
Any idea what we can do to resolve this issue?Carlos Cueto
04/14/2022, 4:27 AMLocalRun
flow's working_dir
parameter. Whenever I specify the following:
flow.run_config = LocalRun(_working_dir_='C:/scripts/GetADUsers', _labels_=["SVRBIPTH01"])
Whenever I register the flow (I'm using Prefect 1.2.0 on MacOS python 3.10) I get the following working_dir on the UI of Prefect Cloud:
/Users/carloscueto/Documents/Python_Scripts/Prefect-Flows/PowerShell/GetADUsers/C:/scripts/GetADUsers
It seems to be adding the path from where I register the script from (on the local machine) to the working_dir string I specified on the run_config.
Has anybody encountered this before? Everything works fine when I register the flow from a Windows computer.Alexander Butler
04/14/2022, 6:13 AMprefect orion start
prefect deployment create ...
prefect work-queue create -t etl -- etl-queue
HERE IS THE GAP -- the response to the above command is something like UUID('...')
, which is useless when setting something up from the CLI without sed/awk?
prefect agent start 'no simple headless way to derive id...'
The less appealing part afterwards is that prefect work-queue ls
renders a table which is pretty in a CLI but useless again to simply get an ID.
Has anyone set up Prefect 2.0 to self deploy in an image along with all their code? The ephemeral nature makes this very advantageous with what seems to be a tiny unconsidered gap.
I am pretty sure a more reliable consistent way to get work queue is all thats needed basically, but if I am totally missing it just lmk.
I am a big fan of the package for the record but now its crunch time production use attempts 🙂Jacob Blanco
04/14/2022, 6:32 AMStéphanie Cérulli
04/14/2022, 6:43 AMStéphanie Cérulli
04/14/2022, 6:44 AM2022-04-14T06:42:11.384990+00:00 - - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
Luuk
04/14/2022, 6:45 AMJacob Blanco
04/14/2022, 6:49 AMJacob Blanco
04/14/2022, 7:02 AMJars
04/14/2022, 7:14 AMEvan
04/14/2022, 7:15 AMStephen Lloyd
04/14/2022, 7:15 AMJacob Blanco
04/14/2022, 7:16 AMJars
04/14/2022, 7:22 AMGaurav kumar
04/14/2022, 7:31 AMJars
04/14/2022, 7:36 AMChris White
04/14/2022, 7:39 AMJacob Blanco
04/14/2022, 7:39 AMStephen Lloyd
04/14/2022, 7:48 AMTraceback (most recent call last):
File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "workable/src/flow.py", line 64, in fivetran_sync
status = FivetranSyncTask.run(
TypeError: method() missing 1 required positional argument: 'self'
creds is passed in from a PrefectSecret task.
@task
def fivetran_sync(connector: str, creds: dict) -> dict:
status = FivetranSyncTask.run(
api_key=creds['api_key'],
api_secret=creds['api_secret'],
connector_id=connector
)
return status
Chris White
04/14/2022, 7:56 AMJohn Muddle
04/14/2022, 12:48 PMGeert-Jan Van den Bogaerde
04/14/2022, 3:13 PMMike
04/14/2022, 4:13 PMAlexander Gorokhov
04/14/2022, 4:24 PMMax Kolasinski
04/14/2022, 4:36 PMJason
04/14/2022, 4:39 PM<https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/docker.py#L610-L613>
Has anyone run into a similar exception? I'm successfully authed to ECR from my local docker --login as well.Jason
04/14/2022, 5:22 PMAlexander Belikov
04/14/2022, 5:39 PMAlexander Belikov
04/14/2022, 5:39 PMrun
but then to register it without parameters... This example essentially emulates my caseAnna Geller
04/14/2022, 5:47 PMKevin Kho
04/14/2022, 5:47 PMPrefectSecret
task. On Server, you need to store local secrets. You can also see more on that here
2. On the parameters, you can also do required=False
so that you can register without a default
3. Parameters defaults on clocks/schedule don’t fulfill the requirement of having a default parameter because you can still trigger ad-hoc runs that would then not have any filled value.prefect
tagAlexander Belikov
04/15/2022, 10:19 AMrequired=False
- I want to be able to register and schedule flows with parameters that are requiredAnna Geller
04/15/2022, 10:25 AMsounds good - I'll keep posting on StackOverflow since it might help wider audience.Actually, could you post to Discourse instead? This is our preferred platform
Alexander Belikov
04/15/2022, 1:14 PMKevin Kho
04/15/2022, 1:39 PM