prefect-community #prefect-community

prefect-community

Apoorva Desai

04/14/2022, 12:20 AM

Hello! I have a custom docker container for my prefect flows and I have copied over a python file with some custom functions onto this docker container. However, my prefect flow is unable to import these functions. I am getting a module not found error. How do you access functions from python files in custom docker containers in prefect flow?

✅ 1

Alexander Butler

04/14/2022, 1:57 AM

I am working on standing up Prefect 2.0 is a production environment. For internal data pipeline and reverse etl uses so no fire hazards on my end to use 2.0 early here. Is there a general preference on YAML vs Code for the deployment specification. I noticed you can configure a flow deployment with YAML but I cant find any information on the schema of that document. For example:

- name: elt-salesforce
  flow_location: ./salesforce_flows.py
  flow_name: elt-salesforce
  tags:
    - salesforce
    - core
  parameters:
    destination: "gcp"
  schedule:
    interval: 3600

Assuming interval is seconds? Can I specify another grain? Can schedule take a dict? If it takes cron, does that take a dict? Honestly schedule is the primary question point. Everything else is straightforward enough.

:discourse: 1

Salim Doost

04/14/2022, 2:53 AM

We’re having issues with deployments on one of our Prefect workspaces all of a sudden (without changing anything on our setup to the best of our knowledge). Our flows are stored as Docker images in AWS ECR. Running a newly created flow leads to the following error:

404 Client Error for <http+docker://localhost/v1.41/containers/create?name=quantum-squid>: Not Found ("No such image: <account-id>.<http://dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>%22|dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>">)

However, we’re able to confirm that the image with this tag exists on EMR. Updating an existing flow by overriding an existing image-tag leads to the following error:

KeyError: 'Task slug <task-name> is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.
- Did you change the flow without re-registering it?
- Did you register the flow without updating it in your storage location (if applicable)?'

Again, we’re able to confirm in AWS ECR that the image got pushed and updated successfully. Our deployment job didn’t throw any error messages as well. Any idea what we can do to resolve this issue?

Carlos Cueto

04/14/2022, 4:27 AM

Hi. I'm having an issue using a

LocalRun

flow's

working_dir

parameter. Whenever I specify the following:

flow.run_config = LocalRun(_working_dir_='C:/scripts/GetADUsers', _labels_=["SVRBIPTH01"])

Whenever I register the flow (I'm using Prefect 1.2.0 on MacOS python 3.10) I get the following working_dir on the UI of Prefect Cloud:

/Users/carloscueto/Documents/Python_Scripts/Prefect-Flows/PowerShell/GetADUsers/C:/scripts/GetADUsers

It seems to be adding the path from where I register the script from (on the local machine) to the working_dir string I specified on the run_config. Has anybody encountered this before? Everything works fine when I register the flow from a Windows computer.

✅ 1

Alexander Butler

04/14/2022, 6:13 AM

One more question thats far more importantly than the YAML flow deployment config... If I want to bundle a self contained application as a docker image which required no ui interfacing to stand up Prefect 2.0, so

prefect orion start

prefect deployment create ...

prefect work-queue create -t etl -- etl-queue

HERE IS THE GAP -- the response to the above command is something like

UUID('...')

, which is useless when setting something up from the CLI without sed/awk?

prefect agent start 'no simple headless way to derive id...'

The less appealing part afterwards is that

prefect work-queue ls

renders a table which is pretty in a CLI but useless again to simply get an ID. Has anyone set up Prefect 2.0 to self deploy in an image along with all their code? The ephemeral nature makes this very advantageous with what seems to be a tiny unconsidered gap. I am pretty sure a more reliable consistent way to get work queue is all thats needed basically, but if I am totally missing it just lmk. I am a big fan of the package for the record but now its crunch time production use attempts 🙂

Jacob Blanco

04/14/2022, 6:32 AM

Anyone else seeing timeouts against the Prefect API? The Cloud view is not updating consistently and many of our Cloud-managed flows are not starting up.

✅ 3

Stéphanie Cérulli

04/14/2022, 6:43 AM

Yes we have errors on API too

Stéphanie Cérulli

04/14/2022, 6:44 AM

2022-04-14T06:42:11.384990+00:00 - -  urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))

Luuk

04/14/2022, 6:45 AM

Same here, also can't login into the cloud with a new API key

Jacob Blanco

04/14/2022, 6:49 AM

The status page is still green: https://prefect.status.io

Jacob Blanco

04/14/2022, 7:02 AM

@Chris White Is anyone looking at this??

Jars

04/14/2022, 7:14 AM

All of our jobs are in the Scheduled State

Evan

04/14/2022, 7:15 AM

we are also experiencing this, jobs have not been running for about an hour

Stephen Lloyd

04/14/2022, 7:15 AM

Oh, thank goodness. I was going crazy. I’m having the issue as well.

☝️ 1

Jacob Blanco

04/14/2022, 7:16 AM

Fair to say this looks like a general outage.

Jars

04/14/2022, 7:22 AM

Any approximate ETA @Chris White?

Gaurav kumar

04/14/2022, 7:31 AM

How can we pause a task until a specified time and how to resume the same task before the timeout with some external triggers @Chris White

Jars

04/14/2022, 7:36 AM

Just saw my flows start

😓 1

Chris White

04/14/2022, 7:39 AM

Yea it seems our fix has taken some time to have an effect but the good news is that I am seeing late runs begin to get submitted and are now successfully running; we are still investigating though

✅ 1

Jacob Blanco

04/14/2022, 7:39 AM

Same here

👍 1

Stephen Lloyd

04/14/2022, 7:48 AM

I’m getting this error when using the FivetranSyncTask

Traceback (most recent call last):
  File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "workable/src/flow.py", line 64, in fivetran_sync
    status = FivetranSyncTask.run(
TypeError: method() missing 1 required positional argument: 'self'

creds is passed in from a PrefectSecret task.

@task
def fivetran_sync(connector: str, creds: dict) -> dict:
    status = FivetranSyncTask.run(
        api_key=creds['api_key'],
        api_secret=creds['api_secret'],
        connector_id=connector
        )
    return status

Chris White

04/14/2022, 7:56 AM

Hello everyone! The Cloud API issue should now be resolved, and we took a few extra steps that should prevent it from recurring; I don't have a full post mortem yet but in short this appears to have been caused by an incredibly large burst of logs entering the system that caused cascading effects. We'll be sure to take our learnings here to prevent this from occurring again and my sincere apologies for your disruption in service!! I hope everyone continues to have a good morning, afternoon or evening depending on where you are

🎉 2

:thank-you: 11

John Muddle

04/14/2022, 12:48 PM

Hi, I just wanted to highlight this issue https://github.com/PrefectHQ/server/issues/357, we've also seen this, so we're sticking to 2022.01.25 for now. Any ideas when this might be addressed?

Geert-Jan Van den Bogaerde

04/14/2022, 3:13 PM

Hi all, anyone know if there is a way to retrieve the run config parameters, specifically the labels, from within a flow programmatically at flow runtime? Haven't been able to find it in the docs.

Mike

04/14/2022, 4:13 PM

Hello all. Does anyone have any experience with or resources for using Prefect with GCP serverless? (cloud functions/cloud run) We are currently running a lot of our workflows using GCP serverless (via cloud functions or cloud run); and we have been using pub/sub queues as an orchestration tool to communicate between tasks. The pub/sub 10 minute message acknowledgement limit is starting to become a problem as some our tasks are taking longer than that to complete, so we wish to look into alternative tools.

Alexander Gorokhov

04/14/2022, 4:24 PM

Hello Prefect team, How a prefect cloud enterprise customer can get a technical support with a bug with prefect cloud itself?

Max Kolasinski

04/14/2022, 4:36 PM

Hi, I saw that there were some issues earlier today with Prefect Cloud. We’re currently still seeing a lot of our ECSRun tasks fail to begin execution correctly- are we potentially seeing delayed after effects from the earlier service disruption?

Jason

04/14/2022, 4:39 PM

Howdy - I've attempting to push a hello world job to ECR using the Docker storage for use with ECS. It looks like it fails on the last part, after passing health checks, where it's attempting to push to the registry. It's triggering this exception:

<https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/docker.py#L610-L613>

Has anyone run into a similar exception? I'm successfully authed to ECR from my local docker --login as well.

Jason

04/14/2022, 5:22 PM

I have an ECSRun for a hello world that appears to be hanging. I've successfully pushed the image to ECR via the Docker() storage, and the task shows up as scheduled in Prefect Cloud - which can also see the ECS Agent I added following the guide (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production). I'm a bit puzzled on how to proceed to debug why the agent isn't popping the task from the queue in the cloud.

Alexander Belikov

04/14/2022, 5:39 PM

Hi I have a question regarding scheduling prefect flows with parameters, I think I'm missing something simple : how do you pass db creds, if not through parameters (without hard-coding passwords, paths, etc)? I've posted my questions on SO: https://stackoverflow.com/questions/71875658/running-registering-prefect-flows-with-parameters-on-schedule, please let me know what you think. thanks!

Title

Alexander Belikov

04/14/2022, 5:39 PM

here in this example : https://stackoverflow.com/questions/70328855/how-to-organize-prefect-flow-with-the-using-of-cmd-arguments @Anna Geller is suggesting a parameterized

run

but then to register it without parameters... This example essentially emulates my case

Anna Geller

04/14/2022, 5:47 PM

are you on Prefect Cloud or Server? generally speaking, a parameter task is not a good way of passing Secrets such as DB credentials - if you are on Prefect Cloud, you may leverage the Secret backend to store these securely. If you are on Server, you could leverage some external Secrets Manager and retrieve those from within your task when needed

Kevin Kho

04/14/2022, 5:47 PM

Couple of things: 1. DB creds should ideally be secrets instead of parameters. On Prefect Cloud, you can store the secret with us and load it using the

PrefectSecret

task. On Server, you need to store local secrets. You can also see more on that here 2. On the parameters, you can also do

required=False

so that you can register without a default 3. Parameters defaults on clocks/schedule don’t fulfill the requirement of having a default parameter because you can still trigger ad-hoc runs that would then not have any filled value.

👍 2

Also, no need to crosspost btw between StackOverflow and here. Although, it’s actually good for us so we don’t mind. But we get notified as well with StackOverflow if you use the

prefect

tag

:upvote: 1

Alexander Belikov

04/15/2022, 10:19 AM

sounds good - I'll keep posting on StackOverflow since it might help wider audience. My takehomes are : 1) use secrets for creds, 2) hard-code other flow parameters using clocks (as in https://docs.prefect.io/core/concepts/schedules.html#clocks) 3) Not sure about

required=False

- I want to be able to register and schedule flows with parameters that are required

Anna Geller

04/15/2022, 10:25 AM

sounds good - I'll keep posting on StackOverflow since it might help wider audience.

Actually, could you post to Discourse instead? This is our preferred platform

✅ 1

#2 hard-code sounds so bad 😄 those are really just default parameter values. You can always override those from the UI or for a specific run via UI/API/CLI #3 required is useful if you always want to have some values - this post dives much deeper into the topic if you're interested

Alexander Belikov

04/15/2022, 1:14 PM

yes, hard-coding is bad practice 🙂 thank you for sharing this post I haven't seen it before

Kevin Kho

04/15/2022, 1:39 PM

I don’t think the flow parameter on the clock lets you register a Flow that has a required parameter with no default. Because there is still a scenario where you can trigger an ad-hoc run outside a schedule and the parameter isn’t supplied.

:upvote: 1

View count: 5