https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Apoorva Desai

    04/14/2022, 12:20 AM
    Hello! I have a custom docker container for my prefect flows and I have copied over a python file with some custom functions onto this docker container. However, my prefect flow is unable to import these functions. I am getting a module not found error. How do you access functions from python files in custom docker containers in prefect flow?
    ✅ 1
    a
    k
    • 3
    • 35
  • a

    Alexander Butler

    04/14/2022, 1:57 AM
    I am working on standing up Prefect 2.0 is a production environment. For internal data pipeline and reverse etl uses so no fire hazards on my end to use 2.0 early here. Is there a general preference on YAML vs Code for the deployment specification. I noticed you can configure a flow deployment with YAML but I cant find any information on the schema of that document. For example:
    - name: elt-salesforce
      flow_location: ./salesforce_flows.py
      flow_name: elt-salesforce
      tags:
        - salesforce
        - core
      parameters:
        destination: "gcp"
      schedule:
        interval: 3600
    Assuming interval is seconds? Can I specify another grain? Can schedule take a dict? If it takes cron, does that take a dict? Honestly schedule is the primary question point. Everything else is straightforward enough.
    :discourse: 1
    k
    a
    +2
    • 5
    • 13
  • s

    Salim Doost

    04/14/2022, 2:53 AM
    We’re having issues with deployments on one of our Prefect workspaces all of a sudden (without changing anything on our setup to the best of our knowledge). Our flows are stored as Docker images in AWS ECR. Running a newly created flow leads to the following error:
    404 Client Error for <http+docker://localhost/v1.41/containers/create?name=quantum-squid>: Not Found ("No such image: <account-id>.<http://dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>%22|dkr.ecr.ap-northeast-1.amazonaws.com/datascience-prefect:<image-tag-name>">)
    However, we’re able to confirm that the image with this tag exists on EMR. Updating an existing flow by overriding an existing image-tag leads to the following error:
    KeyError: 'Task slug <task-name> is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.
    - Did you change the flow without re-registering it?
    - Did you register the flow without updating it in your storage location (if applicable)?'
    Again, we’re able to confirm in AWS ECR that the image got pushed and updated successfully. Our deployment job didn’t throw any error messages as well. Any idea what we can do to resolve this issue?
    k
    • 2
    • 10
  • c

    Carlos Cueto

    04/14/2022, 4:27 AM
    Hi. I'm having an issue using a
    LocalRun
    flow's
    working_dir
    parameter. Whenever I specify the following:
    flow.run_config = LocalRun(_working_dir_='C:/scripts/GetADUsers', _labels_=["SVRBIPTH01"])
    Whenever I register the flow (I'm using Prefect 1.2.0 on MacOS python 3.10) I get the following working_dir on the UI of Prefect Cloud:
    /Users/carloscueto/Documents/Python_Scripts/Prefect-Flows/PowerShell/GetADUsers/C:/scripts/GetADUsers
    It seems to be adding the path from where I register the script from (on the local machine) to the working_dir string I specified on the run_config. Has anybody encountered this before? Everything works fine when I register the flow from a Windows computer.
    ✅ 1
    k
    a
    s
    • 4
    • 13
  • a

    Alexander Butler

    04/14/2022, 6:13 AM
    One more question thats far more importantly than the YAML flow deployment config... If I want to bundle a self contained application as a docker image which required no ui interfacing to stand up Prefect 2.0, so
    prefect orion start
    prefect deployment create ...
    prefect work-queue create -t etl -- etl-queue
    HERE IS THE GAP -- the response to the above command is something like
    UUID('...')
    , which is useless when setting something up from the CLI without sed/awk?
    prefect agent start 'no simple headless way to derive id...'
    The less appealing part afterwards is that
    prefect work-queue ls
    renders a table which is pretty in a CLI but useless again to simply get an ID. Has anyone set up Prefect 2.0 to self deploy in an image along with all their code? The ephemeral nature makes this very advantageous with what seems to be a tiny unconsidered gap. I am pretty sure a more reliable consistent way to get work queue is all thats needed basically, but if I am totally missing it just lmk. I am a big fan of the package for the record but now its crunch time production use attempts 🙂
    a
    • 2
    • 3
  • j

    Jacob Blanco

    04/14/2022, 6:32 AM
    Anyone else seeing timeouts against the Prefect API? The Cloud view is not updating consistently and many of our Cloud-managed flows are not starting up.
    ✅ 3
  • s

    Stéphanie Cérulli

    04/14/2022, 6:43 AM
    Yes we have errors on API too
  • s

    Stéphanie Cérulli

    04/14/2022, 6:44 AM
    2022-04-14T06:42:11.384990+00:00 - -  urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
  • l

    Luuk

    04/14/2022, 6:45 AM
    Same here, also can't login into the cloud with a new API key
  • j

    Jacob Blanco

    04/14/2022, 6:49 AM
    The status page is still green: https://prefect.status.io
  • j

    Jacob Blanco

    04/14/2022, 7:02 AM
    @Chris White Is anyone looking at this??
    c
    • 2
    • 3
  • j

    Jars

    04/14/2022, 7:14 AM
    All of our jobs are in the Scheduled State
  • e

    Evan

    04/14/2022, 7:15 AM
    we are also experiencing this, jobs have not been running for about an hour
  • s

    Stephen Lloyd

    04/14/2022, 7:15 AM
    Oh, thank goodness. I was going crazy. I’m having the issue as well.
    ☝️ 1
  • j

    Jacob Blanco

    04/14/2022, 7:16 AM
    Fair to say this looks like a general outage.
  • j

    Jars

    04/14/2022, 7:22 AM
    Any approximate ETA @Chris White?
  • g

    Gaurav kumar

    04/14/2022, 7:31 AM
    How can we pause a task until a specified time and how to resume the same task before the timeout with some external triggers @Chris White
    j
    a
    • 3
    • 18
  • j

    Jars

    04/14/2022, 7:36 AM
    Just saw my flows start
    😓 1
  • c

    Chris White

    04/14/2022, 7:39 AM
    Yea it seems our fix has taken some time to have an effect but the good news is that I am seeing late runs begin to get submitted and are now successfully running; we are still investigating though
    ✅ 1
  • j

    Jacob Blanco

    04/14/2022, 7:39 AM
    Same here
    👍 1
  • s

    Stephen Lloyd

    04/14/2022, 7:48 AM
    I’m getting this error when using the FivetranSyncTask
    Traceback (most recent call last):
      File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "/Users/slloyd/projects/dwbi-orchestration/.venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "workable/src/flow.py", line 64, in fivetran_sync
        status = FivetranSyncTask.run(
    TypeError: method() missing 1 required positional argument: 'self'
    creds is passed in from a PrefectSecret task.
    @task
    def fivetran_sync(connector: str, creds: dict) -> dict:
        status = FivetranSyncTask.run(
            api_key=creds['api_key'],
            api_secret=creds['api_secret'],
            connector_id=connector
            )
        return status
    j
    • 2
    • 2
  • c

    Chris White

    04/14/2022, 7:56 AM
    Hello everyone! The Cloud API issue should now be resolved, and we took a few extra steps that should prevent it from recurring; I don't have a full post mortem yet but in short this appears to have been caused by an incredibly large burst of logs entering the system that caused cascading effects. We'll be sure to take our learnings here to prevent this from occurring again and my sincere apologies for your disruption in service!! I hope everyone continues to have a good morning, afternoon or evening depending on where you are
    🎉 2
    :thank-you: 11
    m
    • 2
    • 4
  • j

    John Muddle

    04/14/2022, 12:48 PM
    Hi, I just wanted to highlight this issue https://github.com/PrefectHQ/server/issues/357, we've also seen this, so we're sticking to 2022.01.25 for now. Any ideas when this might be addressed?
    k
    • 2
    • 2
  • g

    Geert-Jan Van den Bogaerde

    04/14/2022, 3:13 PM
    Hi all, anyone know if there is a way to retrieve the run config parameters, specifically the labels, from within a flow programmatically at flow runtime? Haven't been able to find it in the docs.
    k
    a
    • 3
    • 13
  • m

    Mike

    04/14/2022, 4:13 PM
    Hello all. Does anyone have any experience with or resources for using Prefect with GCP serverless? (cloud functions/cloud run) We are currently running a lot of our workflows using GCP serverless (via cloud functions or cloud run); and we have been using pub/sub queues as an orchestration tool to communicate between tasks. The pub/sub 10 minute message acknowledgement limit is starting to become a problem as some our tasks are taking longer than that to complete, so we wish to look into alternative tools.
    k
    • 2
    • 4
  • a

    Alexander Gorokhov

    04/14/2022, 4:24 PM
    Hello Prefect team, How a prefect cloud enterprise customer can get a technical support with a bug with prefect cloud itself?
    k
    • 2
    • 10
  • m

    Max Kolasinski

    04/14/2022, 4:36 PM
    Hi, I saw that there were some issues earlier today with Prefect Cloud. We’re currently still seeing a lot of our ECSRun tasks fail to begin execution correctly- are we potentially seeing delayed after effects from the earlier service disruption?
    a
    k
    • 3
    • 5
  • j

    Jason

    04/14/2022, 4:39 PM
    Howdy - I've attempting to push a hello world job to ECR using the Docker storage for use with ECS. It looks like it fails on the last part, after passing health checks, where it's attempting to push to the registry. It's triggering this exception:
    <https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/docker.py#L610-L613>
    Has anyone run into a similar exception? I'm successfully authed to ECR from my local docker --login as well.
    k
    • 2
    • 3
  • j

    Jason

    04/14/2022, 5:22 PM
    I have an ECSRun for a hello world that appears to be hanging. I've successfully pushed the image to ECR via the Docker() storage, and the task shows up as scheduled in Prefect Cloud - which can also see the ECS Agent I added following the guide (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production). I'm a bit puzzled on how to proceed to debug why the agent isn't popping the task from the queue in the cloud.
    k
    • 2
    • 7
  • a

    Alexander Belikov

    04/14/2022, 5:39 PM
    Hi I have a question regarding scheduling prefect flows with parameters, I think I'm missing something simple : how do you pass db creds, if not through parameters (without hard-coding passwords, paths, etc)? I've posted my questions on SO: https://stackoverflow.com/questions/71875658/running-registering-prefect-flows-with-parameters-on-schedule, please let me know what you think. thanks!
    a
    k
    • 3
    • 9
Powered by Linen
Title
a

Alexander Belikov

04/14/2022, 5:39 PM
Hi I have a question regarding scheduling prefect flows with parameters, I think I'm missing something simple : how do you pass db creds, if not through parameters (without hard-coding passwords, paths, etc)? I've posted my questions on SO: https://stackoverflow.com/questions/71875658/running-registering-prefect-flows-with-parameters-on-schedule, please let me know what you think. thanks!
here in this example : https://stackoverflow.com/questions/70328855/how-to-organize-prefect-flow-with-the-using-of-cmd-arguments @Anna Geller is suggesting a parameterized
run
but then to register it without parameters... This example essentially emulates my case
a

Anna Geller

04/14/2022, 5:47 PM
are you on Prefect Cloud or Server? generally speaking, a parameter task is not a good way of passing Secrets such as DB credentials - if you are on Prefect Cloud, you may leverage the Secret backend to store these securely. If you are on Server, you could leverage some external Secrets Manager and retrieve those from within your task when needed
k

Kevin Kho

04/14/2022, 5:47 PM
Couple of things: 1. DB creds should ideally be secrets instead of parameters. On Prefect Cloud, you can store the secret with us and load it using the
PrefectSecret
task. On Server, you need to store local secrets. You can also see more on that here 2. On the parameters, you can also do
required=False
so that you can register without a default 3. Parameters defaults on clocks/schedule don’t fulfill the requirement of having a default parameter because you can still trigger ad-hoc runs that would then not have any filled value.
👍 2
Also, no need to crosspost btw between StackOverflow and here. Although, it’s actually good for us so we don’t mind. But we get notified as well with StackOverflow if you use the
prefect
tag
:upvote: 1
a

Alexander Belikov

04/15/2022, 10:19 AM
sounds good - I'll keep posting on StackOverflow since it might help wider audience. My takehomes are : 1) use secrets for creds, 2) hard-code other flow parameters using clocks (as in https://docs.prefect.io/core/concepts/schedules.html#clocks) 3) Not sure about
required=False
- I want to be able to register and schedule flows with parameters that are required
a

Anna Geller

04/15/2022, 10:25 AM
sounds good - I'll keep posting on StackOverflow since it might help wider audience.
Actually, could you post to Discourse instead? This is our preferred platform
✅ 1
#2 hard-code sounds so bad 😄 those are really just default parameter values. You can always override those from the UI or for a specific run via UI/API/CLI #3 required is useful if you always want to have some values - this post dives much deeper into the topic if you're interested
a

Alexander Belikov

04/15/2022, 1:14 PM
yes, hard-coding is bad practice 🙂 thank you for sharing this post I haven't seen it before
k

Kevin Kho

04/15/2022, 1:39 PM
I don’t think the flow parameter on the clock lets you register a Flow that has a required parameter with no default. Because there is still a scenario where you can trigger an ad-hoc run outside a schedule and the parameter isn’t supplied.
:upvote: 1
View count: 5