https://prefect.io logo
Join Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • t

    Tushar Gupta

    11/18/2022, 5:06 PM
    Please help https://prefect-community.slack.com/archives/CL09KU1K7/p1668775837419239
    👀 1
    b
    c
    r
    • 4
    • 16
  • m

    Miremad Aghili

    11/18/2022, 5:27 PM
    Hi, our prefect service fails to change status of the flow to failed when container dies or the whole agent dies. (it is prefect 1)
  • j

    Joshua Grant

    11/18/2022, 5:30 PM
    As suggested by Anna, I've looked into her 
    dataflow-ops
    repo and have tried to use the 
    ECSTask
    from 
    prefect-aws
    to circumvent the issue surrounding 
    DockerRegistry
    for ECR, but I keep getting the error 
    KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'"
    . Prefect==2.6.8 & prefect-aws==0.1.8
    ✅ 1
    r
    • 2
    • 7
  • t

    Tibs

    11/18/2022, 6:05 PM
    Hi everyone, is there an example of how to manually set the state of a flow or task to failed in prefect 2?
    ✅ 1
    m
    • 2
    • 6
  • v

    Vipul

    11/18/2022, 8:23 PM
    Hi, is there a way to RETRY the task in Prefect Orion like in the Prefect 1? Thanks
    ✅ 1
    a
    • 2
    • 6
  • s

    Scott Walsh

    11/18/2022, 9:50 PM
    I have a sub_flow that needs to be called by several parent flows. I can do this programmatically and treat the flow as a module and import into each parent flow, but, is there a way I can leverage prefect to do this? For example, can I 'register' a sub flow in prefect and then call it from a parent flow, where the subflow would run within the parent flows infrastructure (not it's own k8s job)? I see that I can use a deployment to deploy the sub flow with the 
    process
    infrastructure setting, but then the sub flow is run by an agent, not the parent flows infrastructure.
    ✅ 1
    a
    • 2
    • 1
  • m

    Michael Z

    11/18/2022, 10:30 PM
    Hi can someone help me understand what the arg on triggers are used for i.e. upstream_states. For example, 
    *prefect.triggers.all_finished*(_upstream_states_)
    . This is for prefect 1 btw.
    ✅ 2
    n
    • 2
    • 1
  • k

    Kimera Joseph

    11/19/2022, 5:58 AM
    Dear All, I am relatively new to prefect. I am using prefect 2.0. I have followed the official Docs on how to create and run deployments. I am using AWS S3 as my block storage. I successfully upload the files to my block in S3 bucket and the first run is always successful. However, when i try to run the deployment from prefect UI, I always get this error "Flow run <run-name> - Flow could not be retrieved from deployment" AND "TypeError: object NoneType can't be used in 'await' expression". I have tried Docker infrastructure and the default LocalProcess. I have changed the IAM role on the AWS EC2 instance am using but does not seem to work. I NEED HELP. THANK YOU
    ✅ 1
    a
    • 2
    • 4
  • k

    Kimera Joseph

    11/19/2022, 6:27 AM
    This is the example am running
    prefect-ecample.py
    ✅ 1
  • k

    komal azram

    11/19/2022, 6:57 AM
    I have just started using prefect 2 and nothing seems working for me. I can't start prefect server using prefect orion start. it says background error at: https://sqlalche.me/e/14/e3q8. When I ran the code I get this sqlite3.OperationalError: table flow already exists. I have tried resetting database but it gives this error sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table flow already exists [SQL: CREATE TABLE flow ( id CHAR(36) DEFAULT ( ( lower(hex(randomblob(4))) || '-' || lower(hex(randomblob(2))) || '-4' || substr(lower(hex(randomblob(2))),2) || '-' || substr('89ab',abs(random()) % 4 + 1, 1) || substr(lower(hex(randomblob(2))),2) || '-' || lower(hex(randomblob(6))) ) ) NOT NULL, created DATETIME DEFAULT (strftime('%Y-%m-%d %H:%M:%f000', 'now')) NOT NULL, updated DATETIME DEFAULT (strftime('%Y-%m-%d %H:%M:%f000', 'now')) NOT NULL, name VARCHAR NOT NULL, tags JSON DEFAULT '[]' NOT NULL, CONSTRAINT pk_flow PRIMARY KEY (id), CONSTRAINT uq_flow__name UNIQUE (name) ) ] (Background on this error at: https://sqlalche.me/e/14/e3q8) An exception occurred. can someone help me with this? I am using ubutu 20.04
    ✅ 1
    a
    • 2
    • 5
  • r

    Rob Fowler

    11/19/2022, 10:04 AM
    I used prefect1 for a year or so, now come back to it all a year later trying to use 2. It is not a very good experience when 99% of all we searches seem to go to v1 info.
    ✅ 1
    a
    • 2
    • 3
  • j

    Jimmy Le

    11/19/2022, 5:47 PM
    Anyone notice deployment build time taking a lot longer than normal?
    ✅ 1
    a
    • 2
    • 3
  • s

    suresh

    11/20/2022, 7:31 AM
    Hi! What is the equivalent to project in Prefect 2.0?
    ✅ 1
    a
    • 2
    • 1
  • k

    komal azram

    11/20/2022, 9:44 AM
    RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight Using prefect cloud but I get this if I re-run my code immediately.
  • k

    komal azram

    11/20/2022, 10:51 AM
    How we schedule flows to run at specific time in prefect 2.0? previously I have used flow.schedule()
    a
    • 2
    • 1
  • d

    Dekel R

    11/20/2022, 12:38 PM
    Hey, I’m developing my first Prefect 2 flow after about a year with Prefect 1 (-: I saw that now to get Prefect’s logger I should run “get_run_logger()“. Now this works inside a flow or a task context - but what should I do when a task calls a regular python function that contains logs (logger.info for example)? For example - In prefect 1 I could write “logger = prefect.context.get(LOGGER_NAME)” at the top of the py file that contained both Prefect tasks and regular python functions (that were used from within tasks), and the logger was used by both. I tried using “get_run_logger()” at the top of the py file but got this error “There is no active flow or task run context.” Thanks @Itay vidal FYI
    ✅ 1
    a
    • 2
    • 7
  • d

    Dekel R

    11/20/2022, 1:54 PM
    Another question regarding slack notifications. I saw this guide https://discourse.prefect.io/t/sending-notifications-in-cloud-1-0-using-automations-cloud-2-0-slack-webhook-blocks-and-notifications/1315 What about sending slack notifications on a local run? In Prefect 1, I added the “slack_webhook_url” to my local “/.prefect/config.toml” file - is there any solution for Prefect 2? Thanks.
    ✅ 1
    t
    • 2
    • 18
  • h

    Howard

    11/21/2022, 2:49 AM
    Hello everyone, I want to modify some source codes, but I don't know how to view the effects after the modification. "prefect dev start" currently has problems or some bugs I think.
    ✅ 1
    p
    • 2
    • 7
  • m

    Mohit Singhal

    11/21/2022, 6:27 AM
    Hi Everyone, I am not getting how to set "name" parameter in run_deployment. Can anyone please help me here?
    h
    t
    • 3
    • 23
  • m

    Michael Holzapfel

    11/21/2022, 7:46 AM
    Hi Everyone, I'm trying out prefect2 with a self hosted orion-server and I noticed that an agent crash left over "running" flow-runs which in turn interfere or block execution for the respective workqueue if a concurrency limit is set. Is there a canonical way to deal with such a situation?
    m
    c
    • 3
    • 3
  • s

    Sachin Ranveer

    11/21/2022, 9:13 AM
    hi getting error while running >> prefect orion start failed to create process. can someone help I have created venv activated it installed prefect >> pip install prefect
    c
    • 2
    • 1
  • a

    An Ninh Vũ

    11/21/2022, 10:16 AM
    Flows being stuck in running stage Hello everyone! I'm running into some problems, can you help me? I'm using Prefect 1.0. I built and coded the flows to submit more than 50 flows simultaneously. Sometimes all of these 50 flows run until the success state. But sometimes about 2 out of 50 are stuck at running state (whether all tasks are finished, or there are tasks left). It kept being "*running*" forever until I CANCEL it. I don't know where to find and fix them, as they kept turning into zombies like that. Can I ask where the problem might be (code, my Prefect agent, conflicts,...?) so that I can check it and try to fix it? Thank you so muchhhhh
    m
    • 2
    • 5
  • f

    FuETL

    11/21/2022, 11:05 AM
    Hey guys there a reason for signals (
    FAIL
    , 
    PrefectStateSignal
    , 
    PrefectSignal
    ) inherit from 
    BaseException
    instead of 
    Exception
    ?
    a
    • 2
    • 9
  • p

    Patrick Alves

    11/21/2022, 11:56 AM
    Hi there, Does anyone use loguru on Prefect 2.0? As the v2.0 does not have way to set a Global logger I was trying to add loguru as an extra logger to prefect. I’ve tried: 
    prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="rootlogger,loguru,root,root logger"
    But it did not worked. Any tips?
    m
    • 2
    • 2
  • m

    Matthew Seligson

    11/21/2022, 2:53 PM
    In Prefect 1.0, how can we determine what user cancelled a flow run?
    c
    • 2
    • 1
  • l

    Lennert Van de Velde

    11/21/2022, 3:05 PM
    Hey, We are developing a flow where a lot of JSON records get queried (in a Task). To save on memory resources, I've configured Result Persistence on the Task, but I'm now noticing a build up of memory usage of a serializer object. When are results serialized and when is this memory released?
    k
    • 2
    • 3
  • j

    James Zhang

    11/21/2022, 3:24 PM
    hi guys, I have this error in Prefect Agent 
    ERROR   | prefect.agent - Failed to submit flow run 'xxxx' to infrastructure.
    how can I investigate the details?
    ✅ 1
    r
    • 2
    • 14
  • g

    Guillermo Galan

    11/21/2022, 4:02 PM
    Hi all! Thanks for a great tool and community. Summary: I'm trying to get a deployment up and running in Azure Container Instances but Prefect Cloud keeps stopping my container after a few minutes. Do you have any ideas of what could be happening? I'm running a flow using deployments in Azure Container Instances. I manage to successfully trigger the job and the container starts running and processing the tasks correctly. After a while, for no apparent reason, the agent receives the instruction from prefect (running on Cloud) to delete the container: 
    16:44:27.874 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Preparing to run command '/opt/prefect/entrypoint.sh python -m prefect.engine' in container 'a53fd966-c635-4605-a8ab-4a3f487a7463' (prefecthq/prefect:2.6.7-python3.9)...
16:44:27.875 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Waiting for container creation...
16:45:37.024 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Running command...
16:45:37.025 | INFO    | prefect.agent - Completed submission of flow run '6bb0fc44-197e-4bfd-846e-877f5dbbf0b8'
16:48:43.454 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Completed command run.
16:48:43.455 | INFO    | prefect.infrastructure.container-instance-job - AzureContainerInstanceJob 'sftp-to-blob-ingestion': Deleting container...
    In the meantime the Orion UI gets frozen in 'running' state and there are no log lines indicating any error or crash.
    s
    r
    • 3
    • 3
  • t

    Tushar Gupta

    11/21/2022, 6:21 PM
    Can some one from @Prefect community help?Please https://prefect-community.slack.com/archives/CL09KU1K7/p1668791166095479
    c
    • 2
    • 4
  • m

    Madison Schott

    11/21/2022, 7:17 PM
    Hi all, I started testing my upgrade to 2.0 and downloaded the new version. I need to deploy an update to my 1.0 version which is still in prod and now I am getting an error with my tasks. How do I revert to the Prefect 1.0 version?
    ✅ 1
    m
    • 2
    • 25
Title
m

Madison Schott

11/21/2022, 7:17 PM
Hi all, I started testing my upgrade to 2.0 and downloaded the new version. I need to deploy an update to my 1.0 version which is still in prod and now I am getting an error with my tasks. How do I revert to the Prefect 1.0 version?
✅ 1
m

Mason Menges

11/21/2022, 7:23 PM
Hey @Madison Schott I think the simplest way would probably be to just setup a separate virtual environment for prefect 1 and prefect 2, the two versions of prefect are completely separate including the Cloud offerings for both of them so it's generally best to keep the two environments as separate as possible.
m

Madison Schott

11/21/2022, 7:23 PM
Makes sense, I think the damage is already done at this point though. How would I revert back?
m

Mason Menges

11/21/2022, 7:26 PM
Do you mean in terms of your code or just the prefect package itself? if it's just the package then you should be able to unintall prefect and then install the prefect version you need for your prefect 1 flows
m

Madison Schott

11/21/2022, 7:33 PM
yeah the package itself
what is the correct version of prefect 1.0? Or the last version of 1.0 that existed before 2.0?
The issue is that I don't know what that version number would be
m

Mason Menges

11/21/2022, 7:34 PM
the latest version of 1.0 is version 1.4.0
m

Madison Schott

11/21/2022, 7:39 PM
Ok I did that now I am getting this error: 
Traceback (most recent call last):
  File "/Users/madisonschott/dbt_snowflake/data_pipeline_test_new_account.py", line 387, in <module>
    data_pipeline_test_new_account.register(project_name="Winc_Prod")
  File "/usr/local/lib/python3.9/site-packages/prefect/core/flow.py", line 1726, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 855, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.9/site-packages/prefect/core/flow.py", line 1515, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.9/site-packages/prefect/storage/docker.py", line 325, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.9/site-packages/prefect/storage/docker.py", line 357, in _build_image
    client = self._get_client()
  File "/usr/local/lib/python3.9/site-packages/prefect/storage/docker.py", line 572, in _get_client
    return docker.APIClient(
  File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
m

Mason Menges

11/21/2022, 7:42 PM
if you run this command in the CLI does it show your cloud 1 tenants? 
prefect auth list-tenants
m

Madison Schott

11/21/2022, 7:43 PM
yup
I think I got it- thanks!
m

Mason Menges

11/21/2022, 7:48 PM
Brilliant, what ended up being the issue if you don't mind my asking?
m

Madison Schott

11/21/2022, 7:55 PM
Actually, false alarm...
Step 13/13 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/data-pipeline-test-new-account.prefect"]' '(3, 9)'
 ---> Running in 1fe1ff1c4327
/opt/prefect/healthcheck.py:125: UserWarning: Your Docker container is using python version (3, 7), but your Flow was serialized using (3, 9); this could lead to unexpected errors in deployment.
  system_check(python_version)

Beginning health checks...
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 130, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 43, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
AttributeError: Can't get attribute '_make_function' on <module 'cloudpickle.cloudpickle' from '/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py'>
I don't remember every changing the Python version, but maybe that happened when I downloaded 2.0?
m

Mason Menges

11/21/2022, 7:57 PM
Maybe but that would be odd since this should be based on the python version you have pointed for the image in your Dockerfile
m

Madison Schott

11/21/2022, 8:01 PM
Is it because of this 
FROM prefecthq/prefect:latest
in my base image?
m

Mason Menges

11/21/2022, 8:03 PM
That'd do it, you'll need to use the base image that corresponds to your python and perfect version for the flow
m

Madison Schott

11/21/2022, 8:03 PM
Can you send me the link to where to find these again?
m

Mason Menges

11/21/2022, 8:05 PM
Yep I think you'll want this one https://hub.docker.com/layers/prefecthq/prefect/1.4.0-python3.9/images/sha256-adf068ab9e87bb543f8e67c2aa069eb89d55190a9dd92e46f9c14870f32d4ccf?context=explore
m

Madison Schott

11/21/2022, 8:08 PM
That did the job, thanks!
m

Mason Menges

11/21/2022, 8:08 PM
No problem 😄
m

Madison Schott

11/21/2022, 10:21 PM
Now I'm running the new flow and getting this error when running 
dbt deps
logs.csv
Everything runs fine locally so I'm not sure why this is happening
everything works fine locally so it must have something to do with the Prefect image I am using
View count: 3