I'm in need of an easy way to login to google account in prefect, but I didn't want to use Selenium. Can I use Playwright in prefect?

I’m trying to set up CI on a flow repo using the template here: https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main_simple.yaml but am running into a 404 issue with the API similar to the one reported here: https://prefect-community.slack.com/archives/CL09KU1K7/p1661349987899419 I’ve double-checked the API URL being used in CI against the one on my local CLI and have verified that the API Key being used is active as well. Is there a better way to debug why I can’t register flows to Prefect Cloud?

Hi Everyone, I'm having an issue with deployment storage. This deployment is using the

process

infrastructure and the

s3

block for filesystem storage. I build my deployment, the yaml is generated, and my flow code is uploaded to s3. I have prefect orion running locally, and this deployment applied to my local prefect. I then start a local agent within my python virtual environment. In orion, I click

Run

on my deployment. The flow run is moved to my local work queue, my local agent picks up the flow run, and builds the infrastructure. The agent then runs the prefect engine, which begins pulling my flow code from s3. In doing so, I receive this flow run failure log:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\prefect\engine.py", line 254, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\prefect\client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\prefect\deployments.py", line 55, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\prefect\filesystems.py", line 399, in get_directory
    return await self.filesystem.get_directory(
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\prefect\filesystems.py", line 260, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\fsspec\spec.py", line 801, in get
    self.get_file(rpath, lpath, **kwargs)
  File "C:\Users\BStefansen\.virtualenvs\lbx-broadband-pipeline-4k_4F-H8\lib\site-packages\fsspec\spec.py", line 769, in get_file
    outfile = open(lpath, "wb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/BSTEFA~1/AppData/Local/Temp/tmp5t7vj96aprefect/aurora\\aurora_constants.py'

Expectation I would expect the prefect engine to use s3 file paths for pulling down the flow code, but it seems to be using local file paths. Is this an issue with my deployment/flow setup, or is this a bug? Environment • Windows 10 • Python 3.10.5 • Prefect 2.2.0

@Anna Geller The team at my company saw this issue you opened with

s3fs

for

DockerContainer

infrastructure blocks, but this is also an issue for me with the

process

infrastructure when using prefect cloud. If I run a deployment in the cloud using

s3

for filesystem storage and

process

for infrastructure, our flow runs error out saying

ModuleNotFoundError: No module named 's3fs'

I'm not sure if this is due to our agent being a

kubernetes

agent and our deployment infrastructure being

process

, or something else. If the

process

just runs a subprocess on a docker image made by the kubernetes agent, then I would assume it's an issue with the default image used by the kubernetes agent

Can I set add a key to

context

in a task such that the key would be available to other downstream tasks within that run?

Hi, I would like to have an input from you guys. I am looing at prefect2.0. I was expected that in a deployment I could specify an environment (docker image) per subflow as my subflows can have a lot of dependencies that are not always compatible between each other. How I envision things is to have 1 image per subflow. As the main flow expands, it seems that allowing such deployments would make everything more compartmented and maintainable. Have you encountered such request before? Is there a way to do that in prefect 2.0?

I am not a pip expert, how do you install optional packages locally when doing development?

pip install -e ".[dev]"

does not install the packages

Hi, after I executed a flow on agent, I did nothing, then the agent will pop up those exception......it doesn't seem to impact function, but it's pretty annoying. Do you have a clue how can i get rid of these? (exceptions will be pasted in thread)

If I'm getting this:

UserWarning: Block document has schema checksum sha256:28a9d16c2589998d5281b0496b25edd76c1a0b159095d0c7fe5eb5dc5ab62549 which does not match the schema checksum for class 'GCS'. This indicates the schema has changed and this block may not load.

Should I delete and recreate my GCS block?

Missing something small, whats the syntax to pass multiple work queues to an agent

we current use ECS for running agents: if I need to issue some kind of IAM permission to the actual flow run, should that be placed in the agent policy or the task policy?

Hello, Prefect 2.0 question: I'm trying to setup a kubernetes agent and I've largely been following the demo found here: (thanks Chris!). I think I was able to figure out the infra-block issue that Chris encountered by adding a KubernetesJob block however, I'm now getting an error message when running my flow:

RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.

I see some mentions of adding a requirements.txt file and including adlfs in that file but that doesn't seem to get picked up using prefect deployment apply ... I also see some mentions around modifying the deployment.yaml file to include EXTRA_PIP_PACKAGES but that equally doesn't seem to do anything. How do I get my agent to install adlfs?

I had a question on

blocks

in prefect 2.0. Can

blocks

scoped to workspaces or could we constrain blocks only to certain deployments? Why you might ask? I want to ensure that configurations / credentials are not (mis)used intentionally or otherwise

Hi, I am occasionally experiencing ‘500 Internal Server Errors’ from the cloud API at (https://api.prefect.cloud/api/accounts/….) while executing a deployed flow in an agent. The error occurs at random points of execution of the flow and leads to a total stop of the flow. I unfortunately have no clue why this happens. I am currently using Prefect 2.1.1 Traceback is in the thread. I would be glad if you could help me.

Hi Everyone, we recently switched to Prefect 2.0 and I was wonder if there was a way to reference the Task object from inside the function, something like this:

@task(name="DefaultPrefectTask")
def to_print(word):
    print(task.name)
    print(task.tags)
    print(f"Direct printing from creating task with decorator {word}")
    return word

Hello Everyone, I'm seeing a weird behavior in Prefect 2.0. So I have a CI/CD runner that takes flow codes and deploy them to Prefect Cloud + S3, so that AWS agents can pick it up and run it as a job on a K8S cluster. The thing is, the runner can't apply a deployment if it doesn't have all the python dependencies of the code already installed, which is weird, considering the fact that it's only job is to create a deployment.yaml file and send it to prefect cloud + S3 and not run the flows. I find this behavior really disturbing and unpractical since people usually have one CI/CD runner and many flows (with different python versions and different package dependencies).

Is it possible to tell Prefect 2.0 to not persist task results to disk?

Hello I am trying to integrate prefect cloud with Airbyte. But I am facing below error

ModuleNotFoundError: No module named 'prefect.tasks.airbyte'; 'prefect.tasks' is not a package

Have I missed to install any dependency ? I didnt see anything on airbyte/prefect pages

How to schedule flow in prefect 2.0 , 1.0 was so easy to use. 😛 ?

Hello! I noticed that we are able to set tags on task's decorators and when defining a deployment but not on flow decorators. There is always the

with tags("test"):

option but still this applies to tasks inside a flow. Is this a design choice? There are some use cases where setting tags for flows could be proved useful for me

What if I have an existing code base (for machine learning) with 4 separate steps; 1. Data preprocess 2. Train 3. Inference 4. Visualizations And I want to introduce Prefect to it but still allow others without prefect installed to run the code?

Is there a best practice for mocking the task decorators (and other decorators)? Should I do something like:

try:
    import prefect
except ImportError
    def @task(func): return func

?

Hi there, I started a Orion Server on a on-prem Linux VM (prefect orion start) and if I access it via browser it shows me my local flow runs. I expected the flow runs which are startet for this particular instance. How can I access the flow runs of the remote instance?

hi there! prefect 2 roadmap question: is there some plan for a

DaskTaskRunner

, but for subflows? E.g. if i have a flow that kicks off a bunch of subflows, i want those to be distributed out and not running on the same instance as the parent flow (or...is that already possible and i'm missing something)? Thanks in advance!

👋 just upgraded to prefect 2.2.0, and seeing some odd behavior with parameterized flows. I have a flow with a parameter

name

, and it looks like prefect is passing the parameters spec as the parameter, so

{
  "type": "object",
  "title": "Parameters",
  "properties": {
    "name": {
      "title": "name",
      "default": "world"
    }
  }
}

is getting passed to my flow, then my flow fails with

prefect.exceptions.SignatureMismatchError: Function expects parameters ['name'] but was provided with parameters ['type', 'title', 'properties']

not really sure what im doing wrong here...

Prefect’s documentation says the following:

Create a flow as a context manager and call your tasks on each other as if they were regular functions. Prefect will track each function call and build up a computational graph that represents your workflow. Critically, no tasks are actually executed at this time.

How is that possible? does the instantiation of a context manager inhibits functions from being actually called?

Task 'sync_canvas': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 859, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 445, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "<string>", line 28, in sync_canvas
TypeError: 'NoneType' object is not iterable

We are using prefect on EC2 spot instances and we think this is likely caused by AWS dropping our spot instances. Is there a way around this while still using spot instances or is this unavoidable with spot instances?

In prefect ui it would be nice to have the option to go to the parent flow