We trigger flows from lambda, and all of our lambda’s started failing. Did something change with prefect’s API?

Hi All! We are experiencing some issues as our Cloud API is rejecting some traffic. We should be fully recovered shortly and I will provide additional details

In an effort to add additional security measures to our API, we experienced a brief outage this afternoon. This issue has been resolved, and we thank you for your patience

Feel free to reach out to the channel, or to me directly, with any questions about this incident or any impact you might have had from it

Question for folks doing CI/CD automation to build & register Prefect Flows. Building Docker storage for one of our flows from our CI/CD system (Screwdriver.io) fails with the exception below. Docker appears to be installed and working properly because

docker run hello-world

succeeds. (There is special "docker-in-docker" config we use with Screwdriver and this works well for many of our other projects.) I can also build Docker storage successfully on my laptop from inside the same image that our CI/CD system is using. I'm sure it's some simple config that I'm missing -- anybody run into this?

15:25:50 Traceback (most recent call last):
15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
15:25:50     "__main__", mod_spec)
15:25:50   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
15:25:50     exec(code, run_globals)
15:25:50   File "/usr/local/lib/python3.7/site-packages/srm_datasci-0.1.0-py3.7.egg/srm/datasci/flows/util/test_flow.py", line 35, in <module>
15:25:50     flow.register("Utilities")
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1412, in register
15:25:50     no_url=no_url,
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 616, in register
15:25:50     serialized_flow = flow.serialize(build=build)  # type: Any
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1221, in serialize
15:25:50     storage = self.storage.build()  # type: Optional[Storage]
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 263, in build
15:25:50     self._build_image(push=push)
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 293, in _build_image
15:25:50     self.pull_image()
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 483, in pull_image
15:25:50     client = self._get_client()
15:25:50   File "/usr/local/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 471, in _get_client
15:25:50     return docker.APIClient(base_url=self.base_url, version="auto")
15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 190, in __init__
15:25:50     self._version = self._retrieve_server_version()
15:25:50   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
15:25:50     'Error while fetching server API version: {0}'.format(e)
15:25:50 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Hi folks!, I'm a disgruntled airflow user here looking to learn more about prefect. I've started to port over a simple ETL workflow with some success.

Do I have to use prefect cloud to get access of the UI dashboard cause my tasks run but I can't see the UI dashboard. I haven't logged into prefect cloud.

(myenv) amod@amod-ThinkPad-T430:~/prefect$ prefect run cloud --name hello-flow --project 'Hello, World!' hello-flow not found

I have followed all the steps from this https://docs.prefect.io/cloud/tutorial/first.html#write-flow yet unable to work.

Hey - is it possible to pass a function to return a default value for a Parameter

I’m getting an error of “Cannot serialize socket object”. The error says “Can’t pickle function <myFunction>: it’s not the same object as Helper_Functions.myFunction”. Any suggestions on how to fix this? I’m mapping over a task with 3 unmapped parameters which as the results of other tasks.

hey everyone - I’m experiencing a problem where my workflow in a remote dask cluster breaks down after a mapping task. The workflow is terminated on the remote cluster but not in the cloud. any thoughts?

I'm attempting to schedule a Flow, but receiving this error message:

prefect.utilities.exceptions.ClientError: Flows with required parameters can not be scheduled automatically.

I understand what the error message is saying, but in the

parameter_defaults

for the

InvervalClock

of the Flow, I have the required Parameters defined. It's easy to remove the required flag for these Parameters, but for code readability I'd like to keep them in there. Why is it considered an error, when all of the

IntervalClock

defined in the

Schedule

of the Flow have a value for the required Parameters defined?

Hi All, we are rolling out a new security platform that appears to be causing some service interruptions. Please bear with us as we try to resolve this issue

Hi Team, how do I troubleshoot this error message:

March 17th 2020 at 5:13:44pm | prefect.CloudTaskRunner
INFO 
Task 'sfdc_download_data': Starting task run...March 17th 2020 at 5:13:44pm | prefect.CloudTaskRunner
ERROR 
Unexpected error: KeyError('text')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 225, in check_task_is_cached
    for key, res in (candidate_state.cached_inputs or {}).items()
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 225, in <dictcomp>
    for key, res in (candidate_state.cached_inputs or {}).items()
KeyError: 'text'

Hi folks, The issues @Tyler Wanner mentioned have been resolved. Impact appears to have been minimal, but as always we're happy to answer any questions you may have.

from prefect import Flow, task

@task
def add_ten(x):
    return x + 10

with Flow('simple map') as flow:
    mapped_result = add_ten.map([1, 2, 3])

I am using this code expected output is 11,12,13 in mapped_result how do I print the result?

is there any open source alternative to prefect cloud?

We are getting some workflow runs failing immediately on Prefect Cloud with just one log message:

March 17th 2020 at 2:32:51am | agent
ERROR 
An error occurred (ThrottlingException) when calling the RunTask operation (reached max retries: 4): Rate exceeded.

Failed to load and execute Flow's environment: TypeError('an integer is required (got type bytes)')

Hello Everyone I’ve had a mild issue about

ShellTask

, in which a

git clone

command doesn’t return some of the stdout lines (despite using

return_all=True

). The command is as follows:

git clone https://{some_url}.git dbt_project -v -b master --depth 1

. Running it from bash logs the following to stdout:

Cloning into 'dbt_project'...
POST git-upload-pack (156 bytes)
POST git-upload-pack (165 bytes)
remote: Counting objects: 40, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 40 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (40/40), done.

While running the same command through

ShellTask

, and iterating over the resulting list of stdout lines logs the following:

[2020-03-19 09:57:50,792] INFO - prefect.Task: print_task | Cloning into 'dbt_project'...
[2020-03-19 09:57:50,792] INFO - prefect.Task: print_task | POST git-upload-pack (168 bytes)
[2020-03-19 09:57:50,793] INFO - prefect.Task: print_task | POST git-upload-pack (177 bytes)

By any means, I am not well versed on bash, and this may very well be an issue about how

git clone

logs to stdout. Wondering if anybody else encountered a similar issue, and if they managed to come up with a fix?

Does prefect or the prefect community use a common solution for job tracking? The use case is: 1. get S3 file listing 2. use <some job tracking solution> to ignore files that have been previously processed successfully, process new files, and retry failed files with < 3 previous attempts 3. kick off subsequent tasks

is there a migration guide or anything for switching to the new result handling implied by https://docs.prefect.io/core/PINs/PIN-16-Results-and-Targets.html and https://github.com/PrefectHQ/prefect/pull/2151/files ? I think I understand the concept but I'm not sure exactly what I'm supposed to do about it 🙂 or maybe the answer is "nothing yet" but then the deprecation warning feels a bit premature...?

Hi all! We are using DBT standalone (cli) right now. I’ve been doing some research on an orchestration tool that can handle retries, etc. At first I thought of Jenkins but then airflow was recommended. After even more reading, Prefect seems to one more recommended choice if you are green fielding. I’m having trouble finding further documentation to answer my questions: Does Prefect replace Airflow? Would it replace parts of DBT as well? Thank you!

Hey Prefect community, I'm trying to understand how to integrate CICD in the flow development process, in part to prevent accidents when pushing flows from a developer's local machine. My first reflex was to create a private git repo with flows' code, and automatically register all flows inside with Cloud only after a PR was merged to `master`/a tagged release was made. This approach destroys flow versioning however, so then I need to parse git history to find only files that got added/modified recently (via Github Actions, or a step in CircleCI, for example)... Not very elegant. Did anyone else run into a similar issue? How did you go about it? Thanks!!

can prefect core alone be used in production systems?

Thank you for the Cloud Hooks feature! We will start using the slack and email hooks soon. I imagine a day when you have tons of integrations tied into this for easy integrations. Keep up the good work!

Now that we have the concept of multiple schedules per flow...has there been any thought to concept of environments for CICD? Like deploy a flow then promote that version to dev -> uat -> prod? We kind of do that now but with different projects and actually registering the flow in different projects. Down side is that it’s 2 flows against your allotment. Would just have to think about ‘config’ per environment. Dbt Cloud does this. Also kind of inline with k8s namespaces and perhaps AWS elastic beanstalk