https://prefect.io logo
Join Slack
Powered by
Hey folks! Has there been given any thought on how...
# ask-community
m
Hey folks! Has there been given any thought on how to add conda support for the optional dependencies? Perhaps I can help?
j
Hey @Michael C. Grant! No plans that I am aware of and we definitely welcome support!
cc @Chris White and @Jim Crist-Harif since they have more experience in this area
m
Yes! Thanks. I have good reason to believe they would have good ideas 😉 and they know we can help.
j
You mean like breaking apart the prefect package on conda forge into say a 
prefect
and 
prefect-core
package?
m
Perhaps, but I'm really talking about building conda equivalents of the various pip options https://docs.prefect.io/core/getting_started/installation.html#optional-dependencies I'm actually currently interested in the new Git option
So, for example, 
conda install prefect-redis prefect-viz
would get you prefect plus those two options.
j
Prefect has a lot of different extras, I'm not sure if we want to make a 
prefect-*
package for each of them. An alternative option might be to document all the optional dependencies. For example, Dask currently has 
dask-core
and 
dask
, any of the other extensions require specific dependencies which we document in our docs. Could go either way here.
m
Well, if I were doing the work—and I might be volunteering—I'd bite the bullet and try and make a separate package for each of the pip options. I know it may seem like a lot but once it's done it's going to be relatively straightforward to maintain, especially if we go with multiple outputs in a single recipe file
But the documentation option gets conda users up and running very quickly so that would be great!
j
Multiple outputs would definitely be the way to do it. I'm mainly concerned about things from a user-perspective. If we have 25 
prefect-*
packages, it may be less clear what the user should install. But I suppose that's true with the documentation-only approach as well.
m
Just to make sure I'm not missing anything, why would it be any less clear than the pip options? In fact, I could go so far as to say the conda instructions would could be something as simple as this: "to utilize one of the optional dependency sets, just install 
prefect-<blah>
, where 
<blah>
is the corresponding pip option. Multiple options can be combined together in a single environment."
j
That's fair. I was thinking from a naive user approach, where they might see what conda packages exist before looking at our docs, and get a bit overwhelmed. But the benefits of using multiple packages probably outweigh the cons here.
m
I'll do a little experimenting
👍 1
This seems surprisingly simple:
Copy code
outputs:
  - name: prefect
  - name: prefect-airtable
    requirements:
      run:
        - prefect =={{ version }}
        - airtable-python-wrapper >=0.11,<0.12
  - name: prefect-aws
    requirements:
      run:
        - prefect =={{ version }}
        - boto3 >=1.9,<2.0
and so on
Not that you should take inspiration from airflow but it looks like the package convention they are using is 
airflow-with-<blah>
Copy code
prefect-0.12.2+34.gb96bd0285-py_0.tar.bz2
prefect-airtable-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-aws-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-azure-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-dask-cloudprovider-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-dropbox-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-gcp-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-ge-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-github-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-google-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-gsheets-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-jira-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-kubernetes-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-mysql-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-postgres-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-pushbullet-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-redis-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-rss-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-snowflake-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-spacy-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-templates-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-test-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-twitter-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-viz-0.12.2+34.gb96bd0285-0.tar.bz2
prefect-dev-0.12.2+34.gb96bd0285-0.tar.bz2
👀 1
j
Thanks! I'll take a look tomorrow morning!
3 Views
Open in Slack
PreviousNext
Powered by