Joël Luijmes

    Joël Luijmes

    1 year ago
    Hey, when pinning my helm deployed prefect to 0.14.0, I noticed the tags 0.14.0 and core-0.14.0. As I’m running it myself I probably need the core- version. However, when I used that version for the Kubernetes Agent, it fails to start (see thread for logs).
    [2020-12-24 17:08:11,786] DEBUG - agent | Verbose logs enabled                        
    [2020-12-24 17:08:11,787] DEBUG - agent | Environment variables: []                   
    [2020-12-24 17:08:11,787] DEBUG - agent | Max polls: None                             
    [2020-12-24 17:08:11,787] DEBUG - agent | Agent address: <http://0.0.0.0:8080>          
    [2020-12-24 17:08:11,787] DEBUG - agent | Log to Cloud: True                          
    [2020-12-24 17:08:11,788] DEBUG - agent | Prefect backend: server                     
    Traceback (most recent call last):         
      File "/usr/local/bin/prefect", line 8, in <module>                                                                                                                        
        sys.exit(cli())                                                                   
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__                                                                                        
        return self.main(*args, **kwargs)      
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main                                                                                            
        rv = self.invoke(ctx)                                                             
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke                                                                                         
        return _process_result(sub_ctx.command.invoke(sub_ctx))                           
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke                                                                                         
        return _process_result(sub_ctx.command.invoke(sub_ctx))                           
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke                                                                                         
        return _process_result(sub_ctx.command.invoke(sub_ctx))                           
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke                                                                                         
        return ctx.invoke(self.callback, **ctx.params)                                    
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke                                                                                          
        return callback(*args, **kwargs)                                                  
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 282, in start                                                                                    
        start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)                                                                                           
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/agent.py", line 108, in start_agent                                                                              
        agent = agent_cls(labels=labels, env_vars=env_vars, **kwargs)                     
      File "/usr/local/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 120, in __init__                                                                    
        from kubernetes import client, config                                             
    ModuleNotFoundError: No module named 'kubernetes'
    I noticed on Docker Hub, that for most images the latest tag aligns with core-0.14.0. But with prefecthq/prefect it does not, then it aligns with 0.14.0
    Chris White

    Chris White

    1 year ago
    Hi @Joël Luijmes, the
    core-0.14.0
    tag only installs the base Prefect package with no additional extras; the
    0.14.0
    tag does install “orchestration extras” which are the standard additional packages required for Cloud deployments (e.g., kubernetes, boto3, etc.) We typically recommend using the 0.14.0 tag for agents and the
    core-0.14.0
    tag for the base image of your flows because it’s slimmer. I’ll open an issue to update our documentation to clarify this distinction!
    @Marvin open “Expand Docker tag documentation to explain extra package installations”
    Marvin

    Marvin

    1 year ago
    Joël Luijmes

    Joël Luijmes

    1 year ago
    So if I pin my version, I should use version 0.14.0? And if I were to manually build the flow images use the core-0.14.0?
    Michael Adkins

    Michael Adkins

    1 year ago
    I think that sounds right @Joël Luijmes — this will be a little easier once we start publishing the helm chart with pinned versions.
    core-
    just excludes the
    all_extras
    dependencies in the pip install, but the K8s agent needs the kubernetes extra so it can’t use
    core-
    . When writing your own flows that need Prefect but not all the extras, it makes sense to start with the
    core-
    base for Docker storage