#StackBounty: #nginx #kubernetes #docker-registry #http-status-code-413 #kube-registry docker push causes http error 413: client intend…

Bounty: 50

I have a minikube K8S cluster running on a hyperv vm.
The cluster has an nginx ingress controller to expose services to outside of the cluster.
Outside of the cluster requests are exposed to the internet through an nginx that runs on the host machine.

Inside the cluster I also have a kube-registry with kube-registry-proxy running to store my docker images.

There however is an issue when I try to push larger images (about 32mb). In this case I get a 413 with the following error in my host nginx logfile (redacted for privacy reasons):

<redactedIP> - - [<redactedLocalDateTime>] "PATCH /v2/<redactedImageName>/blobs/uploads/<redactedUuid>?_state=<redactedState>%3D HTTP/1.1" 413 183 "-" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.114 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 x5C(windowsx5C))"

When looking in my ingress-nginx-controller logs in my k8s cluster, I see the following message:

[error] 371#371: *1004283 client intended to send too large body: 9930410 bytes, client: 127.0.0.1, server: , request: "PATCH /v2/<redactedImageName>/blobs/uploads/<redactedUuid> HTTP/1.1", host: ".<redactedHost>"

And looking at my kube-registry logs, I don’t see the PATCH request at all.
This suggests to me that the issue lies with the ingress-nginx-controller in my cluster.

I have looked into the issue and found some clues, regarding setting the client_max_body_size and client_body_buffer_size, either with the nginx configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  body-size: "1024m"
  proxy-body-size: "1024m"
  client-max-body-size: "1024m"
  client-body-buffer-size: "1024m"

This is set in the deployment of the ingress-nginx-controller using the --configmap argument

I also tried to set it using annotations on the kube-registry ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  labels:
    app: kube-registry-ingress
  name: kube-registry-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "1024m"
    nginx.org/client-max-body-size: "1024m"
    ...

I also made sure that client_max_body_size and client_body_buffer_size were set in the host nginx config.

I verified that the nginx config from the ingress-nginx-controller was set:

http {                       
        client_body_buffer_size         1024m;           
   
        ...

        ## start server <redactedDomain>
        server {                                                                                                                                                                                                                                                                                                                     
                server_name <redactedDomain> ;                                                                                           
                                                                                                                                                                                                                                                                                                                                                          
                listen 80  ;                                                                                                                                                                                                                                                                                                         
                listen 443  ssl http2 ;                                                                                                        
                                                                                                                                                                                                                                                                                                                                                          
                set $proxy_upstream_name "-";                                                                                                                                                                                                                                                                                        
                                                                                                                                               
                ssl_certificate_by_lua_block {                                                                                                                                                                                                                                                                                                            
                        certificate.call()                                                                                                                                                                                                                                                                                           
                }                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                          
                location / {                                                                                                                                                                                                                                                                                                         
                                                                                                                                               
                        set $namespace      "kube-system";                                                                                                                                                                                                                                                                                                
                        set $ingress_name   "kube-registry-ingress";                                                                                                                                                                                                                                                                 
                        set $service_name   "kube-registry-proxy";                                                                             
                        set $service_port   "80";                                                                                                                                                                                                                                                                                                         
                        set $location_path  "/";                                                                                                                                                                                                                                                                                     
                                                                                                                                               
                        rewrite_by_lua_block {                                                                                                                                                                                                                                                                                                            
                                lua_ingress.rewrite({                                                                                                                                                                                                                                                                                
                                        force_ssl_redirect = false,                                                                            
                                        ssl_redirect = true,                                                                                                                                                                                                                                                                                              
                                        force_no_ssl_redirect = false,                                                                                                                                                                                                                                                               
                                        use_port_in_redirects = false,                                                                         
                                })                                                                                                                                                                                                                                                                                                                        
                                balancer.rewrite()                                                                                                                                                                                                                                                                                   
                                plugins.run()                                                                                                  
                        }                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                     
                        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any                                    
                        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`                                                                                                                                                                                                           
                        # other authentication method such as basic auth or external auth useless - all requests will be allowed.                                                                                                                                                                                                    
                        #access_by_lua_block {                                                                                                 
                        #}                                                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                     
                        header_filter_by_lua_block {                                                                                           
                                lua_ingress.header()                                                                                                                                                                                                                                                                                                      
                                plugins.run()                                                                                                                                                                                                                                                                                        
                        }                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                          
                        body_filter_by_lua_block {                                                                                                                                                                                                                                                                                   
                        }                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                          
                        log_by_lua_block {                                                                                                                                                                                                                                                                                           
                                balancer.log()                                                                                                 
                                                                                                                                                                                                                                                                                                                                                          
                                monitor.call()                                                                                                                                                                                                                                                                                       
                                                                                                                                               
                                plugins.run()                                                                                                                                                                                                                                                                                                             
                        }                                                                                                                                                                                                                                                                                                            
                                                                                                                                               
                        port_in_redirect off;                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                                     
                        set $balancer_ewma_score -1;                                                                                           
                        set $proxy_upstream_name "kube-system-kube-registry-proxy-80";                                                                                                                                                                                                                                                                    
                        set $proxy_host          $proxy_upstream_name;                                                                                                                                                                                                                                                               
                        set $pass_access_scheme  $scheme;                                                                                      
                                                                                                                                                                                                                                                                                                                                                          
                        set $pass_server_port    $server_port;                                                                                                                                                                                                                                                                       
                                                                                                                                               
                        set $best_http_host      $http_host;                                                                                                                                                                                                                                                                                              
                        set $pass_port           $pass_server_port;                                                                                                                                                                                                                                                                  
                                                                                                                                               
                        set $proxy_alternative_upstream_name "";                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                                     
                        client_max_body_size                    1024m;                                                                         
                                                                                                                                                                                                                                                                                                                                                          
                        ...                                                                                                                                                                                                                                                                          
                        # Custom headers to proxied server   
                  
                        proxy_connect_timeout                   5s;                                                                                                                                                                                                                                                                  
                        proxy_send_timeout                      60s;                                                                           
                        proxy_read_timeout                      60s;                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                     
                        proxy_buffering                         off;                                                                           
                        proxy_buffer_size                       8k;                                                                                                                                                                                                                                                                                       
                        proxy_buffers                           4 8k;                                                                                                                                                                                                                                                                
                                                                                                                                               
                        proxy_max_temp_file_size                1024m;                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                     
                        proxy_request_buffering                 on;                                                                            
                        ...                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                     
                }                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                          
        }                                                                                                                                                                                                                                                                                                                            
        ## end server <redactedDomain>      

However, this didn’t seem to help.

My question is: how can I prevent a 413 HTTP status code that most likely comes from the ingress-nginx-controller?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.