#StackBounty: #nginx #kubernetes #docker-registry #http-status-code-413 #kube-registry docker push causes http error 413: client intend…

Bounty: 50

I have a minikube K8S cluster running on a hyperv vm.
The cluster has an nginx ingress controller to expose services to outside of the cluster.
Outside of the cluster requests are exposed to the internet through an nginx that runs on the host machine.

Inside the cluster I also have a kube-registry with kube-registry-proxy running to store my docker images.

There however is an issue when I try to push larger images (about 32mb). In this case I get a 413 with the following error in my host nginx logfile (redacted for privacy reasons):

<redactedIP> - - [<redactedLocalDateTime>] "PATCH /v2/<redactedImageName>/blobs/uploads/<redactedUuid>?_state=<redactedState>%3D HTTP/1.1" 413 183 "-" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.114 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 x5C(windowsx5C))"

When looking in my ingress-nginx-controller logs in my k8s cluster, I see the following message:

[error] 371#371: *1004283 client intended to send too large body: 9930410 bytes, client:, server: , request: "PATCH /v2/<redactedImageName>/blobs/uploads/<redactedUuid> HTTP/1.1", host: ".<redactedHost>"

And looking at my kube-registry logs, I don’t see the PATCH request at all.
This suggests to me that the issue lies with the ingress-nginx-controller in my cluster.

I have looked into the issue and found some clues, regarding setting the client_max_body_size and client_body_buffer_size, either with the nginx configmap:

apiVersion: v1
kind: ConfigMap
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx-controller
  namespace: ingress-nginx
  body-size: "1024m"
  proxy-body-size: "1024m"
  client-max-body-size: "1024m"
  client-body-buffer-size: "1024m"

This is set in the deployment of the ingress-nginx-controller using the --configmap argument

I also tried to set it using annotations on the kube-registry ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
    app: kube-registry-ingress
  name: kube-registry-ingress
  namespace: kube-system
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "1024m"
    nginx.org/client-max-body-size: "1024m"

I also made sure that client_max_body_size and client_body_buffer_size were set in the host nginx config.

I verified that the nginx config from the ingress-nginx-controller was set:

http {                       
        client_body_buffer_size         1024m;           

        ## start server <redactedDomain>
        server {                                                                                                                                                                                                                                                                                                                     
                server_name <redactedDomain> ;                                                                                           
                listen 80  ;                                                                                                                                                                                                                                                                                                         
                listen 443  ssl http2 ;                                                                                                        
                set $proxy_upstream_name "-";                                                                                                                                                                                                                                                                                        
                ssl_certificate_by_lua_block {                                                                                                                                                                                                                                                                                                            
                location / {                                                                                                                                                                                                                                                                                                         
                        set $namespace      "kube-system";                                                                                                                                                                                                                                                                                                
                        set $ingress_name   "kube-registry-ingress";                                                                                                                                                                                                                                                                 
                        set $service_name   "kube-registry-proxy";                                                                             
                        set $service_port   "80";                                                                                                                                                                                                                                                                                                         
                        set $location_path  "/";                                                                                                                                                                                                                                                                                     
                        rewrite_by_lua_block {                                                                                                                                                                                                                                                                                                            
                                        force_ssl_redirect = false,                                                                            
                                        ssl_redirect = true,                                                                                                                                                                                                                                                                                              
                                        force_no_ssl_redirect = false,                                                                                                                                                                                                                                                               
                                        use_port_in_redirects = false,                                                                         
                        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any                                    
                        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`                                                                                                                                                                                                           
                        # other authentication method such as basic auth or external auth useless - all requests will be allowed.                                                                                                                                                                                                    
                        #access_by_lua_block {                                                                                                 
                        header_filter_by_lua_block {                                                                                           
                        body_filter_by_lua_block {                                                                                                                                                                                                                                                                                   
                        log_by_lua_block {                                                                                                                                                                                                                                                                                           
                        port_in_redirect off;                                                                                                                                                                                                                                                                                                             
                        set $balancer_ewma_score -1;                                                                                           
                        set $proxy_upstream_name "kube-system-kube-registry-proxy-80";                                                                                                                                                                                                                                                                    
                        set $proxy_host          $proxy_upstream_name;                                                                                                                                                                                                                                                               
                        set $pass_access_scheme  $scheme;                                                                                      
                        set $pass_server_port    $server_port;                                                                                                                                                                                                                                                                       
                        set $best_http_host      $http_host;                                                                                                                                                                                                                                                                                              
                        set $pass_port           $pass_server_port;                                                                                                                                                                                                                                                                  
                        set $proxy_alternative_upstream_name "";                                                                                                                                                                                                                                                                                          
                        client_max_body_size                    1024m;                                                                         
                        # Custom headers to proxied server   
                        proxy_connect_timeout                   5s;                                                                                                                                                                                                                                                                  
                        proxy_send_timeout                      60s;                                                                           
                        proxy_read_timeout                      60s;                                                                                                                                                                                                                                                                                      
                        proxy_buffering                         off;                                                                           
                        proxy_buffer_size                       8k;                                                                                                                                                                                                                                                                                       
                        proxy_buffers                           4 8k;                                                                                                                                                                                                                                                                
                        proxy_max_temp_file_size                1024m;                                                                                                                                                                                                                                                                                    
                        proxy_request_buffering                 on;                                                                            
        ## end server <redactedDomain>      

However, this didn’t seem to help.

My question is: how can I prevent a 413 HTTP status code that most likely comes from the ingress-nginx-controller?

Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.