#StackBounty: #kubernetes #istio Using Gateway + VirtualService + http01 + SDS

Bounty: 100

In the document there is an example about Securing Kubernetes Ingress with Cert-Manager which is not using Gateway + VirtualService.

I have tried to make it work with acme http01 but the certificate can not be issued as in log challenge I have 404 error. Seems it can not access to domain checking challenges. Is there any best practice with the specifications that I mentioned?

[Update 1]

I want to use istio gateway with SDS option for TLS and secure that by using cert-manager with http-01.

According to the documentation I found some example like Securing Kubernetes Ingress with Cert-Manager or Deploy a Custom Ingress Gateway Using Cert-Manager. However these examples are using Kuberenetes Ingress resource itself (Not istio gateway) or like the second example is using dns-01.

I need an instruction which including istio gateway with SDS option for TLS and secure that by using cert-manager with http-01. Istio gateway give me ability to use VirtualService.

Thanks!


Get this bounty!!!

#StackBounty: #kubernetes #kubernetes-helm #istio #telemetry ISTIO: telemetry traffic not showing correctly in grafana & kiali

Bounty: 50

I am new to kubernetes & istio,
trying to apply the bookinfo tutorial to my personal project, i don’t get the same results when monitoring traffic through kiali ui or grafana ui.

i believe i didn’t change much from the bookinfo project, here is the config i used (using helm)

##################################################################################################
# Webapp services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: "{{ .Values.service.name }}-svc"
  namespace: "{{ .Values.service.namespace }}"
  labels:
    app: "{{ .Values.service.name }}"
    service: "{{ .Values.service.name }}-svc"
spec:
  ports:
  - port: {{ .Values.service.port }}
    name: "{{ .Values.service.name }}-http"
  selector:
    app: "{{ .Values.service.name }}"
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: "{{ .Values.service.name }}-{{ .Values.service.version }}"
  namespace: "{{ .Values.service.namespace }}"
  labels:
    app: "{{ .Values.service.name }}"
    version: "{{ .Values.service.version }}"
spec:
  replicas: {{ .Values.productionDeployment.replicaCount }}
  selector:
    matchLabels:
      app: "{{ .Values.service.name }}"
      version: "{{ .Values.service.version }}"
  template:
    metadata:
      labels:
        app: "{{ .Values.service.name }}"
        version: "{{ .Values.service.version }}"
    spec:
      containers:
      - name: "{{ .Values.service.name }}"
        image: "{{ .Values.productionDeployment.image.repository }}:{{ .Values.productionDeployment.image.tag }}"
        imagePullPolicy: {{ .Values.productionDeployment.image.pullPolicy }}
        ports:
        - containerPort: {{ .Values.service.port }}
---

and here’s the istio config i used:

##################################################################################################
# Webapp gateway & virtual service
##################################################################################################
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: "{{ .Values.service.name }}-gateway"
  namespace: "{{ .Values.service.namespace }}"
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: "{{ .Values.service.name }}"
  namespace: "{{ .Values.service.namespace }}"
spec:
  hosts:
  - "*"
  gateways:
  - "{{ .Values.service.name }}-gateway"
  http:
  - match:
    - uri:
        exact: /
    route:
    - destination:
        host: "{{ .Values.service.name }}-svc"
        port:
          number: {{ .Values.service.port }}
---

this is what i see in kiali:
enter image description here

and in grafana: (notice there is no service Request volume)
enter image description here

however in prometheus i see traces:
enter image description here


Get this bounty!!!

#StackBounty: #kubernetes Can't get the usage of PVC through kubelet metrics

Bounty: 100

I have a Kubernetes 1.9.11 cluster on baremetal machines running Coreos 1576.5.0.

Recently I deployed a Glusterfs 4.1.7 cluster, managed by Heketi 8, and created a lot of PVCs to be used by some statfulset applications. The problem is, I can’t get metrics about these PVCs through Kublet’s 10250 port:

curl -k https://aa05:10250/metrics 2>/dev/null | grep kubelet_volume_stats | wc -l
0

So, how can I get these metrics?

Any hints will be appreciated.


Get this bounty!!!

#StackBounty: #networking #nginx #tcp #kubernetes #minikube Configuring TCP services with nginx ingress on minikube/k8s

Bounty: 50

I’m new to k8s/minikube (and to some extent, unix networking in general) so if I ask something that seems to make no sense, I’ll be happy to clarify!

Goal

I want to configure a port-based TCP ingress, as described briefly in the nginx-ingress docs. In particular, I want to use the webpack-dev-server from inside minikube.

Error

When it’s set up according to my best understanding, I still get Failed to load resource: net::ERR_CONNECTION_REFUSED when requesting local.web:3001/client.js. That is, navigating in my browser to ‘local.web/’ brings up the page, but without the bundle that webpack is meant to be producing. The request for that fails.

Configuration

Moving from host machine to minikube pod, I have

/etc/hosts:

On my dev machine, I set local.web to the minikube IP

$ echo "$(minikube ip) local.web" | sudo tee -a /etc/hosts

Ingress:

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "dev-web-ingress",
    "namespace": "dev",
    "selfLink": "/apis/extensions/v1beta1/namespaces/dev/ingresses/dev-web-ingress",
    "uid": "64ebfc93-612e-11e9-8df7-0800270e7244",
    "resourceVersion": "280750",
    "generation": 3,
    "creationTimestamp": "2019-04-17T16:32:30Z",
    "labels": {
      "platform": "advocate",
      "tier": "frontend"
    },
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"labels":{"platform":"advocate","tier":"frontend"},"name":"dev-web-ingress","namespace":"dev"},"spec":{"rules":[{"host":"local.web","http":{"paths":[{"backend":{"serviceName":"dev-adv-web-service","servicePort":"http"},"path":"/"}]}}]}}n",
      "kubernetes.io/ingress.class": "nginx"
    }
  },
  "spec": {
    "rules": [
      {
        "host": "local.web",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "dev-adv-web-service",
                "servicePort": "http"
              }
            }
          ]
        }
      }
    ]
  },
  "status": {
    "loadBalancer": {
      "ingress": [
        {
          "ip": "10.0.2.15"
        }
      ]
    }
  }
}

TCP Services

{
  "kind": "ConfigMap",
  "apiVersion": "v1",
  "metadata": {
    "name": "tcp-services",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/configmaps/tcp-services",
    "uid": "5e456f3e-622e-11e9-bcf8-0800270e7244",
    "resourceVersion": "295220",
    "creationTimestamp": "2019-04-18T23:04:50Z",
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","data":{"3001":"dev/dev-adv-web-service:3001","9290":"dev/dev-echoserver:8080"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"tcp-services","namespace":"dev"}}n"
    }
  },
  "data": {
    "3001": "dev/dev-adv-web-service:3001",
  }
}

Service:

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "dev-adv-web-service",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/services/dev-adv-web-service",
    "uid": "64e3c65d-612e-11e9-8df7-0800270e7244",
    "resourceVersion": "280675",
    "creationTimestamp": "2019-04-17T16:32:30Z",
    "labels": {
      "app": "adv-web",
      "tier": "frontend"
    },
    "annotations": [... edited for clarity]
  },
  "spec": {
    "ports": [
      {
        "name": "http",
        "protocol": "TCP",
        "port": 80,
        "targetPort": 3000,
        "nodePort": 31246
      },
      {
        "name": "http2",
        "protocol": "TCP",
        "port": 3001,
        "targetPort": 3001,
        "nodePort": 31392
      }
    ],
    "selector": {
      "app": "frontend-container",
      "tier": "frontend"
    },
    "clusterIP": "10.108.24.80",
    "type": "LoadBalancer",
    "sessionAffinity": "None",
    "externalTrafficPolicy": "Cluster"
  },
  "status": {
    "loadBalancer": {}
  }
}

Pod

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "dev-adv-web-768767454f-wxvnh",
    "generateName": "dev-adv-web-768767454f-",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/pods/dev-adv-web-768767454f-wxvnh",
    "uid": "65de844e-622c-11e9-bcf8-0800270e7244",
    "resourceVersion": "294073",
    "creationTimestamp": "2019-04-18T22:50:43Z",
    "labels": {
      "app": "frontend-container",
      "pod-template-hash": "768767454f",
      "tier": "frontend"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "ReplicaSet",
        "name": "dev-adv-web-768767454f",
        "uid": "4babd3e7-613d-11e9-8df7-0800270e7244",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },
  "spec": {
    "volumes": [
      {
        "name": "frontend-repo",
        "hostPath": {
          "path": "/Users/me/Projects/code/frontend",
          "type": ""
        }
      },
      {
        "name": "default-token-7rfht",
        "secret": {
          "secretName": "default-token-7rfht",
          "defaultMode": 420
        }
      }
    ],
    "containers": [
      {
        "name": "adv-web-container",
        "image": "localhost:5000/react:dev",
        "command": [
          "npm",
          "run",
          "dev"
        ],
        "ports": [
          {
            "name": "http",
            "containerPort": 3000,
            "protocol": "TCP"
          },
          {
            "name": "http2",
            "containerPort": 3001,
            "protocol": "TCP"
          }
        ],
        "env": [
          {
            "name": "HOSTNAME_PUBLISHED",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "HOSTNAME_PUBLISHED"
              }
            }
          },
          {
            "name": "LOCAL_DOMAIN",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "LOCAL_DOMAIN"
              }
            }
          },
          {
            "name": "HOST",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "HOST"
              }
            }
          },
          {
            "name": "WEBPACK_PUBLISHED_PORT",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "WEBPACK_PUBLISHED_PORT"
              }
            }
          },
          {
            "name": "WEBPACK_LISTEN_PORT",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "WEBPACK_LISTEN_PORT"
              }
            }
          },
          {
            "name": "API_URL",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "API_URL"
              }
            }
          },
          {
            "name": "LOGIN_CALLBACK_URL",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "LOGIN_CALLBACK_URL"
              }
            }
          },
          {
            "name": "NPM_CONFIG_LOGLEVEL",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "dev-frontend-configmap",
                "key": "NPM_CONFIG_LOGLEVEL"
              }
            }
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "frontend-repo",
            "mountPath": "/code"
          },
          {
            "name": "default-token-7rfht",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "Always"
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "ClusterFirst",
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "minikube",
    "securityContext": {},
    "schedulerName": "default-scheduler",
    "tolerations": [
      {
        "key": "node.kubernetes.io/not-ready",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      },
      {
        "key": "node.kubernetes.io/unreachable",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      }
    ],
    "priority": 0,
    "enableServiceLinks": true
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-04-18T22:50:43Z"
      },
      {
        "type": "Ready",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-04-18T22:50:45Z"
      },
      {
        "type": "ContainersReady",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-04-18T22:50:45Z"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-04-18T22:50:43Z"
      }
    ],
    "hostIP": "10.0.2.15",
    "podIP": "172.17.0.13",
    "startTime": "2019-04-18T22:50:43Z",
    "containerStatuses": [
      {
        "name": "adv-web-container",
        "state": {
          "running": {
            "startedAt": "2019-04-18T22:50:44Z"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "localhost:5000/react:dev",
        "imageID": "docker-pullable://localhost:5000/react@sha256:2bfe61ed134044bff4b23f5c057af2f9c480c3c1a1927a485f09f3410528903d",
        "containerID": "docker://57b9b6dafaf2aba8a21d5dd7db3543f4742c00331b49b48dc1561e3b5bd05315"
      }
    ],
    "qosClass": "BestEffort"
  }
}

Hypotheses

One thought was that the namespace on the TCP services ConfigMap was wrong. It’s not clear to me from the docs where that’s supposed to live. I have tried it in the namespace dev, where the ingress, service, and deployment/pods live. I also tried adding the data entry as above to the tcp-services ConfigMap in kube-system.

The logs for the webpack pod show no errors, so I don’t believe the problem is at the application level.

Since the GET local.web/ is returning data from the pod, I am convinced the service is at least partially correct.

I’m willing to perform any debugging you can suggest, and have no illusions that I know anything about what’s going on–I’ll be grateful for any help offered.


Get this bounty!!!

#StackBounty: #amazon-web-services #kubernetes Exposing a K8s TCP Service Endpoint to the Public Internet Without a Load Balancer

Bounty: 100

So I’m working on a project that involves managing many postgres instances inside of a k8s cluster. Each instance is managed using a Stateful Set with a Service for network communication. I need to expose each Service to the public internet via DNS on port 5432.

The most natural approach here is to use the k8s Load Balancer resource and something like external dns to dynamically map a DNS name to a load balancer endpoint. This is great for many types of services, but for databases there is one massive limitation: the idle connection timeout. AWS ELBs have a maximum idle timeout limit of 4000 seconds. There are many long running analytical queries/transactions that easily exceed that amount of time, not to mention potentially long-running operations like pg_restore.

So I need some kind of solution that allows me to work around the limitations of Load Balancers. Node IPs are out of the question since I will need port 5432 exposed for every single postgres instance in the cluster. Ingress also seems less than ideal since it’s a layer 7 proxy that only supports HTTP/HTTPS. I’ve seen workarounds with nginx-ingress involving some configmap chicanery, but I’m a little worried about committing to hacks like that for a large project. ExternalName is intriguing but even if I can find better documentation on it I think it may end up having similar limitations as NodeIP.

Any suggestions would be greatly appreciated.


Get this bounty!!!

#StackBounty: #debian #filesystems #nfs #kubernetes #nfsv4 Failed writes after a while on mounted nfs volume

Bounty: 50

I have mounted an nfs drive on the machine. I can see it with the following command:

$ df -aTh | grep nfs
... unrelated volumes ...
10.200.30.100:/export/pvc-70..1d nfs4  976M  2.0M  907M  1% /var/lib/kubelet/pods/72..1d/volumes/kubernetes.io~nfs/pvc-70..1d

[Where I added ellipsis for brevity’s sake.]
I repeatedly write to a file like from the machine itself (but doing this from a container yields the same result)

$ while true
  do echo test >> /var/lib/kubelet/pods/72..1d/volumes/kubernetes.io~nfs/pvc-70..1d/data.txt
  sleep 5
  done

This works but after a while I get the error:

-bash: /var/lib/kubelet/pods/72..1d/volumes/kubernetes.io~nfs/pvc-70..1d/data.txt: Invalid argument

This goes on for ever or as long as my patience lasts.
Is there any reason nfs should behave like this? Why does it stop allowing me to write so suddenly?
I tried giving 777 permissions but this didn’t affect anything.
Even creating a new file after about ~38 mins the same problem appears. The file is not writeable anymore from the machine. Then, (sometimes) if I read the file, for some reason I can write it again for a while.

I got the following log when the error is occurring:

[816270.396574] NFS: permission(0:231/8193), mask=0x81, res=0
[816270.403198] NFS: permission(0:231/8193), mask=0x3, res=0
[816270.406521] NFS: atomic_open(0:231/8193), data2.txt
[816270.412065] --> nfs_put_client({2})
[816270.415211] --> nfs4_setup_sequence clp ffff8fbd383c2800 session ffff8fbcf6fa1800 sr_slot 4294967295
[816270.419180] --> nfs41_setup_sequence
[816270.420853] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=3
[816270.424797] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[816270.428181] <-- nfs41_setup_sequence slotid=0 seqid=71351
[816270.430539] <-- nfs4_setup_sequence status=0
[816270.433691] encode_sequence: sessionid=1942:1545315805:1942:0 seqid=71351 slotid=0 max_slotid=0 cache_this=1
[816270.439233] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=3
[816270.442573] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[816270.444845] nfs4_free_slot: slotid 1 highest_used_slotid 0
[816270.447379] nfs41_sequence_process: Error 0 free the slot
[816270.451224] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295

Interestingly, when it works it doesn’t use atomic_open but just open.
(Source code here)

I thought I could debug this by setting authorisation to NONE instead of the default. How do I do that?

UPDATE:

I finally managed to reproduce the problem reliably! I turned on logging only for file and I get the following log that shows a transition between when I can write the file without problem and when the operation results in a EINVAL error.

...
[3533332.934037] NFS: open file(/data13.txt)
[3533332.937896] NFS: flush(/data13.txt)
[3533332.941986] NFS: fsync file(/data13.txt) datasync 0
[3533332.947038] NFS: write(/data13.txt, 33@0)
[3533332.952519] NFS: flush(/data13.txt)
[3533332.962836] NFS: fsync file(/data13.txt) datasync 0
[3533332.965829] NFS: release(/data13.txt)
[3533336.975155] docker0: port 2(veth295ffd2) entered blocking state
[3533336.977344] docker0: port 2(veth295ffd2) entered disabled state
[3533336.984250] device veth295ffd2 entered promiscuous mode
[3533336.995705] IPv6: ADDRCONF(NETDEV_UP): veth295ffd2: link is not ready
[3533337.061611] nfsd: initializing export module (net: ffff8fbceb052580).
[3533337.263506] eth0: renamed from veth9c5fe34
[3533337.274301] IPv6: ADDRCONF(NETDEV_CHANGE): veth295ffd2: link becomes ready
[3533337.277259] docker0: port 2(veth295ffd2) entered blocking state
[3533337.279440] docker0: port 2(veth295ffd2) entered forwarding state
[3533337.972794] NFS: open file(/data13.txt)
[3533337.982390] NFS: flush(/data13.txt)
[3533337.984978] NFS: fsync file(/data13.txt) datasync 0
[3533337.991418] NFS: write(/data13.txt, 33@0)
[3533337.998084] NFS: flush(/data13.txt)
[3533338.017990] NFS: fsync file(/data13.txt) datasync 0
[3533338.020636] NFS: release(/data13.txt)
[3533341.385814] NFSD: laundromat service - starting
[3533341.387690] NFSD: laundromat_main - sleeping for 90 seconds
[3533342.869641] NFS: open file(/data.txt)
[3533342.894059] NFS: write(/data.txt, 1@0)
[3533342.896545] NFS: flush(/data.txt)
[3533342.906735] NFS: fsync file(/data.txt) datasync 0
[3533342.910853] NFS: release(/data.txt)
[3533342.914387] NFS: open file(/data.txt)
[3533342.917169] NFS: write(/data.txt, 29@0)
[3533342.920237] NFS: flush(/data.txt)
[3533342.927772] NFS: fsync file(/data.txt) datasync 0
[3533342.931496] NFS: release(/data.txt)
[3533343.002610] NFS: read(/data.txt, 30@30374)
[3533343.233929] docker0: port 2(veth295ffd2) entered disabled state
[3533343.239144] veth9c5fe34: renamed from eth0
[3533343.257428] nfsd_inet6addr_event: removed fe80:0000:0000:0000:f026:2cff:feb1:510b
[3533343.260065] docker0: port 2(veth295ffd2) entered disabled state
[3533343.270280] device veth295ffd2 left promiscuous mode
[3533343.272992] docker0: port 2(veth295ffd2) entered disabled state
[3533343.330925] nfsd: shutting down export module (net: ffff8fbceb052580).
[3533343.337814] nfsd: export shutdown complete (net: ffff8fbceb052580).

In the above log I can write the file without problem until the last messages about shutting down the so called export module.


Get this bounty!!!