#StackBounty: #kubernetes #google-kubernetes-engine Why does kubectl port-forward require the destination service to be bound to localh…

Bounty: 100

I have noticed that kubectl port-forward only works when the process being forwarded to is listening on 127.0.0.1 or localhost. If the process is bound to another address, like the pod IP, then port forwarding doesn’t work.

For example, suppose I have a program that opens a server socket on the pod IP and port 8080, then this port forward does not work:

kubectl port-forward hasura-console-5d48cdb974-sv4l8 8080:8080

When I try to use it, its logs the following:

E0314 21:11:33.613318   93858 portforward.go:400] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod 9902636a8681f668702476c9e4148d6e4cd63ba20f4b3270fdc3c57d9ef9cb97, uid : exit status 1: 2021/03/15 01:11:34 socat[1486564] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

When I restart the pod with the service listening on 127.0.0.1, the port forward works and the above error is not logged.

Do services running in pods need to listen on localhost?


Get this bounty!!!

#StackBounty: #google-cloud-platform #google-kubernetes-engine #google-cloud-endpoints #openapi Google cloud endpoints is not supportin…

Bounty: 50

I developed my back end using Spring boot.

I’m integrating cloud endpoints, but I got a lot of errors because I can upload file (apk, images) with some of my APIs.

Here my swagger file :

 swagger: '2.0'
info:
    description: 'Api Documentation'
    version: '1.0.0'
    title: 'Api Documentation for apk micro service'
    termsOfService: 'urn:tos'
    contact: {}
    license:
        name: 'Apache 2.0'
        url: 'http://www.apache.org/licenses/LICENSE-2.0'
host: "mapk.api.gara.store"
basePath: /
tags:
    -
        name: apk-service-controller
        description: 'Apk Service Controller'
    -
        name: category-controller
        description: 'Category Controller'

schemes:
# Uncomment the next line if you configure SSL for this API.
# - "https"
  - "http"
paths:
    /api/apk/:
        post:
            tags: [apk-service-controller]
            summary: create
            operationId: createUsingPOST
            consumes: [application/json]
            produces: [application/json]
            parameters: [{in: body, name: apkDto, description: apkDto, required: true, schema: {$ref: '#/definitions/ApkDto'}}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
        put:
            tags: [apk-service-controller]
            summary: update
            operationId: updateUsingPUT
            consumes: [application/json]
            produces: [application/json]
            parameters: [{in: body, name: apkDto, description: apkDto, required: true, schema: {$ref: '#/definitions/ApkDto'}}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/activateapkversion:
        post:
            tags: [apk-service-controller]
            summary: activateApkVersion
            operationId: activateApkVersionUsingPOST
            consumes: [application/json]
            produces: ['*/*']
            parameters: [{name: apkId, in: query, description: apkId, required: true, type: integer, format: int64}, {name: codeVersion, in: query, description: codeVersion, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/addapk:
        post:
            tags: [apk-service-controller]
            summary: addApkFile
            operationId: addApkFileUsingPOST
            consumes: [multipart/form-data]
            produces: ['*/*']
            parameters: [{name: file, in: formData, description: file, required: true, type: file}, {name: apkId, in: query, description: apkId, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/ApkVersion'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/addorupdatecover:
        post:
            tags: [apk-service-controller]
            summary: uploadOrUpdateCoverImage
            operationId: uploadOrUpdateCoverImageUsingPOST
            consumes: [multipart/form-data]
            produces: ['*/*']
            parameters: [{name: file, in: formData, description: file, required: true, type: file}, {name: apkId, in: query, description: apkId, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/addorupdateicon:
        post:
            tags: [apk-service-controller]
            summary: uploadOrUpdateIconImage
            operationId: uploadOrUpdateIconImageUsingPOST
            consumes: [multipart/form-data]
            produces: ['*/*']
            parameters: [{name: file, in: formData, description: file, required: true, type: file}, {name: apkId, in: query, description: apkId, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/addscreenshots:
        post:
            tags: [apk-service-controller]
            summary: addOrUpdateScreenShot
            operationId: addOrUpdateScreenShotUsingPOST
            consumes: [multipart/form-data]
            produces: ['*/*']
            parameters: [{name: file, in: formData, description: file, required: true, type: file}, {name: apkId, in: query, description: apkId, required: true, type: integer, format: int64}, {name: index, in: query, description: index, required: true, type: integer, format: int32}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/category/:
        get:
            tags: [category-controller]
            summary: getAllCategories
            operationId: getAllCategoriesUsingGET
            produces: [application/json]
            responses: {'200': {description: OK, schema: {type: array, items: {$ref: '#/definitions/Category'}}}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
            security:
              - api_key: []
        post:
            tags: [category-controller]
            summary: createCategory
            operationId: createCategoryUsingPOST
            consumes: [application/json]
            produces: [application/json]
            parameters: [{name: categoryname, in: query, description: categoryname, required: true, type: string}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Category'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/category/addsubcategory:
        post:
            tags: [category-controller]
            summary: addSubCategory
            operationId: addSubCategoryUsingPOST
            consumes: [application/json]
            produces: ['*/*']
            parameters: [{name: categoryId, in: query, description: categoryId, required: true, type: integer, format: int64}, {name: subcategory, in: query, description: subcategory, required: true, type: string}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Category'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    /api/apk/category/removesubcategory:
        post:
            tags: [category-controller]
            summary: removeSubCategory
            operationId: removeSubCategoryUsingPOST
            consumes: [application/json]
            produces: ['*/*']
            parameters: [{name: categoryId, in: query, description: categoryId, required: true, type: integer, format: int64}, {name: subCategoryId, in: query, description: subCategoryId, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Category'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    '/api/apk/category/{categoryId}':
        delete:
            tags: [category-controller]
            summary: delete
            operationId: deleteUsingDELETE_1
            produces: ['*/*']
            parameters: [{name: categoryId, in: path, description: categoryId, required: true, type: integer, format: int64}]
            responses: {'200': {description: OK}, '204': {description: 'No Content'}, '401': {description: Unauthorized}, '403': {description: Forbidden}}
            security:
              - api_key: []
    '/api/apk/category/{id}':
        get:
            tags: [category-controller]
            summary: getCategoyById
            operationId: getCategoyByIdUsingGET
            produces: ['*/*']
            parameters: [{name: id, in: path, description: id, required: true, type: integer, format: int64}]
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Category'}}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
            security:
              - api_key: []
    '/api/apk/company/{companyId}':
        get:
            tags: [apk-service-controller]
            summary: getAllWebtoonByCompany
            operationId: getAllWebtoonByCompanyUsingGET
            produces: ['*/*']
            parameters: [{name: companyId, in: query, description: companyId, required: false, type: integer, format: int64}]
            responses: {'200': {description: OK, schema: {type: array, items: {$ref: '#/definitions/Apk'}}}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
            security:
              - api_key: []
    /api/apk/reviewandrated:
        post:
            tags: [apk-service-controller]
            summary: addAndUpdateReviewAndNote
            operationId: addAndUpdateReviewAndNoteUsingPOST
            consumes: [application/json]
            produces: [application/json]
            parameters: [{in: body, name: apkReviewDto, description: apkReviewDto, required: true, schema: {$ref: '#/definitions/ApkReviewDto'}}]
            security:
              - api_key: []
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '201': {description: Created}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
    '/api/apk/{id}':
        get:
            tags: [apk-service-controller]
            summary: getApkById
            operationId: getApkByIdUsingGET
            produces: ['*/*']
            parameters: [{name: id, in: path, description: id, required: true, type: integer, format: int64}]
            security:
              - api_key: []            
            responses: {'200': {description: OK, schema: {$ref: '#/definitions/Apk'}}, '401': {description: Unauthorized}, '403': {description: Forbidden}, '404': {description: 'Not Found'}}
        delete:
            tags: [apk-service-controller]
            summary: delete
            operationId: deleteUsingDELETE
            produces: ['*/*']
            parameters: [{name: id, in: path, description: id, required: true, type: integer, format: int64}]
            security:
              - api_key: []
            responses: {'200': {description: OK}, '204': {description: 'No Content'}, '401': {description: Unauthorized}, '403': {description: Forbidden}}

    "/auth/info/googlejwt":
        get:
            description: "Returns the requests' authentication information."
            operationId: "auth_info_google_jwt"
            produces:
                - "application/json"
            responses:
                200:
                  description: "Authenication info."
                  schema:
                    $ref: "#/definitions/authInfoResponse"
            security:
              - api_key: []
                google_jwt: []
    "/auth/info/googleidtoken":
        get:
            description: "Returns the requests' authentication information."
            operationId: "authInfoGoogleIdToken"
            produces:
                - "application/json"
            responses:
                200:
                  description: "Authenication info."
                  schema:
                    $ref: "#/definitions/authInfoResponse"
            security:
              - api_key: []
                google_id_token: []

definitions:
    Apk:
        type: object
        properties:
            apkDownloadUrl: {type: string}
            apkName: {type: string}
            apkPath: {type: string}
            category: {$ref: '#/definitions/Category'}
            changes: {type: string}
            companyId: {type: integer, format: int64}
            containsAds: {type: boolean}
            coverDownloadUrl: {type: string}
            coverPath: {type: string}
            dependencies: {type: array, items: {type: string}}
            description: {type: string}
            displayName: {type: string}
            iconDownloadUrl: {type: string}
            iconPath: {type: string}
            id: {type: integer, format: int64}
            installs: {type: integer, format: int64}
            isFree: {type: boolean}
            listOfapkVersion: {type: array, items: {$ref: '#/definitions/ApkVersion'}}
            maxSdkVersion: {type: string}
            minSdkVersion: {type: string}
            packageName: {type: string}
            permissions: {type: array, items: {type: string}}
            price: {type: number}
            rating: {$ref: '#/definitions/Rating'}
            repositoryGeneratedId: {type: string}
            screenshotList: {type: array, items: {type: string}}
            shortDescription: {type: string}
            size: {type: integer, format: int64}
            status: {type: string, enum: [INITIATED, PUBLISHED, VALIDATION_IN_PROGRESS, REJECTED]}
            subCategory: {$ref: '#/definitions/SubCategory'}
            targetSdkVersion: {type: string}
            userReview: {type: array, items: {$ref: '#/definitions/Review'}}
            versionCode: {type: integer, format: int64}
            versionName: {type: string}
            videoUrl: {type: string}
        title: Apk
    ApkDto:
        type: object
        properties:
            allMandatoryFieldsFilledForCreation: {type: boolean}
            apkId: {type: integer, format: int64}
            categoryId: {type: integer, format: int64}
            companyId: {type: integer, format: int64}
            description: {type: string}
            displayName: {type: string}
            isFree: {type: boolean}
            price: {type: number}
            shortDescription: {type: string}
            subCategoryId: {type: integer, format: int64}
        title: ApkDto
    ApkReviewDto:
        type: object
        properties:
            apkId: {type: integer, format: int64}
            appUserId: {type: integer, format: int64}
            minimumFielsRequiredSet: {type: boolean}
            reviewComment: {type: string}
            reviewId: {type: integer, format: int64}
            stars: {type: integer, format: int32}
        title: ApkReviewDto
    ApkVersion:
        type: object
        properties:
            artifactId: {type: string}
            compileSdkVersion: {type: string}
            compileSdkVersionCodename: {type: string}
            configForSplit: {type: string}
            dependencies: {type: array, items: {type: string}}
            downloadPath: {type: string}
            featureSplit: {type: boolean}
            groupId: {type: string}
            id: {type: integer, format: int64}
            installLocation: {type: string}
            isolatedSplits: {type: boolean}
            label: {type: string}
            maxSdkVersion: {type: string}
            minSdkVersion: {type: string}
            name: {type: string}
            packageName: {type: string}
            path: {type: string}
            permissions: {type: array, items: {type: string}}
            platformBuildVersionCode: {type: string}
            revisionCode: {type: integer, format: int64}
            sharedUserId: {type: string}
            sharedUserLabel: {type: string}
            size: {type: integer, format: int64}
            split: {type: string}
            splitRequired: {type: boolean}
            targetSdkVersion: {type: string}
            usesFeatures: {type: array, items: {type: string}}
            versionCode: {type: integer, format: int64}
            versionName: {type: string}
        title: ApkVersion
    Category:
        type: object
        properties:
            id: {type: integer, format: int64}
            name: {type: string}
            subCategoryList: {type: array, items: {$ref: '#/definitions/SubCategory'}}
        title: Category
    Rating:
        type: object
        properties:
            average: {type: number, format: float}
            ratingId: {type: integer, format: int64}
            stars: {type: object, additionalProperties: {type: integer, format: int32}}
            starsByUserId: {type: object, additionalProperties: {type: integer, format: int32}}
        title: Rating
    Review:
        type: object
        properties:
            appUserId: {type: integer, format: int64}
            comment: {type: string}
            commentDate: {type: string, format: date-time}
            id: {type: integer, format: int64}
            stars: {type: integer, format: int32}
        title: Review
    SubCategory:
        type: object
        properties:
            id: {type: integer, format: int64}
            name: {type: string}
        title: SubCategory

securityDefinitions:
  # This section configures basic authentication with an API key.
  api_key:
    type: "apiKey"
    name: "key"
    in: "query"
  # This section configures authentication using Google API Service Accounts
  # to sign a json web token. This is mostly used for server-to-server
  # communication.
  google_jwt:
    authorizationUrl: ""
    flow: "implicit"
    type: "oauth2"
    # This must match the 'iss' field in the JWT.
    x-google-issuer: "jwt-client.endpoints.sample.google.com"
    # Update this with your service account's email address.
    x-google-jwks_uri: "https://www.googleapis.com/service_accounts/v1/jwk/YOUR-SERVICE-ACCOUNT-EMAIL"
    # This must match the "aud" field in the JWT. You can add multiple
    # audiences to accept JWTs from multiple clients.
    x-google-audiences: "echo.endpoints.sample.google.com"
  # This section configures authentication using Google OAuth2 ID Tokens.
  # ID Tokens can be obtained using OAuth2 clients, and can be used to access
  # your API on behalf of a particular user.
  google_id_token:
    authorizationUrl: ""
    flow: "implicit"
    type: "oauth2"
    x-google-issuer: "https://accounts.google.com"
    x-google-jwks_uri: "https://www.googleapis.com/oauth2/v3/certs"
    # Your OAuth2 client's Client ID must be added here. You can add
    # multiple client IDs to accept tokens from multiple clients.
    x-google-audiences: "YOUR-CLIENT-ID"

I got these following errors with my deploy command gcloud --verbosity=debug endpoints services deploy openapi.yaml :

INFO: Refreshing access_token
DEBUG: Running [gcloud.endpoints.services.deploy] with arguments: [--verbosity: "debug", SERVICE_CONFIG_FILE:1: "[u'openapi.yaml']"]
INFO: No JSON detected in service config. Trying YAML...
DEBUG: (gcloud.endpoints.services.deploy) INVALID_ARGUMENT: Cannot convert to service config.
'location: "unknown location"
kind: ERROR
message: "http: repeated message field 'google.protobuf.Struct.fields' referred to by message 'AddApkFileUsingPOSTRequest' cannot be mapped as an HTTP parameter."

 location: "unknown location"
kind: ERROR
message: "http: cyclic message field 'google.protobuf.Struct.FieldsEntry.value' referred to by message 'AddApkFileUsingPOSTRequest' in method 'method 1.mapk_api_gara_store.AddApkFileUsingPOST' cannot be mapped as an HTTP parameter."

 location: "unknown location"
kind: ERROR
message: "http: cyclic message field 'google.protobuf.ListValue.values' referred to by message 'AddApkFileUsingPOSTRequest' in method 'method 1.mapk_api_gara_store.AddApkFileUsingPOST' cannot be mapped as an HTTP parameter."

location: "unknown location"
kind: ERROR
message: "http: repeated message field 'google.protobuf.Struct.fields' referred to by message 'AddOrUpdateScreenShotUsingPOSTRequest' cannot be mapped as an HTTP parameter."

 location: "unknown location"
kind: ERROR
message: "http: cyclic message field 'google.protobuf.Struct.FieldsEntry.value' referred to by message 'AddOrUpdateScreenShotUsingPOSTRequest' in method 'method 1.mapk_api_gara_store.AddOrUpdateScreenShotUsingPOST' cannot be mapped as an HTTP parameter."

Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 984, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 807, in Run
    resources = command_instance.Run(args)
  File "/usr/lib64/google-cloud-sdk/lib/surface/endpoints/services/deploy.py", line 356, in Run
    validate_only=self.validate_only)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/endpoints/services_util.py", line 333, in PushMultipleServiceConfigFiles
    api_response = client.services_configs.Submit(submit_request)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/third_party/apis/servicemanagement/v1/servicemanagement_v1_client.py", line 273, in Submit
    config, request, global_params=global_params)
  File "/usr/lib64/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 731, in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
  File "/usr/lib64/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
  File "/usr/lib64/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
    http_response, method_config=method_config, request=request)
HttpBadRequestError: HttpError accessing <https://servicemanagement.googleapis.com/v1/services/mapk.api.gara.store/configs:submit?alt=json>: response: <{'status': '400', 'content-length': '3538', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Tue, 17 Mar 2020 20:58:29 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000', 'content-type': 'application/json; charset=UTF-8'}>, content <{
  "error": {
    "code": 400,
    "message": "Cannot convert to service config.n'location: "unknown location"nkind: ERRORnmessage: "http: repeated message field \'google.protobuf.Struct.fields\' referred to by message \'AddApkFileUsingPOSTRequest\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.Struct.FieldsEntry.value\' referred to by message \'AddApkFileUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.AddApkFileUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.ListValue.values\' referred to by message \'AddApkFileUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.AddApkFileUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: repeated message field \'google.protobuf.Struct.fields\' referred to by message \'UploadOrUpdateCoverImageUsingPOSTRequest\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.Struct.FieldsEntry.value\' referred to by message \'UploadOrUpdateCoverImageUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.UploadOrUpdateCoverImageUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.ListValue.values\' referred to by message \'UploadOrUpdateCoverImageUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.UploadOrUpdateCoverImageUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: repeated message field \'google.protobuf.Struct.fields\' referred to by message \'UploadOrUpdateIconImageUsingPOSTRequest\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.Struct.FieldsEntry.value\' referred to by message \'UploadOrUpdateIconImageUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.UploadOrUpdateIconImageUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.ListValue.values\' referred to by message \'UploadOrUpdateIconImageUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.UploadOrUpdateIconImageUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: repeated message field \'google.protobuf.Struct.fields\' referred to by message \'AddOrUpdateScreenShotUsingPOSTRequest\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.Struct.FieldsEntry.value\' referred to by message \'AddOrUpdateScreenShotUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.AddOrUpdateScreenShotUsingPOST\' cannot be mapped as an HTTP parameter."nn location: "unknown location"nkind: ERRORnmessage: "http: cyclic message field \'google.protobuf.ListValue.values\' referred to by message \'AddOrUpdateScreenShotUsingPOSTRequest\' in method \'method 1.mapk_api_gara_store.AddOrUpdateScreenShotUsingPOST\' cannot be mapped as an HTTP parameter."n'",
    "status": "INVALID_ARGUMENT"
  }
}
>
ERROR: (gcloud.endpoints.services.deploy) INVALID_ARGUMENT: Cannot convert to service config.
'location: "unknown location"
kind: ERROR
message: "http: repeated message field 'google.protobuf.Struct.fields' referred to by message 'AddApkFileUsingPOSTRequest' cannot be mapped as an HTTP parameter."

 location: "unknown location"
kind: ERROR
message: "http: cyclic message field 'google.protobuf.Struct.FieldsEntry.value' referred to by message 'AddApkFileUsingPOSTRequest' in method 'method 1.mapk_api_gara_store.AddApkFileUsingPOST' cannot be mapped as an HTTP parameter."

How could I do ? It seems that cloud endpoints is not supporting file. My app is a market place, I need to upload file through my APIs.

Thanks


Get this bounty!!!

#StackBounty: #google-cloud-platform #kubernetes #google-kubernetes-engine #prometheus Correcting clock skew in a GKE cluster

Bounty: 50

I have the following alert configured in prometheus:

alert: ClockSkewDetected
expr: abs(node_timex_offset_seconds{job="node-exporter"})
  > 0.03
for: 2m
labels:
  severity: warning
annotations:
  message: Clock skew detected on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}. Ensure NTP is configured correctly on this host.

This alert is part of the default kube-prometheus stack which I am using.

I find this alert fires for around 10 mins every day or two.

I’d like to know how to deal with this problem (the alert firing!). It’s suggested in this answer that I shouldn’t need to run NTP (via a daemonset I guess) myself on GKE.

I’m also keen to use the kube-prometheus defaults where possible – so I’m unsure about increasing the 0.03 value.


Get this bounty!!!

#StackBounty: #google-cloud-platform #kubernetes #google-kubernetes-engine Google Cloud Kuberbetes run-away systemd 100% CPU usage

Bounty: 50

Last week, after upgrading our GKE cluster to Kubernetes 1.13.6-gke.13, some of our nodes started to fail due to high CPU usage. Pods on these nodes would be CPU starved, work poorly and get killed due to failing liveness checks.

This is what top shows when we SSH into a problem node:

top - 10:11:27 up 5 days, 22 min,  1 user,  load average: 23.71, 21.90, 20.32
Tasks: 858 total,   3 running, 854 sleeping,   0 stopped,   1 zombie
%Cpu(s): 33.5 us, 30.9 sy,  0.0 ni, 35.5 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  30157.0 total,  14125.6 free,   4771.2 used,  11260.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24762.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0  114636  22564   4928 R  65.2   0.1   5254:53 systemd
    356 message+  20   0  110104  98784   2916 S  33.8   0.3   2705:49 dbus-daemon
   1236 root      20   0 3174876 147328  57464 S  22.0   0.5   1541:29 kubelet
    366 root      20   0   12020   3864   3396 S  21.6   0.0   1705:21 systemd-logind
   1161 root      20   0 5371428 586832  26756 S  18.7   1.9   1848:18 dockerd
    432 root      20   0 5585144  49040  13420 S  11.1   0.2 848:54.06 containerd
  23797 root      20   0  994620   8784   6088 S   3.0   0.0  96:58.79 containerd-shim
  45359 root      20   0  994620   8968   5600 S   3.0   0.0 100:28.46 containerd-shim
  35913 root      20   0 1068352   8192   5600 S   2.3   0.0 104:54.12 containerd-shim
  10806 root      20   0  994620   8908   5596 S   2.0   0.0 102:57.45 containerd-shim
  15378 root      20   0  994620   9084   5600 S   2.0   0.0 102:24.08 containerd-shim
  33141 root      20   0  994620   8856   5848 S   2.0   0.0  95:26.89 containerd-shim
  34299 root      20   0  994620   8824   5848 S   2.0   0.0  90:55.28 containerd-shim
  48411 root      20   0  994620   9488   6216 S   2.0   0.0  95:38.56 containerd-shim
1824641 root      20   0 1068352   6836   5716 S   2.0   0.0  65:45.81 containerd-shim
  10257 root      20   0  994620   9404   5596 S   1.6   0.0 101:10.45 containerd-shim
  15400 root      20   0 1068352   8916   6216 S   1.6   0.0  97:47.99 containerd-shim
  22895 root      20   0 1068352   8408   5472 S   1.6   0.0 102:55.97 containerd-shim
  29969 root      20   0  994620   9124   5600 S   1.6   0.0  98:32.32 containerd-shim
  34720 root      20   0  994620   8444   5472 S   1.6   0.0  97:23.98 containerd-shim
  10073 root      20   0 1068352   9796   6152 S   1.3   0.0 100:54.30 containerd-shim

To attempt to resolve the issue we recreated all the nodes. We created a new pool with equivalent resources and migrated all pods over by scaling down the old pool to 0 nodes. (This was difficult because at least 2 of our previous nodes failed to shut down, even after a long time. In the end we had to shut down the underlying VMs to kill those nodes.) At first this seemed to help, node CPU usage and load averages were low, but then the problem returned.

Next we created yet another pool, this time with twice as much CPU. It didn’t help. Some nodes still had extremely high CPU usage for systemd, dbus-daemon, kubelet etc.

We kept that setup though. Now we are running with tons of extra CPU per node and, although expensive, that masks the problem (there’s enough CPU to also run actual pods in addition to the problematic system services).

How do we find out what’s actually wrong here?


Snippet from journalctl -u kubelet:

Jul 04 05:49:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 05:49:34.849808    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 05:54:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 05:54:34.850598    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 05:59:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 05:59:34.851797    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:04:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:04:34.858344    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:09:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:09:34.859626    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:14:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:14:34.861142    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:19:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:19:34.862185    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:24:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:24:34.863160    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:29:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:29:34.864156    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:34:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:34:34.865041    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:39:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:39:34.866044    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:44:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:44:34.866969    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:49:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:49:34.867979    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:54:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:54:34.869429    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 06:59:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 06:59:34.870359    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:04:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:04:34.871190    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:09:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:09:34.872063    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:14:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:14:34.873240    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:19:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:19:34.874123    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:24:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:24:34.875010    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:29:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:29:34.876612    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:34:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:34:34.877420    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:39:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:39:34.878368    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:44:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:44:34.879221    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:49:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:49:34.880239    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:54:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:54:34.881172    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 07:59:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 07:59:34.881868    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:04:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:04:34.882653    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:09:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:09:34.883432    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:14:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:14:34.884175    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:19:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:19:34.885043    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:24:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:24:34.885845    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Jul 04 08:29:34 gke-cluster0-pool-1-9c29a343-vqtq kubelet[1236]: I0704 08:29:34.886690    1236 container_manager_linux.go:434] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service

Output from strace -c -p `pidof systemd`:

strace: Process 1 attached
strace: Process 1 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 24.20    0.109101           5     22827           lstat
 17.45    0.078654           9      8610           sendmsg
 15.51    0.069914           7     10549           read
 10.96    0.049406           3     17310           getpid
  6.45    0.029075           8      3806           openat
  6.08    0.027385           7      3783           readlinkat
  5.76    0.025945           3      7579           fstat
  4.47    0.020167           4      5700           getrandom
  3.62    0.016301           4      3806           close
  3.14    0.014133           7      1892           access
  2.28    0.010278           5      1924        11 stat
  0.03    0.000145           4        33           epoll_wait
  0.02    0.000089           4        22           readlink
  0.01    0.000029           3        11           prctl
  0.01    0.000029           3        11           clock_gettime
  0.01    0.000027           2        11           getgid
  0.01    0.000026           2        11           geteuid
  0.00    0.000020           2        11           getuid
  0.00    0.000020           2        11           getegid
------ ----------- ----------- --------- --------- ----------------
100.00    0.450744                 87907        11 total

Since dbus is very active, I took a look at that too: dbus-monitor --system --profile | head -n 20

dbus-monitor: unable to enable new-style monitoring: org.freedesktop.DBus.Error.AccessDenied: "Rejected send message, 1 matched rules; type="method_call", sender=":1.165" (uid=5004 pid=769854 comm="dbus-monitor --system --profile ") interface="org.freedesktop.DBus.Monitoring" member="BecomeMonitor" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)". Falling back to eavesdropping.
#type    timestamp    serial    sender    destination    path    interface    member
#                    in_reply_to
sig    1562263380.765023    2    org.freedesktop.DBus    :1.165    /org/freedesktop/DBus    org.freedesktop.DBus    NameAcquired
sig    1562263380.953812    132870362    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d5d1be6e08bfe7552d6f9ee50a943eca88dd0dd749ec248594aa0be91879a2cdb_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.957890    132870363    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d5d1be6e08bfe7552d6f9ee50a943eca88dd0dd749ec248594aa0be91879a2cdb_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.957912    132870364    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d539cd8b8367913182e438aa1b3b05714c8f3f131e20bcadabdeb850c375a8125_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.957918    132870365    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d539cd8b8367913182e438aa1b3b05714c8f3f131e20bcadabdeb850c375a8125_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.957923    132870366    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2dcontainers_2d4c04ee8d1bf693ff2c9300b198b2b47bbf2c240265af5b9edc1f07b6cbd0c1ce_2dmounts_2dshm_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958014    132870367    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2dcontainers_2d4c04ee8d1bf693ff2c9300b198b2b47bbf2c240265af5b9edc1f07b6cbd0c1ce_2dmounts_2dshm_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958020    132870368    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d872dd2c0a63a9f3b5c9c5e4972e06fcf89d28b4c7f59aea7ea4d38f5a6bf0db6_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958025    132870369    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d872dd2c0a63a9f3b5c9c5e4972e06fcf89d28b4c7f59aea7ea4d38f5a6bf0db6_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958029    132870370    :1.157    <none>    /org/freedesktop/systemd1/unit/home_2dkubernetes_2dcontainerized_5fmounter_2drootfs_2dvar_2dlib_2dkubelet_2dpods_2d2f4e6eae_5cx2d9e51_5cx2d11e9_5cx2db4ee_5cx2d42010a80000f_2dvolumes_2dkubernetes_2eio_5cx7esecret_2dfluentd_5cx2dgcp_5cx2dtoken_5cx2dzfrkb_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958111    132870371    :1.157    <none>    /org/freedesktop/systemd1/unit/home_2dkubernetes_2dcontainerized_5fmounter_2drootfs_2dvar_2dlib_2dkubelet_2dpods_2d2f4e6eae_5cx2d9e51_5cx2d11e9_5cx2db4ee_5cx2d42010a80000f_2dvolumes_2dkubernetes_2eio_5cx7esecret_2dfluentd_5cx2dgcp_5cx2dtoken_5cx2dzfrkb_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958117    132870372    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2dkubelet_2dpods_2d2f4e6eae_5cx2d9e51_5cx2d11e9_5cx2db4ee_5cx2d42010a80000f_2dvolumes_2dkubernetes_2eio_5cx7esecret_2dfluentd_5cx2dgcp_5cx2dtoken_5cx2dzfrkb_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958121    132870373    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2dkubelet_2dpods_2d2f4e6eae_5cx2d9e51_5cx2d11e9_5cx2db4ee_5cx2d42010a80000f_2dvolumes_2dkubernetes_2eio_5cx7esecret_2dfluentd_5cx2dgcp_5cx2dtoken_5cx2dzfrkb_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958126    132870374    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d667d713fe45ea0af609c85fbfd3aafbca9494574fe10509bda8cd3d13d1e6b66_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958223    132870375    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d667d713fe45ea0af609c85fbfd3aafbca9494574fe10509bda8cd3d13d1e6b66_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958229    132870376    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d0a9f899f23c0965b43cae81dd04f46a78e4b1b063fa5679323146b06932474a9_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958233    132870377    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d0a9f899f23c0965b43cae81dd04f46a78e4b1b063fa5679323146b06932474a9_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged
sig    1562263380.958238    132870378    :1.157    <none>    /org/freedesktop/systemd1/unit/var_2dlib_2ddocker_2doverlay2_2d875734c124694cd54a7c26516f740cedcb853a18ff5805a89669747c1188b65d_2dmerged_2emount    org.freedesktop.DBus.Properties    PropertiesChanged


Get this bounty!!!