openshift-docs
  • 不知所措的openshift kubernetes
  • 3scale
    • 在openshift使用3scale-operator部署3scale
  • Hyperledger-Fabric
    • Hyperledger Fabric on openshift 3.11
  • adminer
    • helm部署adminer
  • admission-controller
    • k8s nameapce增加默认node-selector和defaultTolerations
  • cert-manager
    • cert-manager-1.5升级到1.6
  • cicd
    • Argocd定时备份到us3
    • Argocd添加服务流程
    • Argocd自定义健康检查
    • helm安装argocd
    • k8s1.22部署gitlab对接keycloak
    • 使用Tekton+Helm-Chart+ArgoCD构建GitOps CICD
    • 使用 Tekton 构建CI流程
    • 使用argocd-notifications通知Tekton触发api-test
    • 使用 docker buildx 构建多CPU架构镜像
    • 使用image-syncer同步多CPU架构镜像到私有仓库
    • 开源helm chart 发布到 https://artifacthub.io/
    • 快速编写通用helm chart
  • client-go
    • k8s client-go 创建ingress示例
  • cluster-monitor-opertor
    • Openshift3.11 alertmanager 持久化
    • cluster-monitor-operator alertmanager配置
    • cluster-monitor-operator添加外部metrics
    • openshift3.11-cluster-monitoring-operator数据持久化
  • config-syncer
    • k8s使用config-syncer(kubed)同步secret
  • dns
    • k8s coredns 优化
    • k8s 使用coredns 自定义hosts 解析
  • dnsmasq
    • MAC 环境使用 dnsmasq 配置openshift相关自定义域名
    • 配置dnsmasq apps通配解析
  • elasticsearch
    • Elasticsearch查询重复数据
    • elasticsearch-kibana-8.10创建向量索引模板
    • openshift3.11中使用ECK安装filebeat+elasticsearch+kibana收集日志初探
    • openshift3.11部署eck1.6+es7.14.1
    • 使用kibana修改数据流索引mapping
  • etcd
    • k8s 1.22 使用cronjob 备份etcd
    • k8s1.22使用CronJob定时备份etcd到US3
    • 使用cronjob备份etcd
    • 恢复openshift3.11-etcd数据快照
  • flowiseai
    • argocd2.2.1+helm3.9-chart+k8s1.22部署flowise
  • ingress-nginx
    • ingress-nginx启用header名称中下划线
  • ipfs
    • golang计算文件ipfs cid
    • helm安装ipfs-cluster
  • kafka
    • banzaicloud-stable/kafka-operator+local-path迁移主机
    • 使用bitnami/kafka部署外部可访问的 kafka
  • keycloak
    • openshift使用keycloak登录
  • kong
    • Kong使用ip-pestriction插件配置IP白名单
    • kong admin api 使用 go-kong 调用
    • kong manager页面显示空白,报错net:ERR_HTTP2_PROTOCOL_ERROR
    • kong helm 安装
    • kong 自定义默认error html
    • 使用kong转发TCP服务
  • kube-flannel
    • kube-flannel-v0.20.1升级v0.22.2
  • kubeadm
    • RockLinux+kubeadm+k8s-1.22.16 升级到1.22.17
    • RockLinux+kubeadm+k8s-1.22.2 升级到1.22.16
  • kubevirt
    • Kubevirt on Openshift
    • kubebirt 中使用 cloud-init
    • kubevirt限制vm发布主机
    • openshift-3.11-kubevirt从v0.19.0升级到v0.27.0
    • 使用alpine-make-vm-image制作alpine-qcow2云镜像
    • 使用virtualbox自定义Alpine-vrit云镜像
  • load-balance
    • ucloud 添加负载均衡报文转发配置
  • metrics-sever
    • k8s-1.22安装metrics-server
  • mongodb
    • 使用argocd部署mongo-express
    • 阿里云 Mongodb副本集实例使用
  • mysql
    • Helm部署mysql
    • helm安装phpmyadmin
    • mysql批量修改utf8mb3为utf8mb4字符集
    • 部署MySQL Server Exporter
  • openfaas
    • OpenFaaS定时任务
    • OpenFaas使用Go模板创建Function
    • helm 安装openfaas
  • operator
    • 使用Operator-SDK构建基于Helm 的 Operator
  • playwright
    • 使用playwright截图Kibana图表
  • prometheus-operator
    • helm+kube-prometheus-stack-prometheus-operator+local-path(storageclass)部署的prometheus迁移主机
    • k8s 1.22 环境 kube-prometheus-stack 22.x 升级至 41.x
    • 使用helm+kube-prometheus-stack只部署prometheus
  • proxy
    • 使用快代理使用海外代理访问海外网站
  • rancher
    • helm 安装rancher 2.6.3
    • rancher-backup使用US3备份
    • rancher2.6.3升级至rancher2.6.9
    • rancher2.6.9对接keycloak
    • 解决rancher-v2.6.3报helm-operator更新rancher-webhook异常问题
    • 解决更新rancher2.6.13后报webhook和fleet chart版本不支持
  • raspberry-pi
    • mac os golang编译ARM环境go-sqlite3项目
    • 无头(headless) raspberry pi 4 ssh wifi 安装(mac)
    • 树莓派4B+raspberry-pi-os-buster在线安装k3s
    • 树莓派Raspberry Pi OS 设置静态ip
    • 树莓派raspberry-pi-os(32bit)安装docker
    • 树莓派raspberry pi os开启ssh
    • 树莓派安装centos7并简单优化
  • rbac
    • openshift给没能打开web terminal终端的用户添加权限
  • registry
    • 使用image-syncer同步所需镜像到仓库
  • ssh
    • Mac OSX ssh隧道使用方法
  • storage
    • lvm分区配置备份与恢复测试
    • openshift3.11使用nfs-client-provisioner+UCloud-UFS提供动态pv存储
    • openshift3.11使用nfs-client-provisioner+阿里云NAS提供动态nfs
    • openshift3.11配置local volume
    • openshift动态nfs
  • tracing
    • Ipfs cluseter使用分布式追踪系统jaeper tracing
  • troubleshooting
    • coredns service 连接超时(connection timed out; no servers could be reached)
    • etcdDatabaseHighFragmentationRatio 异常处理
    • helm更新服务报错提示statefulset更新是被禁止的
    • k8s如果防止容器中出现僵尸进程
    • kubevirt api server 证书过期问题导致openshfit调度异常
    • macOS Chrome访问https://registry-console-default.appsxxx.xxx.xxx/页面显示ERR_CERT_INVALID,且不能点继续
    • master 主机df 卡死
    • openshift project Terminaing处理
    • OpenShift Docker Registry 500
    • 解决openshift3.11 node NotReady csr Pending
    • openshift3.11-pvc-delete-Terminating-hang
    • openshift3.11清理Terminating 状态project
    • pod pending event报错cni无可用IP
    • ucloud环境开启selinux后/var/log/messages不能写入问题
    • ucloud环境开启selinux
    • 解决openshift3.11不能下载redhat registry.access.redhat.com中镜像问题
    • 证书未过期但是报NET::ERR_CERT_AUTHORITY_INVALID证书错误处理
  • walletconnect
    • WalletConnect-Relay 部署
Powered by GitBook
On this page
  • 故障描述
  • 故障排查
  • 使用curl 访问失败资源
  • 登录相同网关但非k8s集群内主机 curl 测试
  • 查看kong服务日志
  • 进制kong 服务pod debug
  • 解决问题
  • 参考

Was this helpful?

  1. kong

kong manager页面显示空白,报错net:ERR_HTTP2_PROTOCOL_ERROR

Previouskong admin api 使用 go-kong 调用Nextkong helm 安装

Last updated 1 year ago

Was this helpful?

故障描述

kong manager 页面打开后显示空白, 打开浏览器检查-网络查看加载url, 发现有三个资源加载异常, http code 200, 但是显示net:ERR_HTTP2_PROTOCOL_ERROR

故障排查

使用curl 访问失败资源

curl -v  https://kong.example.com/assets/monaco-editor.90904fcf.3_3_0_0.js

结果http code 200, 但是数据结果返回不完整,并且报错curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2),部分内容如下:

*   Trying x.x.x.x:443...
* Connected to kong.example.com (x.x.x.x) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.example.com
*  start date: Dec  6 00:00:00 2022 GMT
*  expire date: Dec 23 23:59:59 2023 GMT
*  subjectAltName: host "kong.example.com" matched cert's "*.example.com"
*  issuer: C=US; O=DigiCert, Inc.; CN=RapidSSL Global TLS RSA4096 SHA256 2022 CA1
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /assets/monaco-editor.90904fcf.3_3_0_0.js]
* h2h3 [:scheme: https]
* h2h3 [:authority: kong.example.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7f78ba80a800)
> GET /assets/monaco-editor.90904fcf.3_3_0_0.js HTTP/2
> Host: kong.example.com
> user-agent: curl/7.88.1
> accept: */*
>
< HTTP/2 200
< content-type: application/javascript; charset=UTF-8
< date: Thu, 27 Jul 2023 11:30:30 GMT
< expires: Wed, 25 Oct 2023 11:30:30 GMT
< cache-control: max-age=7776000
< cache-control: public
< x-frame-options: sameorigin
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-permitted-cross-domain-policies: master-only
< x-kong-upstream-latency: 7
< x-kong-proxy-latency: 0
< via: kong/3.3.1.0-enterprise-edition
...
ined,this._last=Dt.Undefined,this._size=0}unshift(e){return this._insert(e,!1)}push(e){return this._insert(e,!0)}_insert(e,t){const i=new Dt(e);if(this._first===Dt.Undefined)this._first=i,this._last=i;else if(t){const s=this._last;this._last=i,i.prev=s,s.next=i}else{const s=this._first;this._first=i,i.next=s,s.prev=i}this._size+=1;let n=!1;return()=>{n||(n=!0,this._remove(i))}}shift(){if(this._first!==Dt.Undefined){const e=this._first.element;return this._remove(this._first),e}}pop(){if(this._last!==Dt.Undefined){const e=this._last.element;return thi* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Connection #0 to host kong.example.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
s._remove(this._last),e}}_remove(e){if(e.prev!==Dt.Undefined&&e.next!==Dt.Undefined){const t=e.prev;t.next=e.next,e.next.prev=t}else e.prev===Dt.Undefined&&e.next===Dt.Undefined?(this._first=Dt.Undefined,this._last=Dt.Undefined):e.next===Dt.Undefined?(this._last=this._last.prev,this._last.next=Dt.Undefined):e.prev===Dt.Undefined&&(this._first=this._first.next,this._first.prev=Dt.Undefined);this._size-=1}*[Symbol.iterator](){let e=this._first;for(;e!==Dt.Undefined;)yield e.element,e=e.next}toArray(){const e=[];for(let t=this._first;t!==Dt.Undefined;t=t.next)e.push(t.element);return e}}var se;(function(o){o.None=()=>B.None;function e(C){return(w,L=null,D)=>{let S=!1,E;return E=C(I=>{if(!S)return E?E.dispose():S=!0,w.call(L,I)},null,D),S&&E.dispose(),E}}o.once=e;function t(C,w){return l((L,D=null,S)=>C(E=>L.call(D,w(E)),null,S))}o.map=t;function i(C,w){return l((L,D=null,S)=>C(E=>{w(E),L.call(D,E)},null,S))}o.forEach=i;function n(C,w){return l((L,D=null,S)=>C(E=>w(E)&&L.call(D,E),null,S))}o.filter=n;function s(C){return C}o.signal=s;function r(...C){return(w,L=null,D)=>ds(...C.map(S=>S(E=>w.call(L,E),null,D)))}o.any=r;functio%

登录相同网关但非k8s集群内主机 curl 测试

curl -v  https://kong.example.com/assets/monaco-editor.90904fcf.3_3_0_0.js

结果也是数据加载不完整,但是报错与直接外网访问有些不同,错误信息为curl: (18) transfer closed with outstanding read data remaining, 部分内容如下:

...
{constructor(){const e=[],t={type:"number",description:m("rulers.size","Number of monospace characters at which this editor ruler will render.")};super(83,"rulers",e,{type:"array",items:{anyOf:[t,{type:["object"],properties:{column:t,color:{type:"string",description:m("rulers.color","Color of this editor ruler."),format:"color-hex"}}}]},default:e,description:m("rulers","Render vertical rulers after a certain number of monospac* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
e characters. Use multiple values for multiple rulers. No rulers are drawn if array is empty.")})}validate(e){if(Array.isArray(e)){let t=[];for(let i of e)if(typeof i=="number")t.push({column:ct.clampedInt(i,0,0,1e4),color:null});else if(i&&typeof i=="object"){const n=i;t.push({column:ct.clampedInt(n.column,0,0,1e4),color:n.color})}return t.sort((i,n)=>i.column-n.column),t}return this.defaultValue}}function wL(o,e){if(typeof o!="string")return e;switch(o){case"hidden":return 2;case"visible":return 3

查看kong服务日志

kubectl logs test-kong-f9cc74965-g5cxc -n kong -c proxy --tail 1000

发现如下异常日志


2023/07/27 11:28:56 [crit] 2379#0: *22526 mkdir() "/kong_prefix/proxy_temp/6" failed (13: Permission denied) while reading upstream, client: 172.17.43.158, server: kong, request: "GET /assets/monaco-editor.90904fcf.3_3_0_0.js HTTP/1.1", upstream: "https://10.128.3.208:8445/assets/monaco-editor.90904fcf.3_3_0_0.js", host: "kong.example.com"

原来是 访问 失败的js 使用http2 协议时会缓存数据到 /kong_proxy/proxy_temp/6 提示没有限制而失败

进制kong 服务pod debug

kubectl exec -it test-kong-f9cc74965-g5cxc -n kong -c proxy sh
 ls -l kong_prefix/ -l

total 32
drwx------    2 kong     nogroup          6 Jul 28 06:59 client_body_temp
drwx------    2 kong     nogroup          6 Jul 28 06:59 fastcgi_temp
lrwxrwxrwx    1 kong     nogroup         19 Jul 28 06:59 gui -> /usr/local/kong/gui
drwxr-xr-x    2 kong     nogroup         24 Jul 28 06:59 gui_config
drwxr-xr-x    2 kong     nogroup        179 Jul 28 06:59 logs
-rw-r--r--    1 kong     nogroup       2626 Jul 28 06:59 nginx-kong-gui-include.conf
-rw-r--r--    1 kong     nogroup       4742 Jul 28 06:59 nginx-kong-stream.conf
-rw-r--r--    1 kong     nogroup      15418 Jul 28 06:59 nginx-kong.conf
-rw-r--r--    1 kong     nogroup        406 Jul 28 06:59 nginx.conf
drwxr-xr-x    2 kong     nogroup         23 Jul 28 06:59 pids
drwxr-xr-x    2 kong     nogroup          6 Jul 28 06:59 profiling
drwx------    2 root     root          6 Jul 28 06:59 proxy_temp
drwx------    2 kong     nogroup          6 Jul 28 06:59 scgi_temp
drwxr-xr-x    2 kong     nogroup         27 Jul 28 06:59 ssl
srw-rw-rw-    1 kong     nogroup          0 Jul 28 06:59 stream_rpc.sock
srw-rw-rw-    1 kong     nogroup          0 Jul 28 06:59 stream_tls_passthrough.sock
srw-rw-rw-    1 kong     nogroup          0 Jul 28 06:59 stream_tls_terminate.sock
srw-rw-rw-    1 kong     nogroup          0 Jul 28 06:59 stream_worker_events.sock
drwx------    2 kong     nogroup          6 Jul 28 06:59 uwsgi_temp
srw-rw-rw-    1 kong     nogroup          0 Jul 28 06:59 worker_events.sock

发现 proxy_temp 目录为root owner, 这个是因为最近修改error_template_html 挂载到这个目录,导致这个目录owner 变为root

解决问题

修改helm values 中 error template html 相关配置内容,修改挂载目录到一个新目录,不影响原有proxy_temp 目录

env:
  ...
  error_template_html: "/kong_prefix/error_template/error_template.html"
...
extraConfigMaps:
  - name: error-template-html
    mountPath: /kong_prefix/error_template

更新helm chart

helm upgrade --install test kong-2.25.0.tgz --namespace kong -f values.yaml

等Kong 服务正常更新启动后,再访问kong manager 页面后正常

参考

https://developer.aliyun.com/article/975820

image-20230728143410682