集群内同步
使用argocd 部署 config-syncer(kubed)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: dev-config-syncer
namespace: cicd
spec:
destination:
name: cloud-k8s-test
namespace: kube-system
project: dev-config-syncer
source:
chart: kubed
helm:
valueFiles:
- values.yaml
repoURL: https://charts.appscode.com/stable/
targetRevision: v0.13.2
在kube-system 创建源 secret/configmap 配置
略
源 Secret 设置 kubed.appscode.com/sync Namespace Selector 注解
kubectl annotate secret netwarps-com-tls kubed.appscode.com/sync="tls=netwarps" -n kube-system
kubectl annotate secret registry-pld-cicd kubed.appscode.com/sync="registry=registry-pld-cicd" -n kube-system
同步目标namespace 设置对应label
for ns in adminer-system bigphoto elastic-system geth ipfs nft tekton-pipelines
do
kubectl label namespace $ns tls=netwarps
kubectl label namespace $ns registry=registry-pld-cicd
done
查看同步的registry secret
kubectl get secret -A |grep registry-pld-cicd
adminer-system registry-pld-cicd kubernetes.io/dockerconfigjson 1 13m
bigphoto registry-pld-cicd kubernetes.io/dockerconfigjson 1 13d
elastic-system registry-pld-cicd kubernetes.io/dockerconfigjson 1 13m
geth registry-pld-cicd kubernetes.io/dockerconfigjson 1 13m
ipfs registry-pld-cicd kubernetes.io/dockerconfigjson 1 13m
kube-system registry-pld-cicd kubernetes.io/dockerconfigjson 1 15m
nft registry-pld-cicd kubernetes.io/dockerconfigjson 1 34d
tekton-pipelines registry-pld-cicd
清理集群内同步的secret,谨慎操作，会同步删除集群内所有关联secret
# kubectl annotate secret netwarps-com-tls kubed.appscode.com/sync- -n kube-system
# kubectl annotate secret registry-pld-cicd kubed.appscode.com/sync- -n kube-system
跨集群同步
使用argocd 部署 config-syncer(kubed)
注：默认的config-syncer 跨集群同步，不支持rancher 代理的k8s cluster api地址, 相同的域名端口会被误判为同一个集群，导致同步失败，下面部署用的config-syncer 镜像是我修改过代码支持了rancher 代理后重新制作的docker 镜像
quay.io/netwarps/kubed@sha256:05e894200d732407763c552ce81633b79e63f7c6e7aaee5fcc7a13e1e1660b4c
完整argocd 部署 prod-config-syncer.yaml 如下:（部署到和rancher 相同k8s 集群,同步时使用rancher 内部代理后的k8s地址）
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prod-config-syncer
namespace: cicd
spec:
destination:
name: cloud-hk-kugga-prod
namespace: kube-system
project: prod-config-syncer
source:
chart: kubed
helm:
parameters:
# 默认为unicorn，同步后会打入同步后对应配置的label
- name: config.clusterName
value: cloud-hk-kugga-prod
- name: logLevel
value: "5"
- name: operator.tag
value: 05e894200d732407763c552ce81633b79e63f7c6e7aaee5fcc7a13e1e1660b4c
- name: operator.registry
value: quay.io/netwarps
- name: operator.repository
value: kubed@sha256
valueFiles:
- values.yaml
values: |-
config:
kubeconfigContent: |-
apiVersion: v1
kind: Config
clusters:
- name: "geth-uat-k8s"
cluster:
insecure-skip-tls-verify: true
server: "https://rancher.cattle-system.svc/k8s/clusters/c-m-xxxxxxxx"
- name: "geth-prod-k8s"
cluster:
insecure-skip-tls-verify: true
server: "https://rancher.cattle-system.svc/k8s/clusters/c-m-xxxxxxxx"
- name: "cloud-k8s-test"
cluster:
insecure-skip-tls-verify: true
server: "https://rancher.cattle-system.svc/k8s/clusters/c-m-xxxxxxxx"
users:
- name: "geth-uat-k8s"
user:
token: "kubeconfig-u-gdwznkhv4p:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
- name: "geth-prod-k8s"
user:
token: "kubeconfig-u-gdwzng6j9j:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
- name: "cloud-k8s-test"
user:
token: "kubeconfig-u-v5rjddpgpx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
contexts:
- name: "geth-uat-k8s"
context:
user: "geth-uat-k8s"
cluster: "geth-uat-k8s"
- name: "geth-prod-k8s"
context:
user: "geth-prod-k8s"
cluster: "geth-prod-k8s"
- name: "cloud-k8s-test"
context:
user: "cloud-k8s-test"
cluster: "cloud-k8s-test"
repoURL: https://charts.appscode.com/stable/
targetRevision: v0.13.2
在kube-system 创建源 secret/configmap 配置
略
配置跨集群secret同步的集群（默认同步kube-system namespace）
kubectl annotate secret netwarps-com-tls kubed.appscode.com/sync-contexts="cloud-k8s-test,geth-uat-k8s,geth-prod-k8s" -n kube-system
kubectl annotate secret registry-pld-cicd kubed.appscode.com/sync-contexts="cloud-k8s-test,geth-uat-k8s,geth-prod-k8s" -n kube-system
注：相关配置只同步到指定目标k8s 集群的 kube-system namespace ，如果需要同步其它namespace 参考集群内同步配置
配置集群内同步的secret
kubectl annotate secret netwarps-com-tls kubed.appscode.com/sync="tls=netwarps" -n kube-system
kubectl annotate secret registry-pld-cicd kubed.appscode.com/sync="registry=registry-pld-cicd" -n kube-system
清理跨集群secret同步的集群，谨慎操作，会同步删除关联目标集群secret
# kubectl annotate secret netwarps-com-tls kubed.appscode.com/sync-contexts- -n kube-system
# kubectl annotate secret registry-pld-cicd kubed.appscode.com/sync-contexts- -n kube-system
参考：
https://github.com/kubeops/config-syncer
https://appscode.com/products/kubed/v0.12.0/setup/install/
https://appscode.com/products/kubed/v0.12.0/
https://appscode.com/products/kubed/v0.12.0/guides/config-syncer/intra-cluster/
https://github.com/paradeum-team/config-syncer/tree/jyliu-dev/charts/kubed