# Kubevirt on Openshift ## Kubevirt 架构 参考： KubeVirt使用面向服务的体系结构和编排模式构建。 ### KubeVirt包含的一系列服务 ``` | Cluster | (virt-controller) | ------------+--------------------------------------- | Kubernetes | (VMI CRD) | ------------+--------------------------------------- | DaemonSet | (virt-handler) (vm-pod M) | M: Managed by KubeVirt CRD: Custom Resource Definition ``` kubevirt支持docker 及 crio 两种容器运行时 需要虚拟化服务的用户与虚拟化 API通信，后者又与Kubernetes集群通信以安排所请求的虚拟机实例（VMI），调度，网络和存储都委托给Kubernetes，而KubeVirt则提供虚拟化功能。 ### 应用布局 * Cluster * KubeVirt 组件 * virt-controller * virt-handler * libvirtd * … * KubeVirt 管理 Pods * VMI Foo * VMI Bar * … ### 架构图 ![](https://379859240-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LZILiWXMjweRuuuoQKf%2F-Lmt0C8gZ8Fdn1sP4mXI%2F-Lmt0Cn3zj0Lx0w4XZXR%2Farchitecture.png?generation=1566472203299912\&alt=media) ### 示例流程：创建和删除VMI 下面的流程演示了KubeVirt中存在的几个(不是所有)组件之间的通信流。通常，可以将通信模式看作是编排，其中所有组件都自己执行操作，以实现VMI对象提供的状态。 ``` Client K8s API VMI CRD Virt Controller VMI Handler -------------------------- ----------- ------- ----------------------- ---------- listen <----------- WATCH /virtualmachines listen <----------------------------------- WATCH /virtualmachines | | POST /virtualmachines ---> validate | | create ---> VMI ---> observe --------------> observe | | v v validate <--------- POST /pods defineVMI create | | | | | | | schedPod ---------> observe | | | v | validate <--------- PUT /virtualmachines | update ---> VMI ---------------------------> observe | | | launchVMI | | | | : : : : | | | | DELETE /virtualmachines -> validate | | | delete ----> * ---------------------------> observe | | shutdownVMI | | | : : : ``` 免责声明：上图不完全准确，因为有一些临时的解决方法可以避免bugs和解决一些其他问题。 1. 客户端将新的VMI定义发布到K8s API服务器。 2. K8s API服务器验证输入并创建VMI自定义资源定义（CRD）对象。 3. virt-controller观察新VMI对象的创建并创建一个相应的pod。 4. Kubernetes正在主机上安排pod。 5. virt-controller观察到VMI的pod已启动并更新了VMI对象中的nodeName字段。现在已经设置了nodeName，责任转移到virt-handler以进行下一步的操作。 6. virt-handler（DaemonSet）观察到VMI已分配了运行它的主机。 7. virt-handler使用VMI规范，并在VMI的pod中使用libvirtd实例发出创建相应域的信号。 8. 客户端通过virt-api-server删除VMI对象。 9. virt-handler观察删除并关闭域。 ## 安装openshift 使用 crio(with runv) 参考： 主要的修改内容为 ``` [OSEv3:children] masters nodes etcd [OSEv3:vars] # Enable admission controller webhooks openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}} # CRI-O openshift_use_crio=true [nodes] master.example.com openshift_node_group_name='node-config-master-crio' infra.example.com openshift_node_group_name='node-config-infra-crio' node1.example.com openshift_node_group_name='node-config-compute-crio' ``` 需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中 `ansible.hosts.virt.crio.tpl` 安装 `offline-openshift-origin.3.11` 时使用即可 ## 如果 openshift 使用 docker（默认建议） 参考： 主要的修改内容为 ``` [OSEv3:vars] # Enable admission controller webhooks openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}} ``` 需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中 `ansible.hosts.virt.tpl` 安装 `offline-openshift-origin.3.11` 时使用即可 容器运行时使用docker时 需要关闭 节点selinux(启用sriov网络需要) ``` ansible nodes -m shell -a "setenforce 0 && sed -i 's/SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config" ``` ## 安装kubevirt ### 检查webhook配置 登录master主机 检查现有openshift集群master配置文件是否配置了MutatingAdmissionWebhook和ValidatingAdmissionWebhook，如果没有，在master主机修改`/etc/origin/master/master-config.yaml` 在下面位置添加相关配置 ``` admissionConfig: pluginConfig: ... MutatingAdmissionWebhook: configuration: apiVersion: v1 disable: true kind: DefaultAdmissionConfig ValidatingAdmissionWebhook: configuration: apiVersion: v1 disable: false kind: DefaultAdmissionConfig ... ``` 重启master api 和 controllers ``` master-restart api master-restart controllers ``` ### 部署kubevirt 登录安装机 安装oc客户端 ``` yum install -y origin-clients ``` clone kubevirt代码 ``` git clone https://github.com/paradeum-team/kubevirt-ansible.git ``` 登录管理员 ``` oc login -u -p master195.hisun.com:8443 ``` 给ui使用sa 添加管理员权限 ``` oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kubevirt-web-ui:kubevirt-web-ui-operator ``` 添加seviceaccount sriov-device-plugin privileged权限 ``` oc adm policy add-scc-to-user privileged -n kubevirt -z sriov-device-plugin ``` vi roles/kubevirt/templates/kubevirt-config.yaml 添加 `debug.useEmulation: 'true'`， 使用完全虚拟化 内容如下： ``` apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-config labels: kubevirt.io: "" data: feature-gates: "DataVolumes,SRIOV,LiveMigration,CPUManager" debug.useEmulation: 'true' ``` 修改变量 vi vars/all.yml ``` ... ### KubeVirt ### version: 0.19.0 ... ### Web UI ## ... kubevirt_web_ui_version: "v2.0.1" ... ``` 安装kubevirt ``` ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml ``` ## 下载virtctl ``` curl -L -o /usr/bin/virtctl https://github.com/kubevirt/kubevirt/releases/download/v0.19.0/virtctl-v0.19.0-linux-amd64 chmod +x /usr/bin/virtctl ``` ## 制作kubevirt 使用的 vmdisks ### 下载，并解压centos7.6.1811 云通用镜像 ``` # curl -O https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1811.qcow2.xz # xz -d CentOS-7-x86_64-GenericCloud-1811.qcow2.xz ``` ### 制作vm containerDisk 镜像 在kubevirt 0.20之前，containerDisk映像需要将 kubevirt/container-disk-v1alpha 作为基本映像。 ``` cat << END > Dockerfile FROM kubevirt/container-disk-v1alpha:v0.19.0 ADD CentOS-7-x86_64-GenericCloud-1905.qcow2 /disk END docker build -t offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905 . docker push offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905 ``` ## 发布 centos7 vm 访问 使用containerDisk 发布非持久化vm ``` apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: centos7-1905 spec: running: true template: metadata: labels: vm.kubevirt.io/name: centos7-1905 spec: domain: cpu: cores: 1 devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - bridge: {} name: default resources: requests: memory: 500M networks: - name: default pod: {} hostname: centos7-1905 terminationGracePeriodSeconds: 5 volumes: - containerDisk: image: 'offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905' name: rootdisk - cloudInitNoCloud: userData: | #cloud-config ssh_pwauth: yes chpasswd: list: | root:123456 centos:123456 expire: false name: cloudinitdisk ``` 使用 data-volume + localvolume 发布持久化 vm localvolume 使用参考：[openshift3.11配置local-volume](https://github.com/ss75710541/openshift-docs/blob/master/storage/openshift3.11%E9%85%8D%E7%BD%AElocal-volume.md) ``` apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: centos7 spec: dataVolumeTemplates: - metadata: name: centos7-rootdisk spec: pvc: accessModes: - ReadWriteOnce dataSource: null resources: requests: storage: 8Gi storageClassName: fast-disks source: http: url: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1907.qcow2.xz status: {} running: true template: metadata: creationTimestamp: null labels: vm.kubevirt.io/name: centos7 spec: domain: cpu: cores: 1 devices: disks: - bootOrder: 1 disk: bus: virtio name: rootdisk - bootOrder: 3 disk: bus: virtio name: cloudinitdisk interfaces: - bootOrder: 2 masquerade: {} name: nic0 rng: {} machine: type: '' resources: requests: memory: 512M hostname: centos7 networks: - name: nic0 pod: {} terminationGracePeriodSeconds: 0 volumes: - dataVolume: name: centos7-rootdisk name: rootdisk - cloudInitNoCloud: userData: | #cloud-config ssh_pwauth: yes chpasswd: list: | root:123456 centos:123456 expire: false name: cloudinitdisk status: created: true ``` 创建service, 暴露vm 内端口 ``` virtctl expose vm centos7-1905 --name vmiservice-centos7 --port 8100 --target-port 8100 ``` ## 卸载kubevirt 卸载kubevirt ``` ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml -e apb_action=deprovision ``` ## 附录 kubevirt 操作 API地址 --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://liujinye.gitbook.io/openshift-docs/kubevirt/kubevirt-on-openshift.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.