> For the complete documentation index, see [llms.txt](https://liujinye.gitbook.io/openshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://liujinye.gitbook.io/openshift-docs/kubevirt/kubevirt-on-openshift.md). # Kubevirt on Openshift ## Kubevirt 架构 参考： KubeVirt使用面向服务的体系结构和编排模式构建。 ### KubeVirt包含的一系列服务 ``` | Cluster | (virt-controller) | ------------+--------------------------------------- | Kubernetes | (VMI CRD) | ------------+--------------------------------------- | DaemonSet | (virt-handler) (vm-pod M) | M: Managed by KubeVirt CRD: Custom Resource Definition ``` kubevirt支持docker 及 crio 两种容器运行时 需要虚拟化服务的用户与虚拟化 API通信，后者又与Kubernetes集群通信以安排所请求的虚拟机实例（VMI），调度，网络和存储都委托给Kubernetes，而KubeVirt则提供虚拟化功能。 ### 应用布局 * Cluster * KubeVirt 组件 * virt-controller * virt-handler * libvirtd * … * KubeVirt 管理 Pods * VMI Foo * VMI Bar * … ### 架构图 ![](/files/-Lmt0Cn3zj0Lx0w4XZXR) ### 示例流程：创建和删除VMI 下面的流程演示了KubeVirt中存在的几个(不是所有)组件之间的通信流。通常，可以将通信模式看作是编排，其中所有组件都自己执行操作，以实现VMI对象提供的状态。 ``` Client K8s API VMI CRD Virt Controller VMI Handler -------------------------- ----------- ------- ----------------------- ---------- listen <----------- WATCH /virtualmachines listen <----------------------------------- WATCH /virtualmachines | | POST /virtualmachines ---> validate | | create ---> VMI ---> observe --------------> observe | | v v validate <--------- POST /pods defineVMI create | | | | | | | schedPod ---------> observe | | | v | validate <--------- PUT /virtualmachines | update ---> VMI ---------------------------> observe | | | launchVMI | | | | : : : : | | | | DELETE /virtualmachines -> validate | | | delete ----> * ---------------------------> observe | | shutdownVMI | | | : : : ``` 免责声明：上图不完全准确，因为有一些临时的解决方法可以避免bugs和解决一些其他问题。 1. 客户端将新的VMI定义发布到K8s API服务器。 2. K8s API服务器验证输入并创建VMI自定义资源定义（CRD）对象。 3. virt-controller观察新VMI对象的创建并创建一个相应的pod。 4. Kubernetes正在主机上安排pod。 5. virt-controller观察到VMI的pod已启动并更新了VMI对象中的nodeName字段。现在已经设置了nodeName，责任转移到virt-handler以进行下一步的操作。 6. virt-handler（DaemonSet）观察到VMI已分配了运行它的主机。 7. virt-handler使用VMI规范，并在VMI的pod中使用libvirtd实例发出创建相应域的信号。 8. 客户端通过virt-api-server删除VMI对象。 9. virt-handler观察删除并关闭域。 ## 安装openshift 使用 crio(with runv) 参考： 主要的修改内容为 ``` [OSEv3:children] masters nodes etcd [OSEv3:vars] # Enable admission controller webhooks openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}} # CRI-O openshift_use_crio=true [nodes] master.example.com openshift_node_group_name='node-config-master-crio' infra.example.com openshift_node_group_name='node-config-infra-crio' node1.example.com openshift_node_group_name='node-config-compute-crio' ``` 需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中 `ansible.hosts.virt.crio.tpl` 安装 `offline-openshift-origin.3.11` 时使用即可 ## 如果 openshift 使用 docker（默认建议） 参考： 主要的修改内容为 ``` [OSEv3:vars] # Enable admission controller webhooks openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}} ``` 需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中 `ansible.hosts.virt.tpl` 安装 `offline-openshift-origin.3.11` 时使用即可 容器运行时使用docker时 需要关闭 节点selinux(启用sriov网络需要) ``` ansible nodes -m shell -a "setenforce 0 && sed -i 's/SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config" ``` ## 安装kubevirt ### 检查webhook配置 登录master主机 检查现有openshift集群master配置文件是否配置了MutatingAdmissionWebhook和ValidatingAdmissionWebhook，如果没有，在master主机修改`/etc/origin/master/master-config.yaml` 在下面位置添加相关配置 ``` admissionConfig: pluginConfig: ... MutatingAdmissionWebhook: configuration: apiVersion: v1 disable: true kind: DefaultAdmissionConfig ValidatingAdmissionWebhook: configuration: apiVersion: v1 disable: false kind: DefaultAdmissionConfig ... ``` 重启master api 和 controllers ``` master-restart api master-restart controllers ``` ### 部署kubevirt 登录安装机 安装oc客户端 ``` yum install -y origin-clients ``` clone kubevirt代码 ``` git clone https://github.com/paradeum-team/kubevirt-ansible.git ``` 登录管理员 ``` oc login -u -p master195.hisun.com:8443 ``` 给ui使用sa 添加管理员权限 ``` oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kubevirt-web-ui:kubevirt-web-ui-operator ``` 添加seviceaccount sriov-device-plugin privileged权限 ``` oc adm policy add-scc-to-user privileged -n kubevirt -z sriov-device-plugin ``` vi roles/kubevirt/templates/kubevirt-config.yaml 添加 `debug.useEmulation: 'true'`， 使用完全虚拟化 内容如下： ``` apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-config labels: kubevirt.io: "" data: feature-gates: "DataVolumes,SRIOV,LiveMigration,CPUManager" debug.useEmulation: 'true' ``` 修改变量 vi vars/all.yml ``` ... ### KubeVirt ### version: 0.19.0 ... ### Web UI ## ... kubevirt_web_ui_version: "v2.0.1" ... ``` 安装kubevirt ``` ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml ``` ## 下载virtctl ``` curl -L -o /usr/bin/virtctl https://github.com/kubevirt/kubevirt/releases/download/v0.19.0/virtctl-v0.19.0-linux-amd64 chmod +x /usr/bin/virtctl ``` ## 制作kubevirt 使用的 vmdisks ### 下载，并解压centos7.6.1811 云通用镜像 ``` # curl -O https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1811.qcow2.xz # xz -d CentOS-7-x86_64-GenericCloud-1811.qcow2.xz ``` ### 制作vm containerDisk 镜像 在kubevirt 0.20之前，containerDisk映像需要将 kubevirt/container-disk-v1alpha 作为基本映像。 ``` cat << END > Dockerfile FROM kubevirt/container-disk-v1alpha:v0.19.0 ADD CentOS-7-x86_64-GenericCloud-1905.qcow2 /disk END docker build -t offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905 . docker push offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905 ``` ## 发布 centos7 vm 访问 使用containerDisk 发布非持久化vm ``` apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: centos7-1905 spec: running: true template: metadata: labels: vm.kubevirt.io/name: centos7-1905 spec: domain: cpu: cores: 1 devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - bridge: {} name: default resources: requests: memory: 500M networks: - name: default pod: {} hostname: centos7-1905 terminationGracePeriodSeconds: 5 volumes: - containerDisk: image: 'offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905' name: rootdisk - cloudInitNoCloud: userData: | #cloud-config ssh_pwauth: yes chpasswd: list: | root:123456 centos:123456 expire: false name: cloudinitdisk ``` 使用 data-volume + localvolume 发布持久化 vm localvolume 使用参考：[openshift3.11配置local-volume](https://github.com/ss75710541/openshift-docs/blob/master/storage/openshift3.11%E9%85%8D%E7%BD%AElocal-volume.md) ``` apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: centos7 spec: dataVolumeTemplates: - metadata: name: centos7-rootdisk spec: pvc: accessModes: - ReadWriteOnce dataSource: null resources: requests: storage: 8Gi storageClassName: fast-disks source: http: url: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1907.qcow2.xz status: {} running: true template: metadata: creationTimestamp: null labels: vm.kubevirt.io/name: centos7 spec: domain: cpu: cores: 1 devices: disks: - bootOrder: 1 disk: bus: virtio name: rootdisk - bootOrder: 3 disk: bus: virtio name: cloudinitdisk interfaces: - bootOrder: 2 masquerade: {} name: nic0 rng: {} machine: type: '' resources: requests: memory: 512M hostname: centos7 networks: - name: nic0 pod: {} terminationGracePeriodSeconds: 0 volumes: - dataVolume: name: centos7-rootdisk name: rootdisk - cloudInitNoCloud: userData: | #cloud-config ssh_pwauth: yes chpasswd: list: | root:123456 centos:123456 expire: false name: cloudinitdisk status: created: true ``` 创建service, 暴露vm 内端口 ``` virtctl expose vm centos7-1905 --name vmiservice-centos7 --port 8100 --target-port 8100 ``` ## 卸载kubevirt 卸载kubevirt ``` ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml -e apb_action=deprovision ``` ## 附录 kubevirt 操作 API地址