Kubevirt 架构
参考：
https://github.com/kubevirt/kubevirt/blob/master/docs/architecture.md https://github.com/kubevirt/kubevirt/blob/master/docs/components.md
KubeVirt使用面向服务的体系结构和编排模式构建。
KubeVirt包含的一系列服务
|
Cluster | (virt-controller)
|
------------+---------------------------------------
|
Kubernetes | (VMI CRD)
|
------------+---------------------------------------
|
DaemonSet | (virt-handler) (vm-pod M)
|
M: Managed by KubeVirt
CRD: Custom Resource Definition
kubevirt支持docker 及 crio 两种容器运行时
需要虚拟化服务的用户与虚拟化 API通信，后者又与Kubernetes集群通信以安排所请求的虚拟机实例（VMI），调度，网络和存储都委托给Kubernetes，而KubeVirt则提供虚拟化功能。
应用布局
架构图
示例流程：创建和删除VMI
下面的流程演示了KubeVirt中存在的几个(不是所有)组件之间的通信流。通常，可以将通信模式看作是编排，其中所有组件都自己执行操作，以实现VMI对象提供的状态。
Client K8s API VMI CRD Virt Controller VMI Handler
-------------------------- ----------- ------- ----------------------- ----------
listen <----------- WATCH /virtualmachines
listen <----------------------------------- WATCH /virtualmachines
| |
POST /virtualmachines ---> validate | |
create ---> VMI ---> observe --------------> observe
| | v v
validate <--------- POST /pods defineVMI
create | | |
| | | |
schedPod ---------> observe |
| | v |
validate <--------- PUT /virtualmachines |
update ---> VMI ---------------------------> observe
| | | launchVMI
| | | |
: : : :
| | | |
DELETE /virtualmachines -> validate | | |
delete ----> * ---------------------------> observe
| | shutdownVMI
| | |
: : :
免责声明：上图不完全准确，因为有一些临时的解决方法可以避免bugs和解决一些其他问题。
客户端将新的VMI定义发布到K8s API服务器。
K8s API服务器验证输入并创建VMI自定义资源定义（CRD）对象。
virt-controller观察新VMI对象的创建并创建一个相应的pod。
virt-controller观察到VMI的pod已启动并更新了VMI对象中的nodeName字段。现在已经设置了nodeName，责任转移到virt-handler以进行下一步的操作。
virt-handler（DaemonSet）观察到VMI已分配了运行它的主机。
virt-handler使用VMI规范，并在VMI的pod中使用libvirtd实例发出创建相应域的信号。
客户端通过virt-api-server删除VMI对象。
安装openshift 使用 crio(with runv)
参考： https://docs.openshift.com/container-platform/3.11/cnv_install/cnv_install.html#cnv_introduction_to_cnv-cnv-install
主要的修改内容为
[OSEv3:children]
masters
nodes
etcd
[OSEv3:vars]
# Enable admission controller webhooks
openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}}
# CRI-O
openshift_use_crio=true
[nodes]
master.example.com openshift_node_group_name='node-config-master-crio'
infra.example.com openshift_node_group_name='node-config-infra-crio'
node1.example.com openshift_node_group_name='node-config-compute-crio'
需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中
ansible.hosts.virt.crio.tpl
安装 offline-openshift-origin.3.11 时使用即可
如果 openshift 使用 docker（默认建议）
参考： https://docs.openshift.com/container-platform/3.11/cnv_install/cnv_install.html#cnv_introduction_to_cnv-cnv-install
主要的修改内容为
[OSEv3:vars]
# Enable admission controller webhooks
openshift_master_admission_plugin_config={"ValidatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}},"MutatingAdmissionWebhook":{"configuration":{"kind": "DefaultAdmissionConfig","apiVersion": "v1","disable": false}}}
需要修改的内容，已经在offline安装包（openshift官方为基础自研）中添加到下面模板文件中
ansible.hosts.virt.tpl
安装 offline-openshift-origin.3.11 时使用即可
容器运行时使用docker时 需要关闭 节点selinux(启用sriov网络需要)
ansible nodes -m shell -a "setenforce 0 && sed -i 's/SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config"
安装kubevirt
检查webhook配置
登录master主机
检查现有openshift集群master配置文件是否配置了MutatingAdmissionWebhook和ValidatingAdmissionWebhook，如果没有，在master主机修改/etc/origin/master/master-config.yaml
在下面位置添加相关配置
admissionConfig:
pluginConfig:
...
MutatingAdmissionWebhook:
configuration:
apiVersion: v1
disable: true
kind: DefaultAdmissionConfig
ValidatingAdmissionWebhook:
configuration:
apiVersion: v1
disable: false
kind: DefaultAdmissionConfig
...
重启master api 和 controllers
master-restart api
master-restart controllers
部署kubevirt
登录安装机
安装oc客户端
yum install -y origin-clients
clone kubevirt代码
git clone https://github.com/paradeum-team/kubevirt-ansible.git
登录管理员
oc login -u <admin_user> -p <admin_password> master195.hisun.com:8443
给ui使用sa 添加管理员权限
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kubevirt-web-ui:kubevirt-web-ui-operator
添加seviceaccount sriov-device-plugin privileged权限
oc adm policy add-scc-to-user privileged -n kubevirt -z sriov-device-plugin
vi roles/kubevirt/templates/kubevirt-config.yaml
添加 debug.useEmulation: 'true'， 使用完全虚拟化
内容如下：
apiVersion: v1
kind: ConfigMap
metadata:
name: kubevirt-config
labels:
kubevirt.io: ""
data:
feature-gates: "DataVolumes,SRIOV,LiveMigration,CPUManager"
debug.useEmulation: 'true'
修改变量
vi vars/all.yml
...
### KubeVirt ###
version: 0.19.0
...
### Web UI ##
...
kubevirt_web_ui_version: "v2.0.1"
...
安装kubevirt
ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml
下载virtctl
curl -L -o /usr/bin/virtctl https://github.com/kubevirt/kubevirt/releases/download/v0.19.0/virtctl-v0.19.0-linux-amd64
chmod +x /usr/bin/virtctl
制作kubevirt 使用的 vmdisks
下载，并解压centos7.6.1811 云通用镜像
# curl -O https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1811.qcow2.xz
# xz -d CentOS-7-x86_64-GenericCloud-1811.qcow2.xz
制作vm containerDisk 镜像
在kubevirt 0.20之前，containerDisk映像需要将 kubevirt/container-disk-v1alpha 作为基本映像。
cat << END > Dockerfile
FROM kubevirt/container-disk-v1alpha:v0.19.0
ADD CentOS-7-x86_64-GenericCloud-1905.qcow2 /disk
END
docker build -t offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905 .
docker push offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905
发布 centos7 vm
访问 https://kubevirt-web-ui.xxx.xxx.com
使用containerDisk 发布非持久化vm
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
name: centos7-1905
spec:
running: true
template:
metadata:
labels:
vm.kubevirt.io/name: centos7-1905
spec:
domain:
cpu:
cores: 1
devices:
disks:
- disk:
bus: virtio
name: rootdisk
- disk:
bus: virtio
name: cloudinitdisk
interfaces:
- bridge: {}
name: default
resources:
requests:
memory: 500M
networks:
- name: default
pod: {}
hostname: centos7-1905
terminationGracePeriodSeconds: 5
volumes:
- containerDisk:
image: 'offlineregistry.dataman-inc.com:5000/vmdisks/centos7:1905'
name: rootdisk
- cloudInitNoCloud:
userData: |
#cloud-config
ssh_pwauth: yes
chpasswd:
list: |
root:123456
centos:123456
expire: false
name: cloudinitdisk
使用 data-volume + localvolume 发布持久化 vm
localvolume 使用参考：openshift3.11配置local-volume
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
name: centos7
spec:
dataVolumeTemplates:
- metadata:
name: centos7-rootdisk
spec:
pvc:
accessModes:
- ReadWriteOnce
dataSource: null
resources:
requests:
storage: 8Gi
storageClassName: fast-disks
source:
http:
url: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1907.qcow2.xz
status: {}
running: true
template:
metadata:
creationTimestamp: null
labels:
vm.kubevirt.io/name: centos7
spec:
domain:
cpu:
cores: 1
devices:
disks:
- bootOrder: 1
disk:
bus: virtio
name: rootdisk
- bootOrder: 3
disk:
bus: virtio
name: cloudinitdisk
interfaces:
- bootOrder: 2
masquerade: {}
name: nic0
rng: {}
machine:
type: ''
resources:
requests:
memory: 512M
hostname: centos7
networks:
- name: nic0
pod: {}
terminationGracePeriodSeconds: 0
volumes:
- dataVolume:
name: centos7-rootdisk
name: rootdisk
- cloudInitNoCloud:
userData: |
#cloud-config
ssh_pwauth: yes
chpasswd:
list: |
root:123456
centos:123456
expire: false
name: cloudinitdisk
status:
created: true
创建service, 暴露vm 内端口
virtctl expose vm centos7-1905 --name vmiservice-centos7 --port 8100 --target-port 8100
卸载kubevirt
卸载kubevirt
ansible-playbook -i localhost playbooks/kubevirt.yml -e@vars/all.yml -e apb_action=deprovision
附录
kubevirt 操作 API地址
https://kubevirt.io/api-reference/v0.19.0/operations.html
