{"id":1125,"date":"2015-09-22T21:46:56","date_gmt":"2015-09-22T19:46:56","guid":{"rendered":"https:\/\/elkano.org\/blog\/?p=1125"},"modified":"2015-09-25T11:00:16","modified_gmt":"2015-09-25T09:00:16","slug":"live-migration-openstack-ubuntu-14-04","status":"publish","type":"post","link":"https:\/\/elkano.org\/blog\/live-migration-openstack-ubuntu-14-04\/","title":{"rendered":"Live migration with OpenStack on Ubuntu 14.04"},"content":{"rendered":"

In this post I going to configure the compute nodes to enable the instance live migration on kvm instances backed with CEPH. In my set-up cinder volumes and the nova instances ephimeral disks are backed with CEPH so all the compute nodes can see all the storage.<\/p>\n

Assuming that cinder and nova is correctly integrated with CEPH we have to follow these steps to set up live migration:<\/p>\n

In libvirt-bin service configuration file we have to enable -l flag to libvirt-bin service args so it listen through tcp socket.
\n\/etc\/default\/libvirt-bin<\/strong><\/p>\n

# Defaults for libvirt-bin initscript (\/etc\/init.d\/libvirt-bin)\r\n# This is a POSIX shell fragment\r\n\r\n# Start libvirtd to handle qemu\/kvm:\r\nstart_libvirtd="yes"\r\n\r\n# options passed to libvirtd, add "-l" to listen on tcp\r\nlibvirtd_opts="-d -l"\r\n<\/pre>\n
In libvirtd configuration, set the options needed to listen on tcp:
\n\/etc\/libvirt\/libvirtd.conf<\/strong><\/p>\n
# Flag listening for secure TLS connections on the public TCP\/IP port.\r\nlisten_tls = 0\r\n# Listen for unencrypted TCP connections on the public TCP\/IP port.\r\nlisten_tcp = 1\r\ntcp_port = "16509"\r\n# Override the default configuration which binds to all network\r\n# interfaces. This can be a numeric IPv4\/6 address, or hostname\r\nlisten_addr = "172.17.16.117"\r\n# Authentication.\r\n#\r\n#  - none: do not perform auth checks. If you can connect to the\r\n#          socket you are allowed. This is suitable if there are\r\n#          restrictions on connecting to the socket (eg, UNIX\r\n#          socket permissions), or if there is a lower layer in\r\n#          the network providing auth (eg, TLS\/x509 certilos resultadosficates)\r\nauth_unix_ro = "none"\r\nauth_unix_rw = "none"\r\nauth_tcp = "none"\r\n<\/pre>\n
Because we are setting no auth for tcp connection you should take other actions for your production environment to ensure only certain servers are allowed to connect to this port, for example using iptables.<\/p>\n
Configure qemu user and group with root.
\n\/etc\/libvirt\/qemu.conf<\/strong><\/p>\n
# The user for QEMU processes run by the system instance. It can be\r\n# specified as a user name or as a user id. The qemu driver will try to\r\n# parse this value first as a name and then, if the name doesn't exist,\r\n# as a user id.\r\nuser = "root"\r\n# The group for QEMU processes run by the system instance. It can be\r\n# specified in a similar way to user.\r\ngroup = "root"\r\n# Whether libvirt should dynamically change file ownership\r\n# to match the configured user\/group above. Defaults to 1.\r\n# Set to 0 to disable file ownership changes.\r\ndynamic_ownership = 0\r\n<\/pre>\n
Once the changes are made restart the libvirt-bin service:<\/p>\n
\r\n$ sudo service libvirt-bin restart\r\nlibvirt-bin stop\/waiting\r\nlibvirt-bin start\/running, process 21411\r\n<\/pre>\n
Check if libvirt-bin is listening on tcp port 16509<\/p>\n
\r\n$ sudo netstat -npta | grep 16509  \r\ntcp        0      0 172.17.16.117:16509     0.0.0.0:*               LISTEN    21411\/libvirtd  \r\n<\/pre>\n
Set the needed flags in libvirt for live migration:
\n\/etc\/nova\/nova.conf<\/strong><\/p>\n
[libvirt]\r\n[..]\r\nlive_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED\r\nlive_migration_uri=qemu+tcp:\/\/%s\/system\r\n<\/pre>\n
Assuming that the compute nodes have different hardware you have to set up a common cpu model in nova.conf configuration file. You can set kvm64, the most compatible mode across Intel and AMD platforms or if you have intel cpus, like me, you can set SandyBridge. In any case, the mode you selected must be supported in all compute nodes.<\/p>\n
\/etc\/nova\/nova.conf<\/strong><\/p>\n
[libvirt]\r\n[..]\r\ntype = qemu\r\ncpu_mode=custom\r\ncpu_model=kvm64\r\n<\/pre>\n
[libvirt]\r\n[..]\r\ntype = qemu\r\ncpu_mode=custom\r\ncpu_model=SandyBridge\r\n<\/pre>\n
You can see all the cpu modes that kvm support with:<\/p>\n
\r\n$ \/usr\/bin\/qemu-system-x86_64 -cpu help\r\nx86           qemu64  QEMU Virtual CPU version 2.0.0                  \r\nx86           phenom  AMD Phenom(tm) 9550 Quad-Core Processor         \r\nx86         core2duo  Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz \r\nx86            kvm64  Common KVM processor                            \r\nx86           qemu32  QEMU Virtual CPU version 2.0.0                  \r\nx86            kvm32  Common 32-bit KVM processor                     \r\nx86          coreduo  Genuine Intel(R) CPU           T2600  @ 2.16GHz \r\nx86              486                                                  \r\nx86          pentium                                                  \r\nx86         pentium2                                                  \r\nx86         pentium3                                                  \r\nx86           athlon  QEMU Virtual CPU version 2.0.0                  \r\nx86             n270  Intel(R) Atom(TM) CPU N270   @ 1.60GHz          \r\nx86           Conroe  Intel Celeron_4x0 (Conroe\/Merom Class Core 2)   \r\nx86           Penryn  Intel Core 2 Duo P9xxx (Penryn Class Core 2)    \r\nx86          Nehalem  Intel Core i7 9xx (Nehalem Class Core i7)       \r\nx86         Westmere  Westmere E56xx\/L56xx\/X56xx (Nehalem-C)          \r\nx86      SandyBridge  Intel Xeon E312xx (Sandy Bridge)                \r\nx86          Haswell  Intel Core Processor (Haswell)                  \r\nx86       Opteron_G1  AMD Opteron 240 (Gen 1 Class Opteron)           \r\nx86       Opteron_G2  AMD Opteron 22xx (Gen 2 Class Opteron)          \r\nx86       Opteron_G3  AMD Opteron 23xx (Gen 3 Class Opteron)          \r\nx86       Opteron_G4  AMD Opteron 62xx class CPU                      \r\nx86       Opteron_G5  AMD Opteron 63xx class CPU                      \r\nx86             host  KVM processor with all supported host features (only available in KVM mode)\r\n<\/pre>\n
After these changes, if you see a message like this:<\/p>\n
\r\n$ sudo nova live-migration 6fba9cbe-66e2-484d-ba90-18ad519865ff host3\r\nERROR (BadRequest): Unacceptable CPU info: CPU doesn't have compatibility.\r\n<\/pre>\n
It could be caused by this bug #1082414<\/a>. In Juno, as a work around, you can comment out the line number 5010 “self._compare_cpu(source_cpu_info)” in libvirt driver:
\n\/usr\/lib\/python2.7\/dist-packages\/nova\/virt\/libvirt\/driver.py<\/strong><\/p>\n
# Compare CPU\r\nsource_cpu_info = src_compute_info['cpu_info']\r\n#self._compare_cpu(source_cpu_info)\r\n<\/pre>\n
In KILO this bug should be fixed, so no changes are needed in driver.py.<\/p>\n
I’m not so sure that the following is a requirement for the live migration, but it definitively is to enable the migration process and the instance resize, because some commands are run through a ssh connection.<\/p>\n
Enable ssh access between compute nodes with nova user. First edit each \/etc\/passwd file and enable shell access for your nova user:
\n\/etc\/passwd<\/strong><\/p>\n
nova:x:108:113::\/var\/lib\/nova:\/bin\/sh\r\n<\/pre>\n
Put this ssh configuration file in your nova home directory to avoid checking host’s keys between the compute nodes.
\n\/var\/lib\/nova\/.ssh\/config<\/strong><\/p>\n
Host *\r\n    StrictHostKeyChecking no\r\n<\/pre>\n
For each compute node create a rsa key pair as the nova user:<\/p>\n
$ ssh-keygen \r\nGenerating public\/private rsa key pair.\r\nEnter file in which to save the key (\/var\/lib\/nova\/.ssh\/id_rsa): \r\nCreated directory '\/var\/lib\/nova\/.ssh'.\r\nEnter passphrase (empty for no passphrase): \r\nEnter same passphrase again: \r\nYour identification has been saved in \/var\/lib\/nova\/.ssh\/id_rsa.\r\nYour public key has been saved in \/var\/lib\/nova\/.ssh\/id_rsa.pub.\r\nThe key fingerprint is:\r\ne1:97:a7:f5:10:71:bb:1f:9a:91:dd:c8:66:22:be:49 nova@host\r\nThe key's randomart image is:\r\n+--[ RSA 2048]----+\r\n|            . .  |\r\n|             o . |\r\n|        .   . .  |\r\n|       . . . ooo.|\r\n|        S + =o*o.|\r\n|         o = *+..|\r\n|          E  o. .|\r\n|         . o     |\r\n|          o      |\r\n+-----------------+\r\n<\/pre>\n
Copy all the created public keys content to an authorized_keys file and share it in all the compute nodes for the nova user:
\n\/var\/lib\/nova\/.ssh\/authorized_keys<\/strong><\/p>\n
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDl+XPbYlzlDm3F+5N2SCiZlCRL\/wZ9WAD3xwC5uNeza7NbQwy9jL5t2jHQn+bLMHP27GJO5Afl0cx9aPMe+mUvXDf0kk1yhND\/eqRauNjQ\/NONhUT9VDMiQBL7F28xWD+d0XTSr\/G1\/ddYxt\/ouoZF94nPXCLmzqY4JdwWCq2VV\/ChJRAXqs0tzPpOxmAGWNm7+mOxL4SFiFRCHR4LxxveV5rf10EzrOJFOEewUQ51yTqn8tuIs59nPuVzwNezYVJ4iZM3gcdm+rnE\/40I\/sodePDhiuIVkcT0Zl1stGVxVJrpsUtzE8+YsZLe+aH\/IlsHXMPdpCIbinyv0vmzIG1H nova@host1\r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUTvfP4RmRdRXIlWn72X+y+DKnwiDlz9iWqB+0zVhMmy3T4bYY4Okw5qXCZ6xOA2BLzsuY07QLNdFCHDs6FjPjEtT+A8U4w3x4aZDwS+jgl6eC3vpTU\/rkEpCDF\/KOvkvoP+U8zuKS4r1r5+UAoFAKvDCM8RGGwY6mC2+uEqv23at9OIrWrbkdHVlVnxhSYk4prg2PnePMFchs3Sh9yEaLw\/3F2wGBJGjYbVkfAu87UbQy6mRqWepJx8qSP2XYvIuVKleYpHS41Vk3H\/+L4tTR0ibYBD+eDR80IRN4qGE6vzdf7hJW1Gl0Ozx9fzSzO0u6f\/8254PqrNxya0PMmCbb nova@host2\r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDapvnExGGOKVx0XVqTPNWTwXR0kXLfzb2se1slb7oAL7clZShhUKDwFHOVRO16tV7k\/VD3mEf0Z+VBmU2MyxXa5nOIwbBCIIy9E\/01fXh9QcP5dn1Qs8GzsoNh4j3AHSDbmYgsaG0d+BrBxmF\/HpU+qZvBOMudT8reXT++5VQFNMP5cXkd6b8gyeYlrRH2SAaa7kIy44z3ZqQHzmFA+TJwYSrMoawgpdDE75HWQMAgiECXFK2Nb71+gd9sHOttzNPGmSx6TmbkHAi1W9rGYSZ88n1+19tHbnyZi+Qn8HYvKmLMyQFhje71DMwzK3FzbSpZuTaMfiEslRS9skYD6OTd nova@host3\r\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCrBNLab4QNjAIwGm7Ajc0CGHrtSlLnbV447vAdc\/QWRoU+yiBlv4NxWq3aOogczuq6ar3hufXAnUX7ClMTon6f2Fcq\/cv2D5V8YkXG7NtZQUKj0F6R27dEOUMPX64w2PGZen2QpcJNxLJXokbdTnDRc2odJ+0kw8rGKWDPioeLDjw5Qrb6EfddxWBJLbk3+gravyc2zHWMCzLUhRU4JMxBMutk3AXV2XBUflnOBoUMFixv8Mrm4wWQE3w29dZGL6wYtl2dAt9YENo9UIko\/jVreuAc5gTIr4v1iywzaDivLT2HR2BjqTkABOd9cuWw6o7ZS0lTTPf8skGxAGNSOoQT nova@host4\r\n<\/pre>\n
Check if you are able to run the ls command in a remote host in your compute nodes:<\/p>\n
$ ssh nova@host1 ls -l \/etc\/nova\/nova.conf \r\n-rw-r----- 1 nova nova 3329 sep 21 11:17 \/etc\/nova\/nova.conf\r\n<\/pre>\n
Now, you should be able to do an instance live migration between your compute nodes and the instance resize\/migration should work too without problems.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this post I going to configure the compute nodes to enable the instance live migration on kvm instances backed with CEPH. In my set-up cinder volumes and the nova instances ephimeral disks are backed with CEPH so all the compute nodes can see all the storage. Assuming that cinder and nova is correctly integrated […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,128],"tags":[121,159,38,149],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1125"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=1125"}],"version-history":[{"count":29,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1125\/revisions"}],"predecessor-version":[{"id":1154,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1125\/revisions\/1154"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=1125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=1125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=1125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}