[关闭]
@Alex-Zhao 2018-08-08T10:49:40.000000Z 字数 9828 阅读 173

在neutron中使用keepalived中的VIP，无法ping通VIP

OpenStack

在OpenStack中给同事分配3个虚拟机，配置了keepalived，发现VIP ping不通，即时是同网段同宿主一样ping不通。
排查发现ICMP包都没有进入宿主机物理网卡。所以考虑到应该物理交换机根本就不知道这个IP的mac地址发送至哪里。或者ARP数据包根本没有从宿主机出去，由于这个IP不是从neutron dhcp agent下发的，可能被neutron隔离。
So,根据目前的现象和思路继续排查，发现在OpenStack的安全组（iptables）中有几条记录

  1. [root@compute152 ~]# iptables -vnL neutron-linuxbri-s389017b5-f
  2. Chain neutron-linuxbri-s389017b5-f (1 references)
  3.  pkts bytes target     prot opt in     out     source               destination         
  4. 18884 2406K RETURN     all  --  *      *       10.230.46.131        0.0.0.0/0            MAC FA:16:3E:A3:35:C4 /* Allow traffic from defined IP/MAC pairs. */
  5.     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Drop traffic without an IP/MAC allow rule. */

以上信息时针对一个虚拟机进行流量过滤，如果源IP不是neutron的给定IP和mac地址就DROP掉，可以清楚，VIP的ARP信息根本就没有出来，直接被DROP掉了。

解决方法：
解决方法就是通过neutron中的allow_address_pairs来实现。

  1. 1、 keepalived + allow_address_pairs实现虚拟机高可用
  2. # keepalived实现instance high available（Neutron flat网络模式下）
  4. [root@openstack-1 ~(keystone_admin)]# neutron net-list （找出pub-net的id号）
  6. [root@openstack-1 ~(keystone_admin)]# neutron port-list --netework_id=54f5ea9b-5d05-42e3-995f-c00e6824be25(这个id就是pub-net的id)   查看这个网络下的所有port信息
  8. [root@openstack-1 ~(keystone_admin)]# neutron port-create --fixed-ip ip_address=202.106.179.130(虚拟ip地址) --security-group default pub-net  创建keepalived的vip
  10. [root@openstack-1 ~(keystone_admin)]# neutron port-update  fee2f24e-87a1-4e23-b60b-8d4a33f9257f(这个id是web2的port id) --allowed_address_pairs list=true type=dict ip_address=202.106.179.130   
  12. [root@openstack-1 ~(keystone_admin)]# neutron port-update   4082ae4a-5af6-43ea-9370-fa493fb9ad67(这个id是web1的port id)  --allowed_address_pairs list=true type=dict ip_address=202.106.179.130 （这个ip地址就可以被keepavlied当作vip用了）
  14. [root@openstack-1 ~(keystone_admin)]# neutron port-show  4082ae4a-5af6-43ea-9370-fa493fb9ad67  查看信息

以上信息也可以通过curl访问执行：

  1. curl -i https://10.224.159.113:443/v2.0/ports/08ccf4de-f6e2-4d4d-bcdf-55532e93f32f -X PUT -d '{"port":{"allowed_address_pairs":[{"ip_address": "10.224.148.59"}]}}' -H "X-Auth-Token: 1969e5caed5949b98c64a2556d5b43e2" -H "Accept: application/json" -H "Content-Type: application/json" 
  1. 2、 关闭neutron port的安全组特性
  2. neutron port-update --no-security-groups $port_id
  3. neutron port-update $port_id --port-security-enabled=False

执行完成后通过neutron port-show --id来查看对应的端口已经有一条allow_address_pairs记录

  1. [root@controller1 ~]# neutron port-show a1aa5dd1-34bf-4dd9-9f4c-40ff138bcb9b
  2. +-----------------------+--------------------------------------------------------------------------------------+
  3. | Field                 | Value                                                                                |
  4. +-----------------------+--------------------------------------------------------------------------------------+
  5. | admin_state_up        | True                                                                                 |
  6. | allowed_address_pairs | {"ip_address": "10.230.40.205", "mac_address": "fa:16:3e:0a:cc:5b"}                  |
  7. | binding:host_id       | compute151                                                                           |
  8. | binding:profile       | {}                                                                                   |
  9. | binding:vif_details   | {"port_filter": true}                                                                |
  10. | binding:vif_type      | bridge                                                                               |
  11. | binding:vnic_type     | normal                                                                               |
  12. | created_at            | 2018-04-14T02:54:12Z                                                                 |
  13. | description           |                                                                                      |
  14. | device_id             | a72f1072-33fc-410d-91b7-5d550a5ee227                                                 |
  15. | device_owner          | compute:nova                                                                         |
  16. | extra_dhcp_opts       |                                                                                      |
  17. | fixed_ips             | {"subnet_id": "e489820a-2b92-4249-a01e-f40db49ad380", "ip_address": "10.230.40.206"} |
  18. | id                    | a1aa5dd1-34bf-4dd9-9f4c-40ff138bcb9b                                                 |
  19. | mac_address           | fa:16:3e:0a:cc:5b                                                                    |
  20. | name                  |                                                                                      |
  21. | network_id            | 2060b35b-f0f3-461d-8b2d-7ad28baebd3e                                                 |
  22. | port_security_enabled | True                                                                                 |
  23. | project_id            | 34906ac2c713454d8bdfbd09fa32a6fb                                                     |
  24. | revision_number       | 30563                                                                                |
  25. | security_groups       | 214b5ee1-19df-4fe2-970d-ef4fa3574fd0                                                 |
  26. | status                | ACTIVE                                                                               |
  27. | tenant_id             | 34906ac2c713454d8bdfbd09fa32a6fb                                                     |
  28. | updated_at            | 2018-04-19T09:42:46Z                                                                 |
  29. +-----------------------+--------------------------------------------------------------------------------------+

再次查看iptables发现在该虚拟机链中多了一条规则。

  1. [root@compute151 ~]# iptables -vnL neutron-linuxbri-sa1aa5dd1-3
  2. Chain neutron-linuxbri-sa1aa5dd1-3 (1 references)
  3.  pkts bytes target     prot opt in     out     source               destination         
  4.     2   168 RETURN     all  --  *      *       10.230.40.205        0.0.0.0/0            MAC FA:16:3E:0A:CC:5B /* Allow traffic from defined IP/MAC pairs. */
  5.   187 16908 RETURN     all  --  *      *       10.230.40.206        0.0.0.0/0            MAC FA:16:3E:0A:CC:5B /* Allow traffic from defined IP/MAC pairs. */
  6.     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Drop traffic without an IP/MAC allow rule. */

如何清空allowed_address_pairs：

  1. neutron port-update c2dad3ae-c545-447e-b75a-ed4568d25c89 --no-allowed-address-pairs

续：
以上步骤只是允许VIP可以通信，但是在keepalived中需要使用VRRP协议，VRRP是一种多播协议，在OpenStack中默认是不允许多播通信的，所以需要在安全组中开启VRRP（协议号112）协议，而且这个协议不能放在default安全组中，需要新创建个端口组，将端口更新安全组。

  1. [root@localhost ~]# neutron port-list --network_id=6fdfeb22-8887-4d94-a46e-64010b3cb4ce | egrep "46.101|46.102"
  2. | 0bf67d69-6b3b-4d0c-9f30-e63afb18ac04 |      | fa:16:3e:fb:dc:37 | {"subnet_id": "d326a3e4-18c3-4404-86e9-9916d9ebc7f4", "ip_address": "10.230.46.102"} |
  3. | ad8b7a99-fce8-446c-a8ef-4aca05e4d844 |      | fa:16:3e:1a:fe:1f | {"subnet_id": "d326a3e4-18c3-4404-86e9-9916d9ebc7f4", "ip_address": "10.230.46.101"} |
  5. neutron port-update 929cf04f-094d-4318-8f1e-77773eade576 --security-group vrrp
  6. neutron port-update 4e18a5c0-7845-4c6a-b32a-93b93493f701 --security-group vrrp
  9. [root@localhost ~]# neutron security-group-show vrrp
  10. +----------------------+--------------------------------------------------------------------+
  11. | Field                | Value                                                              |
  12. +----------------------+--------------------------------------------------------------------+
  13. | created_at           | 2018-08-08T01:54:56Z                                               |
  14. | description          |                                                                    |
  15. | id                   | 7da53cec-e267-4cc7-847f-a63341da58a8                               |
  16. | name                 | vrrp                                                               |
  17. | project_id           | 22938c8796b74e6cbb4bba13c9bde822                                   |
  18. | revision_number      | 10                                                                 |
  19. | security_group_rules | {                                                                  |
  20. |                      |      "remote_group_id": null,                                      |
  21. |                      |      "direction": "egress",                                        |
  22. |                      |      "protocol": "112",                                            |
  23. |                      |      "description": "",                                            |
  24. |                      |      "ethertype": "IPv4",                                          |
  25. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  26. |                      |      "port_range_max": null,                                       |
  27. |                      |      "updated_at": "2018-08-08T01:55:44Z",                         |
  28. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  29. |                      |      "port_range_min": null,                                       |
  30. |                      |      "revision_number": 1,                                         |
  31. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  32. |                      |      "created_at": "2018-08-08T01:55:44Z",                         |
  33. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  34. |                      |      "id": "00421a6a-5f69-4c20-83df-e188e31d52f2"                  |
  35. |                      | }                                                                  |
  36. |                      | {                                                                  |
  37. |                      |      "remote_group_id": null,                                      |
  38. |                      |      "direction": "ingress",                                       |
  39. |                      |      "protocol": "112",                                            |
  40. |                      |      "description": "",                                            |
  41. |                      |      "ethertype": "IPv4",                                          |
  42. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  43. |                      |      "port_range_max": null,                                       |
  44. |                      |      "updated_at": "2018-08-08T01:55:34Z",                         |
  45. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  46. |                      |      "port_range_min": null,                                       |
  47. |                      |      "revision_number": 1,                                         |
  48. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  49. |                      |      "created_at": "2018-08-08T01:55:34Z",                         |
  50. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  51. |                      |      "id": "05e5b4ec-f73c-4e6d-84ca-07f0b676d096"                  |
  52. |                      | }                                                                  |
  53. |                      | {                                                                  |
  54. |                      |      "remote_group_id": null,                                      |
  55. |                      |      "direction": "ingress",                                       |
  56. |                      |      "protocol": "tcp",                                            |
  57. |                      |      "description": "",                                            |
  58. |                      |      "ethertype": "IPv4",                                          |
  59. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  60. |                      |      "port_range_max": 65535,                                      |
  61. |                      |      "updated_at": "2018-08-08T01:56:25Z",                         |
  62. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  63. |                      |      "port_range_min": 1,                                          |
  64. |                      |      "revision_number": 1,                                         |
  65. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  66. |                      |      "created_at": "2018-08-08T01:56:25Z",                         |
  67. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  68. |                      |      "id": "144ad77f-77a1-4ee0-9ab9-d51a478f46e8"                  |
  69. |                      | }                                                                  |
  70. |                      | {                                                                  |
  71. |                      |      "remote_group_id": null,                                      |
  72. |                      |      "direction": "egress",                                        |
  73. |                      |      "protocol": "tcp",                                            |
  74. |                      |      "description": "",                                            |
  75. |                      |      "ethertype": "IPv4",                                          |
  76. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  77. |                      |      "port_range_max": 65535,                                      |
  78. |                      |      "updated_at": "2018-08-08T01:56:39Z",                         |
  79. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  80. |                      |      "port_range_min": 1,                                          |
  81. |                      |      "revision_number": 1,                                         |
  82. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  83. |                      |      "created_at": "2018-08-08T01:56:39Z",                         |
  84. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  85. |                      |      "id": "20ee05da-f30e-40d1-a355-b837050c786c"                  |
  86. |                      | }                                                                  |
  87. |                      | {                                                                  |
  88. |                      |      "remote_group_id": null,                                      |
  89. |                      |      "direction": "egress",                                        |
  90. |                      |      "protocol": "udp",                                            |
  91. |                      |      "description": "",                                            |
  92. |                      |      "ethertype": "IPv4",                                          |
  93. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  94. |                      |      "port_range_max": 65535,                                      |
  95. |                      |      "updated_at": "2018-08-08T01:56:46Z",                         |
  96. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  97. |                      |      "port_range_min": 1,                                          |
  98. |                      |      "revision_number": 1,                                         |
  99. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  100. |                      |      "created_at": "2018-08-08T01:56:46Z",                         |
  101. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  102. |                      |      "id": "27c018cf-519c-42fe-aa06-18cfb157f073"                  |
  103. |                      | }                                                                  |
  104. |                      | {                                                                  |
  105. |                      |      "remote_group_id": null,                                      |
  106. |                      |      "direction": "ingress",                                       |
  107. |                      |      "protocol": "icmp",                                           |
  108. |                      |      "description": "",                                            |
  109. |                      |      "ethertype": "IPv4",                                          |
  110. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  111. |                      |      "port_range_max": null,                                       |
  112. |                      |      "updated_at": "2018-08-08T01:56:02Z",                         |
  113. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  114. |                      |      "port_range_min": null,                                       |
  115. |                      |      "revision_number": 1,                                         |
  116. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  117. |                      |      "created_at": "2018-08-08T01:56:02Z",                         |
  118. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  119. |                      |      "id": "766079fc-ff02-4ed7-915e-dd1c1cd87fb7"                  |
  120. |                      | }                                                                  |
  121. |                      | {                                                                  |
  122. |                      |      "remote_group_id": null,                                      |
  123. |                      |      "direction": "ingress",                                       |
  124. |                      |      "protocol": "udp",                                            |
  125. |                      |      "description": "",                                            |
  126. |                      |      "ethertype": "IPv4",                                          |
  127. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  128. |                      |      "port_range_max": 65535,                                      |
  129. |                      |      "updated_at": "2018-08-08T01:56:32Z",                         |
  130. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  131. |                      |      "port_range_min": 1,                                          |
  132. |                      |      "revision_number": 1,                                         |
  133. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  134. |                      |      "created_at": "2018-08-08T01:56:32Z",                         |
  135. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  136. |                      |      "id": "88417b5c-f391-4f76-acf1-fc7a96003124"                  |
  137. |                      | }                                                                  |
  138. |                      | {                                                                  |
  139. |                      |      "remote_group_id": null,                                      |
  140. |                      |      "direction": "egress",                                        |
  141. |                      |      "protocol": null,                                             |
  142. |                      |      "description": null,                                          |
  143. |                      |      "ethertype": "IPv4",                                          |
  144. |                      |      "remote_ip_prefix": null,                                     |
  145. |                      |      "port_range_max": null,                                       |
  146. |                      |      "updated_at": "2018-08-08T01:54:56Z",                         |
  147. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  148. |                      |      "port_range_min": null,                                       |
  149. |                      |      "revision_number": 1,                                         |
  150. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  151. |                      |      "created_at": "2018-08-08T01:54:56Z",                         |
  152. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  153. |                      |      "id": "e45a99b6-380d-40ff-8e35-205260ab8d74"                  |
  154. |                      | }                                                                  |
  155. |                      | {                                                                  |
  156. |                      |      "remote_group_id": null,                                      |
  157. |                      |      "direction": "egress",                                        |
  158. |                      |      "protocol": "icmp",                                           |
  159. |                      |      "description": "",                                            |
  160. |                      |      "ethertype": "IPv4",                                          |
  161. |                      |      "remote_ip_prefix": "0.0.0.0/0",                              |
  162. |                      |      "port_range_max": null,                                       |
  163. |                      |      "updated_at": "2018-08-08T01:56:11Z",                         |
  164. |                      |      "security_group_id": "7da53cec-e267-4cc7-847f-a63341da58a8",  |
  165. |                      |      "port_range_min": null,                                       |
  166. |                      |      "revision_number": 1,                                         |
  167. |                      |      "tenant_id": "22938c8796b74e6cbb4bba13c9bde822",              |
  168. |                      |      "created_at": "2018-08-08T01:56:11Z",                         |
  169. |                      |      "project_id": "22938c8796b74e6cbb4bba13c9bde822",             |
  170. |                      |      "id": "fd97f2db-eced-49c3-86b1-15dc6f6c3e5e"                  |
  171. |                      | }                                                                  |
  172. | tenant_id            | 22938c8796b74e6cbb4bba13c9bde822                                   |
  173. | updated_at           | 2018-08-08T01:56:46Z                                               |
  174. +----------------------+--------------------------------------------------------------------+
添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注