Some errors got installing/removing packages after repairing a broken root file system:
dpkg: warning: files list file for package 'lsof' missing; assuming package has no files currently installed
I fixed these messages creating the package file list again:
dpkg-deb -c /var/cache/apt/archives/lsof_4.86+dfsg-1_amd64.deb | awk {'print $6'} | cut -f2- -d. | sed 's|^/$|/.|' | sed 's|/$||' > /var/lib/dpkg/info/lsof.list
And then:
# apt-get -d install lsof --reinstall
This is a simple post showing how to test if large MTU is working in your network using ping tool:
When jumbo frames are not enabled in all networking devs:
# ping -s 8000 -M do -c 5 172.16.64.75 PING 172.16.64.75 (172.16.64.75) 2000(2028) bytes of data. From 172.16.64.68 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 172.16.64.68 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 172.16.64.68 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 172.16.64.68 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 172.16.64.68 icmp_seq=1 Frag needed and DF set (mtu = 1500) --- 172.16.64.75 ping statistics --- 0 packets transmitted, 0 received, +5 errors
And with enabled jumbo frames:
# ping -s 8000 -M do -c 5 172.16.64.75 PING 172.16.64.75 (172.16.64.75) 8000(8028) bytes of data. 8008 bytes from 172.16.64.75: icmp_seq=1 ttl=64 time=0.461 ms 8008 bytes from 172.16.64.75: icmp_seq=2 ttl=64 time=0.360 ms 8008 bytes from 172.16.64.75: icmp_seq=3 ttl=64 time=0.402 ms 8008 bytes from 172.16.64.75: icmp_seq=4 ttl=64 time=0.410 ms 8008 bytes from 172.16.64.75: icmp_seq=5 ttl=64 time=0.347 ms --- 172.16.64.75 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3998ms rtt min/avg/max/mdev = 0.347/0.396/0.461/0.040 ms
I found these messages in my Ubuntu 14.04 server filling the syslog:
Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 121 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 122 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 123 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 124 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 125 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 126 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 133 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 134 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 135 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 136 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 137 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 138 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 146 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 147 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 148 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 149 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 150 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 151 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 158 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 159 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 160 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 161 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 162 affinity_hint subset empty Jul 4 14:49:14 h80-stack01 /usr/sbin/irqbalance: irq 163 affinity_hint subset empty
Seems to be a bug reported in Ubuntu, to avoid it, I followed these steps:
Edit file /etc/init/irqbalance.conf and change the line
exec /usr/sbin/irqbalance $DOPTIONS
for
exec /usr/sbin/irqbalance $OPTIONS $DOPTIONS
Edit /etc/default/irqbalance and add this line:
OPTIONS="--hintpolicy=ignore"
The file should look like this:
#Configuration for the irqbalance daemon #Should irqbalance be enabled? ENABLED="1" #Balance the IRQs only once? ONESHOT="0" OPTIONS="--hintpolicy=ignore"
Finally restart irqbalance service:
$ sudo service irqbalance restart
If the service restart does not work try rebooting machine.
I had to recover LVM metadata from one physical volume and here are the steps I followed. I am not sure why I got this situation, I’ve used LVM for a long time and this is the only one that I have to recover the metadata. The physical volume was backed by ISCSI lun.
Running pvdisplay command showed as no LVM volume:
$ sudo pvdisplay /dev/mapper/26a326a6471427270 No physical volume label read from /dev/mapper/26a326a6471427270 Failed to read physical volume "/dev/mapper/26a326a6471427270"
ISCSI session was OK:
$ sudo iscsiadm -m session tcp: [25] 172.16.68.51:3260,1 iqn.2012-12:backups tcp: [26] 172.16.68.151:3260,1 iqn.2012-12:backups tcp: [27] 172.16.64.10:3260,1 iqn.2012-12:backups tcp: [28] 172.16.66.10:3260,1 iqn.2012-12:backups
multipath detected the device paths OK:
26a326a6471427270 dm-11 SCST_BIO,j2jdqBrpZ2sYmXO3 size=13T features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 30:0:0:0 sdaf 65:240 active ready running |- 28:0:0:0 sdah 66:16 active ready running |- 29:0:0:0 sdag 66:0 active ready running `- 31:0:0:0 sdai 66:32 active ready running
$ sudo dmsetup ls --tree vgsatar6_C02-ovznhshare05 (253:52) └─26a326a6471427270 (253:11) ├─ (66:32) ├─ (66:0) ├─ (66:16) └─ (65:240)
$ sudo ls -l /dev/vgsatar6_C02/ovznhshare05 lrwxrwxrwx 1 root root 8 ene 21 17:06 /dev/vgsatar6_C02/ovznhshare05 -> ../dm-52
Also the device was mounted and running fine:
$ mount /dev/mapper/vgsatar6_C02-ovznhshare05 on /var/san/share05 type ext3 (rw,relatime,errors=continue,barrier=0,data=ordered)
But, I was sure that if the server was restarted the device was going to be lost and not mounted again.
The steps to recover the metadata were as follows:
umount the device:
$ sudo umount /var/san/share05
Open your last LVM backup for that VG in /etc/lvm/backup. In my case it was /etc/lvm/backup/vgsatar6_C02
Indentify the UUID for your physical volume in that file:
physical_volumes {
pv0 {
id = "1Babrx-VzKI-7Yav-9hGp-2KX2-DRgT-zdpAql"
device = "/dev/mapper/26a326a6471427270" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 27288010752 # 12,707 Terabytes
pe_start = 2048
pe_count = 3331055 # 12,707 Terabytes
}
}
In my case the Id was “1Babrx-VzKI-7Yav-9hGp-2KX2-DRgT-zdpAql”.
Recreate the physical volume again using the information saved in the backup:
$ sudo pvcreate --uuid "1Babrx-VzKI-7Yav-9hGp-2KX2-DRgT-zdpAql" --restorefile /etc/lvm/backup/vgsatar6_C02 /dev/mapper/26a326a6471427270
Restore volume group:
$sudo vgcfgrestore vgsatar6_C02
Activate logical volume and mount it again:
$ sudo lvchange -ay /dev/vgsatar6_C02/ovznhshare05 $ sudo mount /dev/vgsatar6_C02/ovznhshare05 /var/san/share05
And that’s all, I hope that will help someone.
For more info look at redhat documentation
I am testing the real bandwidth I can get with a 10G network connection. My server has an intel X540-AT2 network card with 2 10G interfaces.
The server is configured to use bonding in balance-alb mode, but in this test only one interface comes into play because the iperf client only gets a connection from one MAC address.
$ sudo cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: adaptive load balancing Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:f9:3d:54 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:f9:3d:55 Slave queue ID: 0
Frist run in the server iperf in server mode:
$ sudo iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------
An from one client run iperf connecting to the server IP
The first test is using a max transmision unit of 1500 bytes (default)):
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m ------------------------------------------------------------ Client connecting to 172.17.16.78, TCP port 5001 TCP window size: 92.9 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.16.79 port 52458 connected with 172.17.16.78 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 535 MBytes 4.49 Gbits/sec [ 3] 1.0- 2.0 sec 594 MBytes 4.98 Gbits/sec [ 3] 2.0- 3.0 sec 554 MBytes 4.64 Gbits/sec [ 3] 3.0- 4.0 sec 553 MBytes 4.64 Gbits/sec [ 3] 4.0- 5.0 sec 565 MBytes 4.74 Gbits/sec [ 3] 5.0- 6.0 sec 605 MBytes 5.07 Gbits/sec [ 3] 6.0- 7.0 sec 597 MBytes 5.01 Gbits/sec [ 3] 7.0- 8.0 sec 587 MBytes 4.92 Gbits/sec [ 3] 8.0- 9.0 sec 602 MBytes 5.05 Gbits/sec [ 3] 0.0-10.0 sec 5.67 GBytes 4.87 Gbits/sec [ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
With this test I can barely get around 5Gbits/sec. It seems too poor for a 10G network card.
In the next test I’ve changed the MTU to 9000 (jumbo frames) in all network devices (server, client and switch):
$ sudo iperf -c 172.17.16.78 -i1 -t 10 -m ------------------------------------------------------------ Client connecting to 172.17.16.78, TCP port 5001 TCP window size: 92.9 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.16.79 port 52101 connected with 172.17.16.78 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 971 MBytes 8.14 Gbits/sec [ 3] 1.0- 2.0 sec 1.09 GBytes 9.37 Gbits/sec [ 3] 2.0- 3.0 sec 1.05 GBytes 9.04 Gbits/sec [ 3] 3.0- 4.0 sec 1.06 GBytes 9.14 Gbits/sec [ 3] 4.0- 5.0 sec 1.10 GBytes 9.43 Gbits/sec [ 3] 5.0- 6.0 sec 1.11 GBytes 9.51 Gbits/sec [ 3] 6.0- 7.0 sec 1009 MBytes 8.46 Gbits/sec [ 3] 7.0- 8.0 sec 1.04 GBytes 8.94 Gbits/sec [ 3] 8.0- 9.0 sec 1.11 GBytes 9.56 Gbits/sec [ 3] 9.0-10.0 sec 1.07 GBytes 9.21 Gbits/sec [ 3] 0.0-10.0 sec 10.6 GBytes 9.08 Gbits/sec [ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
And now, it looks very different, I can saturate the network link.
To change MTU, I’ve changed it in the server and in the client, in all network devices: bond interface and its slaves. Also the switch must support it. You can change the device MTU easly with this command:
$ sudo ip link set dev eth0 mtu 1500