Virtual Interfaces don't failover in cluster

[Glen-afk]

Hello,

I've built a pair of Zevenet CE servers and configured them as a cluster. When I shutdown the Primary server the cluster IP correctly fails over to the Backup server. I can run a continuous ping and see a single drop and then it is reachable again. I've created some additional Virtual Interfaces with unique IPs I want to use for services in a LSLB l4xnat farm. When I shutdown the Primary server a ping of the IPs for these Virtual Interfaces drops and doesn't recover until the Primary server is powered on. What am I missing here that could be causing this?

Config example:
Server1
eth0 IP: 172.16.0.1

Server2
eth0 IP: 172.16.0.2

Cluster Virtual Interface
IP 172.16.0.3

Virtual Interfaces for services
IP 172.16.0.4
IP 172.16.0.5

[a-stoyanov]

Heya,

I've recently had to deploy a 2-node 5.11 CE cluster and ran into the same issue.
After some troubleshooting I found the zevenet-ce-cluster (1.3) package does not implement any gratuitous arp update procedures when the cluster master is failed over, which is really silly.
What this means is the arp tables of all on link neighbours including the next hop router are not updated when the virtual IPs are switched over. Ucarp does this natively only for the $cluster_ip. Any other virtual IPs are failed over but no garp is broadcast for the mac address change.

I added an extra one-liner to fix this in the cluster master startup script.

Create a new shell script:
nano /root/garp-update.sh

#!/bin/bash
#Print all bound IPs and execute arping garp (excludes link-local and loopback)
ip address | awk '/inet/ {print $2}' | grep -ve ^::1 -e ^127 -e ^fe80 | awk -F '/' '{print $1}' | while read line ; do arping -c 2 -U $line ; done

Make executable:
chmod +x /root/garp-update.sh

Edit the cluster startup script:
nano /usr/local/zevenet/app/ucarp/sbin/zevenet-ce-cluster-start

Add the extra 5th step to the startup script to execute the new shell script:

#!/usr/bin/perl

print "Executing Zevenet Cluster transtion to MASTER...\n";
#1 we wait for 1 second to the first replication
sleep (1);
#2 we start zevenet service
my $exec=system("/etc/init.d/zevenet start");
#3 update cluster status
my $exec=system("echo master > /etc/zevenet-ce-cluster.status");
#4 start zeninotify
my $exec=system("nohup /usr/local/zevenet/app/zeninotify/zeninotify.pl >/dev/null 2>&1 &");
#5 send gratuitous arp update for all bound IPs (excluding link-local and loopback)
my $exec=system("/root/garp-update.sh");

Zevenet devs should really add this fix in the next release.
@cano-devel @adanmarin-zevenet

[emiliocampos-zevenet]

Thanks for the code, we will review this for the following version. Have a nice day El lun, 23 ago 2021 a las 13:22, Angel Stoyanov ***@***.***>) escribió:

…

Heya, I've recently had to deploy a 2-node 5.11 CE cluster and ran into the same issue. After some troubleshooting I found the zevenet-ce-cluster (1.3) package *does not* implement any gratuitous arp update procedures when the cluster master is failed over, which is really silly. What this means is the arp tables of all on link neighbours including the next hop router are not updated when the virtual IPs are switched over. Ucarp does this natively only for the $cluster_ip. Any other virtual IPs are failed over but no garp is broadcast for the mac address change. I added an extra one-liner to fix this in the cluster startup script nano /usr/local/zevenet/app/ucarp/sbin/zevenet-ce-cluster-start Add the extra 5th step to the startup script so it looks like this #!/usr/bin/perl print "Executing Zevenet Cluster transtion to MASTER...\n"; #1 we wait for 1 second to the first replication sleep (1); #2 we start zevenet service my $exec=system("/etc/init.d/zevenet start"); #3 update cluster status my $exec=system("echo master > /etc/zevenet-ce-cluster.status"); #4 start zeninotify my $exec=system("nohup /usr/local/zevenet/app/zeninotify/zeninotify.pl >/dev/null 2>&1 &"); #5 send gratuitous arp update for all bound IPs (excluding link-local and loopback) my $exec=system("ip a | awk '/inet/ {print $2}' | grep -ve ^::1 -e ^127 -e ^fe80 | awk -F "/" '{print $1}' | while read line ; do arping -c 2 -U $line ; done"); Zevenet devs should really add this fix in the next release. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#99 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBQEPAMFXJDJCZGZCAZAWLT6IVQPANCNFSM4Y6H3AEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Emilio CamposZEVENET Teamwww.zevenet.com Review ZEVENET Product at Gartner Insights <https://gtnr.io/LfXtqnsSr> <https://www.linkedin.com/company/zevenet> <https://twitter.com/zevenet> <https://www.facebook.com/zevenet> <https://github.com/zevenet> [image: ZEVENET] <https://www.zevenet.com/signature/> DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

[dboc]

Hello,

@a-stoyanov I used your script, it worked in Zevenete 5.12 . Thanks

I noticed that Zevenet 5.12 only send ping with the physical IP to the gateway, you could see it here in Code:

zlb/usr/share/perl5/Zevenet/Net/Util.pm

Line 151 in 5d8acc1

sub sendGPing # ($pif)

sub sendGPing    # ($pif)
{
	&zenlog( __FILE__ . ":" . __LINE__ . ":" . ( caller ( 0 ) )[3] . "( @_ )",
			 "debug", "PROFILING" );
	my ( $pif ) = @_;
	my $if_conf = &getInterfaceConfig( $pif );
	my $gw      = $if_conf->{ gateway };

	if ( $gw )
	{
		my $ping_bin = &getGlobalConfiguration( 'ping_bin' );
		my $pingc    = &getGlobalConfiguration( 'pingc' );
		my $ping_cmd = "$ping_bin -c $pingc -I $if_conf->{addr} $gw";

		&zenlog( "Sending $pingc ping(s) to gateway $gw from $if_conf->{addr}",
				 "info", "NETWORK" );
		&logAndRunBG( "$ping_cmd" );
	}
}

I do not know if all the cases needs to send ping for all Virtual Interfaces, but in my case i needed it.