Gluster geo-replication performance tweaks

 How to manupulate the performance of gluster geo-rep?

Gluster geo-replication follows two methods to sync data from primary location to a secondary location. 

• rsync
• tarssh

tarsh can be prefered when there is large number of small files to be synced,

while rsync performs better with large files or combinations of different small and large files.

Let us see how we can tune in the performance of gluster geo-replication with respect to rsync, as it is the most common way used in syncing.

How do we analyse the performace here?

Following command comes in handy:

#gluster volume geo-replication <primary> <ip>::<secondary> config rsync-perfpormance true   

gsyncd log documents the output of rsync-performance on executing the above command.

We can also try various sync options: 

#gluster volume geo-replication <primary> <ip>::<secondary> config rsync-options <any rsync option>

Some rsync options can work the best on your data set, while some others can decrease the performance of your setup.

Here is how I tested various rsync options on my setup:

Test with compression:

# rsync -aR0 -z --inplace -e "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no" $src/* ipaddress:$dest/

/var/log/glusterfs/geo-replication/master_127.0.0.1_slave/gsyncd.log:18817:[2020-08-19 15:51:53.127716] I [resource(worker /d/backends/master1):1525:rsync] SSH: rsync performance [{data=Number of files: 3 (reg: 2, dir: 1), Number of regular files transferred: 2, Total file size: 8,742 bytes, Total transferred file size: 8,742 bytes, Literal data: 8,742 bytes, Matched data: 0 bytes, Total bytes sent: 3,982, Total bytes received: 54, sent 3,982 bytes  received 54 bytes  2,690.67 bytes/sec}]

Total bytes sent:  3,982 bytes

Total bytes received: 54

Speed: 2690.67 bytes/sec

Test without  compression:

# rsync -aR0 --inplace -e "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no" $src/* ipaddress:$dest/

[2020-08-20 06:29:15.749972] I [resource(worker /d/backends/master4):1525:rsync] SSH: rsync performance [{data=Number of files: 3 (reg: 2, dir: 1), Number of regular files transferred: 2, Total file size: 8,742 bytes, Total transferred file size: 8,742 bytes, Literal data: 8,742 bytes, Matched data: 0 bytes, Total bytes sent: 3,982, Total bytes received: 54, sent 3,982 bytes  received 54 bytes  8,072.00 bytes/sec}]

Total bytes sent:  3,982 bytes

Total bytes received: 54

Speed: 8,072.00 bytes/sec

Test with cipher aes128-ctr:

# rsync -aR0 --inplace -e "ssh -c aes128-ctr -oPasswordAuthentication=no -oStrictHostKeyChecking=no" $src/* ipaddress:$dest/

[2020-08-20 07:53:05.342032] I [master(worker /d/backends/master4):1513:crawl] _GMaster: slave's time [{stime=(1597909967, 0)}]

[2020-08-20 07:53:06.137345] I [resource(worker /d/backends/master4):1524:rsync] SSH: rsync performance [{data=Number of files: 3 (reg: 2, dir: 1), Number of regular files transferred: 2, Total file size: 8,742 bytes, Total transferred file size: 8,742 bytes, Literal data: 8,742 bytes, Matched data: 0 bytes, Total bytes sent: 9,195, Total bytes received: 54, sent 9,195 bytes  received 54 bytes  6,166.00 bytes/sec}]

Total bytes sent: 9,195

Total bytes received: 54

Speed: 6,166.00 bytes/sec

Test with compression and aes128-ctr:

# rsync -aR0 -z --inplace -e "ssh -c aes128-ctr -oPasswordAuthentication=no -oStrictHostKeyChecking=no" $src/* ipaddress:$dest/

[2020-08-21 11:00:17.971377] I [resource(worker /d/backends/master4):1522:rsync] SSH: rsync performance [{data=Number of files: 3 (reg: 2, dir: 1), Number of regular files transferred: 2, Total file size: 8,742 bytes, Total transferred file size: 8,742 bytes, Literal data: 8,742 bytes, Matched data: 0 bytes, Total bytes sent: 3,982, Total bytes received: 54, sent 3,982 bytes  received 54 bytes  8,072.00 bytes/sec}]

[2020-08-21 11:00:17.971586] I [master(worker /d/backends/master4):1997:syncjob] Syncer: Sync Time Taken [{job=3}, {num_files=2}, {return_code=0}, {duration=0.4362}]

Total bytes sent: 3,982

Total bytes received: 54

Speed:  8,072.00 bytes/sec

Go ahead and play with various different rsync options and check which works well for you.