- Sync function merge (master and slave functions are more more or less the same)
- Tree function merge (current and after tree functions are the same except for output filename and logging)
- Sync function merge (master and slave functions are the same, reduces code size and maintain effort)
- Tree function merge (current and after tree functions are the same except for output filename and logging, reduces code size and maintain effort)
- Tree functions execute piped commands (grep, awk) on master when launched on remote slave which can cause more bandwith usage
- Fast sync mode (without config file, directly via command line by specifying two directories)
FAR FUTURE IMPROVEMENTS
-----------------------
- Rethink of .osync_workdir/state/* files with PIDs, Host and Task Names to better identify multiple instances on the same fileset
- Improve Master / Slave schema to Multimaster schema
KNOWN ISSUES
------------
- If master and remote slave systems don't have rsync in the same path, execution may fail (RSYNC_PATH is always configured on master, even when executed on slave)
- On MSYS, osync does not propagate deletions
- None yet, need more testing on MSYS environment
RECENT CHANGES
--------------
- 02 Nov. 2013: Osync 0.99 RC2
- Minor improvement on operating system detection
- Improved RunLocalCommand execution hook
- Minor improvements on permission checks
- Made more portability improvements (mostly for FreeBSD, must be run with bash shell)
- Added local and remote operating system detection
- Added forced usage of MSYS find on remote MSYS hosts
A two way sync script based that adds script fault tolerance from obackup project along with multiple usefull options.
A two way sync script with fault tolerance, resuming and delete / conflict backups.
## About
Having created obackup script in order to make reliable quick backups, i searched for a nice tool to handle two (or more) way sync scenarios in a reliable way.
While unison handles these scenarios, it's pretty messy to configure, slow, won't handle ACLs and won't resume if something bad happened.
I searched for a nice tool to handle two (or more) way sync scenarios in a reliable way, easy to use and automate.
While unison handles these scenarios, it's pretty messy to configure, slow, won't handle ACLs and won't automatically resume if something bad happened.
Then i read about bitpocket, a nice script provided by Marcin Kulik (sickill) at https://github.com/sickill/bitpocket.git
Bitpocked inspired me to write my own implementation of a two way sync script, implementing features i wanted among:
- Fault tolerance with resume scenarios
- Email alerts
- Fault tolerance with resume scenarios
- Email alerts
- Logging facility
- Soft deletition and multiple backups handling
- Before / after command execution
- Time control
Osync uses a master / slave sync schema. It can sync local or remote directories. By definition, master replica should always be a local directory on the system osync runs on.
Osync uses a master / slave sync schema. It can sync local and local or local and remote directories. By definition, master replica should always be a local directory on the system osync runs on.
Also, osync uses pidlocks to prevent multiple concurrent sync processes on/to the same master / slave replica. Be sure a sync process is finished before launching next one.
You may launch concurrent sync processes on the same system but only for different master replicas.
Currently, it has been tested on CentOS 5, CentOS 6, Debian 6.0.7, Linux Mint 14 and Ubuntu 12.
Currently, it has been tested on CentOS 5, CentOS 6, Debian 6.0.7, Linux Mint 14, Ubuntu 12.
Osync also runs on FreeBSD and Windows MSYS environment, altough it is not fully tested yet.
## Installation
Osync developpment is still not finished. It's currently at beta stage. Please read CHANGELOG.md for a list of known bugs.
Keep in mind that Osync has been designed to not delete any data, but rather make backups or soft deletes.
Nevertheless, as we're still in beta stage, please make a backup of your data before using Osync.
Nevertheless, still consider making backups of your data before trying a sync tool.
First, grab a fresh copy of osync and make it executable:
$ git clone https://github.com/deajan/osync
$ chmod +x ./osync.sh
Osync needs to run with bash shell. Using any other shell will most probably result in lots of errors.
There is no need to intialize anything. You can begin sync with two already filled directories.
You only have to copy the sync.conf file to let's say your.conf and then edit it according to your needs.
You only have to customize the sync.conf file according to your needs.
Osync needs a pair of private / public RSA keys to perform remote SSH connections.
Also, using SUDO_EXEC option requires to configure /etc/sudoers file.
Documentation is being written, meanwhile you can check Obackup documentation at http://netpower.fr/projects/obackup/documentation.html for the two configurations points above.
Also, running sync as superuser requires to configure /etc/sudoers file.
Please read the documentation on author's site.
## Usage
Once you've setup a personalized sync.conf file, you may run osync with the following test run:
Once you've customized a sync.conf file, you may run osync with the following test run:
$ ./osync.sh /path/to/your.conf --dry
@ -61,8 +61,7 @@ Once you're confident about your fist runs, you may add osync as cron task with:
$ ./osync.sh /path/to/your.conf --silent
You may then find osync output in /var/log/osync-*.log
Also, you may always find detailed rsync command results at /tmp/osync_* if verbose switch wasn't specified.
You may then find osync output in /var/log/osync-*.log (or current directory if /var/log is not writable).
###### Osync - Rsync based two way sync engine with fault tolerance
###### (L) 2013 by Orsiris "Ozy" de Jong (www.netpower.fr)
#### Config file rev 1010201302
#### Config file rev 0211201302
## Sync job identification, any string you want, no spaces
SYNC_ID="sync_test"
## ---------- GENERAL OPTIONS
## Leaving this empty will create a logfile at /var/log/osync_version_SYNC_ID.log (or current directory if /var/log doesn't exist)
LOGFILE=""
## Sync job identification
SYNC_ID="sync_test"
## Directories to synchronize
## Directories to synchronize. Master must be on the system Osync runs on. Slave can be either on the same system, or on a remote one.
MASTER_SYNC_DIR="/home/git/osync/test/dir1"
SLAVE_SYNC_DIR="/home/git/osync/test/dir2"
## Create sync directories if they do not exist
CREATE_DIRS=no
## List of directories to exclude in sync on both sides (rsync patterns, wildcards work). Must be relative paths. List is separated by PATH SEPARATOR CHAR defined below (semicolon by default).
## Log file location. Leaving this empty will create a logfile at /var/log/osync_version_SYNC_ID.log (or current directory if /var/log doesn't exist)
LOGFILE=""
## List of directories to exclude from sync on both sides (rsync patterns, wildcards work).
## Paths are relative to sync dirs. List elements are separated by a semicolon.
RSYNC_EXCLUDE_PATTERN="tmp;archives"
## You might change this separator case in the unholy case that your filename may contain semicolons. Change it then to whatever unholy char you want.
## List elements separator char. You may set an alternative seperator char for your directories lists above.
PATH_SEPARATOR_CHAR=";"
## Generate an alert if master or slave have lass space than given value in KB.
## Generate an alert if master or slave replicas have less free space than given value in KB.
## ssh compression should be used unless your remote connection is good enough (LAN)
SSH_COMPRESSION=yes
## Check for connectivity to remote host before launching remote sync task. Be sure the hosts responds to ping. Failing to ping will stop sync.
REMOTE_HOST_PING=no
## Check for internet access by pinging one or more 3rd party hosts before remote sync task. Leave empty if you don't want this check to be be performed. Failing to ping will stop sync.
## If you use this function, you should set more than one 3rd party host, and be sure you can ping them.
## Be aware some DNS like opendns redirect false hostnames. Also, this adds an extra execution time of a bit less than a minute.
## Remote rsync executable path. Leave this empty in most cases
REMOTE_RSYNC_PATH=""
## Preserve ACLS. Make sure target FS can hold ACLs or you'll get loads of errors.
## ---------- MISC OPTIONS
## Preserve ACLS. Make sure source and target FS can manage same ACLs or you'll get loads of errors.
PRESERVE_ACL=no
## Preserve Xattr
## Preserve Xattr. Make sure source and target FS can manage same Xattrs or you'll get loads of errors.
PRESERVE_XATTR=no
## Let RSYNC compress file transfers. Do not use if you already enabled SSH compression.
## Let RSYNC compress file transfers. Do not use this if both master and slave replicas are on local system. Also, do not use this if you already enabled SSH compression.
RSYNC_COMPRESS=yes
## Maximum execution time (in seconds) for sync process. Soft exec time only generates warning. Hard exec time will generate warning and stop sync process.
## Maximum execution time (in seconds) for sync process. Soft exec time only generates a warning. Hard exec time will generate a warning and stop sync process.
SOFT_MAX_EXEC_TIME=7200
HARD_MAX_EXEC_TIME=10600
## If the same file exists on both sides, newer version will be used. If both files have the same timestamp but differ, CONFILCT_PREVALANCE sets winner
CONFLICT_PREVALANCE=master
## Keep a backup of a file if gets updated from remote side
## ---------- BACKUP AND TRASH OPTIONS
## Enabling this option will keep a backup of a file on the target replica if it gets updated from the source replica. Backups will be made to .osync_workdir/backups
CONFLICT_BACKUP=yes
## Keep multiple backups of a file if it gets updated from remote side. This can be very space consuming
## Keep multiple backup versions of the same file. Warning, This can be very space consuming.
CONFLICT_BACKUP_MULTIPLE=no
## Number of days to keep backups
## Osync will clean backup files after a given number of days. Setting this to 0 will disable cleaning and keep backups forever. Warning: This can be very space consuming.
CONFLICT_BACKUP_DAYS=30
## If the same file exists on both replicas, newer version will be synced. However, if both files have the same timestamp but differ, CONFILCT_PREVALANCE sets winner replica.
CONFLICT_PREVALANCE=master
## On deletition propagation to sync partner, keep a backup of deleted files on sync partner
## On deletition propagation to the target replica, a backup of the deleted files can be kept. Deletions will be kept in .osync_workdir/deleted
SOFT_DELETE=yes
## Number of days to keep deleted files
## Osync will clean deleted files after a given number of days. Setting this to 0 will disable cleaning and keep deleted files forever. Warning: This can be very space consuming.
SOFT_DELETE_DAYS=30
## ---------- RESUME OPTIONS
## Try to resume an aborted sync task
RESUME_SYNC=yes
## Number maximum resume tries before initating a new sync
## Number maximum resume tries before initating a fresh sync.
RESUME_TRY=2
## When a pidlock exists on slave that does not correspond to master's sync-id, force pidlock removal. Be carefull with this option if you have multiple masters.
## When a pidlock exists on slave replica that does not correspond to master's sync-id, force pidlock removal. Be carefull with this option if you have multiple masters.
FORCE_STRANGER_LOCK_RESUME=no
## ---------- ALERT OPTIONS
## List of alert mails separated by spaces
DESTINATION_MAILS="your@alert.tld"
## Windows only mail options (used by sendemail.exe)
## Windows (MSYS environment) only mail options (used by sendemail.exe)
SENDER_MAIL="alert@your.system"
SMTP_SERVER=smtp.your.isp.com
SMTP_USER=
SMTP_PASSWORD=
## Run local commands before and after sync task
## ---------- EXECUTION HOOKS
## Commands can will be run before and / or after sync process (remote execution will only happen if REMOTE_SYNC is set).
LOCAL_RUN_BEFORE_CMD=""
LOCAL_RUN_AFTER_CMD=""
## Run commands on remote slave befre and after sync task
REMOTE_RUN_BEFORE_CMD=""
REMOTE_RUN_AFTER_CMD=""
## Maximum execution time (in seconds) for commands before sync task. Commands get killed if not finished after MAX_EXC_TIME. Set this to 0 to disable killing.
## Max execution time of commands before they get force killed. Leave 0 if you don't wan't this to happen. Time is specified in seconds.
MAX_EXEC_TIME_PER_CMD_BEFORE=0
## Maximum execution time (in seconds) for commands after sync task. Commands get killed if not finished after MAX_EXEC_TIME. Set this to 0 to disable killing command.
MAX_EXEC_TIME_PER_CMD_AFTER=0
## Stops osync execution if one of the above commands fail
## Stops Osync execution if one of the above commands fail
deajan
11 years ago