#LyX 2.2 created this file. For more info see http://www.lyx.org/ \lyxformat 508 \begin_document \begin_header \save_transient_properties true \origin unavailable \textclass article \use_default_options true \maintain_unincluded_children false \language english \language_package default \inputencoding utf8x \fontencoding global \font_roman "default" "default" \font_sans "default" "default" \font_typewriter "default" "default" \font_math "auto" "auto" \font_default_family default \use_non_tex_fonts true \font_sc false \font_osf false \font_sf_scale 100 100 \font_tt_scale 100 100 \graphics default \default_output_format default \output_sync 0 \bibtex_command default \index_command default \paperfontsize default \spacing single \use_hyperref true \pdf_title "Osync Configuration guide" \pdf_author "Orsiris " \pdf_bookmarks true \pdf_bookmarksnumbered false \pdf_bookmarksopen false \pdf_bookmarksopenlevel 1 \pdf_breaklinks false \pdf_pdfborder true \pdf_colorlinks false \pdf_backref section \pdf_pdfusetitle true \papersize a4paper \use_geometry true \use_package amsmath 1 \use_package amssymb 1 \use_package cancel 0 \use_package esint 1 \use_package mathdots 1 \use_package mathtools 0 \use_package mhchem 1 \use_package stackrel 0 \use_package stmaryrd 0 \use_package undertilde 0 \cite_engine basic \cite_engine_type default \biblio_style plain \use_bibtopic false \use_indices false \paperorientation portrait \suppress_date false \justification true \use_refstyle 1 \boxbgcolor #d0d0d0 \index Index \shortcut idx \color #008000 \end_index \leftmargin 2cm \topmargin 2cm \rightmargin 2cm \bottommargin 2cm \headheight 1cm \headsep 1cm \footskip 1cm \secnumdepth 3 \tocdepth 3 \paragraph_separation indent \paragraph_indentation 2em \quotes_language swedish \papercolumns 1 \papersides 1 \paperpagestyle default \listings_params "backgroundcolor={\color{white}},basicstyle={\ttfamily},breaklines=true,frame=single" \bullet 0 0 6 -1 \tracking_changes false \output_changes false \html_math_output 0 \html_css_as_file 0 \html_be_strict false \end_header \begin_body \begin_layout Title osync v1.2 Documentation \end_layout \begin_layout Author (C) 2013-2017 by Orsiris de Jong \end_layout \begin_layout Date 25 March 2017 \end_layout \begin_layout Standard \begin_inset CommandInset href LatexCommand href name "http://www.netpower.fr/osync" target "http://www.netpower.fr/osync" \end_inset \end_layout \begin_layout Standard \begin_inset CommandInset line LatexCommand rule offset "0.5ex" width "100col%" height "1pt" \end_inset \end_layout \begin_layout Standard \begin_inset CommandInset toc LatexCommand tableofcontents \end_inset \end_layout \begin_layout Section Introduction \end_layout \begin_layout Subsection Quickstart guide \end_layout \begin_layout Standard osync is a command line two way synchronization tool for Linux / BSD / Android / Busybox / MacOSX and Windows, with an emphasis on reliability and automation on low bandwidth links. \end_layout \begin_layout Standard For those who think TL;DR, a quickstart guide can be found in the README.md file. \end_layout \begin_layout Subsection Basic synchronization problems \end_layout \begin_layout Standard Synchronization is usually found in two flavors, bloc level sync and file level sync. While whole bloc level synchronization is generally a good way of doing, it's also very greedy in network ressources and is not easy to setup. That's where file level sync comes in handy, where only some directories & files need to be synced. \end_layout \begin_layout Standard Imagine you're syncing two remote offices of a same company. If you're syncing a user's home directory or it's roaming profile as a night task, the next day, the user will find it's roaming profile up to date at the remote office. \end_layout \begin_layout Standard But what would happen if two users work on the same file in a public folder, at the same time, on both offices ? Some sync software would stop sync and ask what to do. Others might simply deleted the oldest version of the file, even if it was modified on both sides. \end_layout \begin_layout Standard Also, what would happen if a user uploads a lot of data ? If the link between both offices cannot handle enough data transfer in a given time, any other sync task won't be run, and the sync would continue during the day, when bandwidth is necessary elsewhere. \end_layout \begin_layout Standard What would happen if a power fault occurs while synchronization is going on ? \end_layout \begin_layout Subsection What osync can do \end_layout \begin_layout Subsubsection Making synchronization reliable \end_layout \begin_layout Standard osync is designed to synchronize two folders on both local and / or remote systems. \end_layout \begin_layout Standard It is time controlled, which means you can decide how much time it should spend on a sync task before stopping it and launching the next one. \end_layout \begin_layout Standard It's designed to resume failed or stopped sync tasks in order to gain as much time as possible, or totally restart the sync task if resuming fails. \end_layout \begin_layout Standard It can keep multiple versions of a file in case of conflicts. \end_layout \begin_layout Standard It handles soft deletion. If a user deletes a file on replica A, it will move that file on replica B to the \begin_inset Quotes sld \end_inset .osync_workdir/deleted \begin_inset Quotes srd \end_inset folder. \end_layout \begin_layout Standard It will automatically clean old files (soft deleted and conflict backups) after a defined amount of days. \end_layout \begin_layout Standard It will perform various checks before launching a synchronization, like checking for free disk space against a defined minimum value. \end_layout \begin_layout Subsubsection Making a sysadmin's life easier \end_layout \begin_layout Standard osync is also desgined to ease synchronization setups. \end_layout \begin_layout Standard It will trigger an email alert including the whole sync process execution log if a warning / error is found. \end_layout \begin_layout Standard Pre-processing and post-processing commands can be launched on local and / or remote systems, which may be useful to launch snapshot software, flush or standby virtual machines, etc). \end_layout \begin_layout Standard Multiple concurrent instances of osync can be run as long as they don't sync the same replicas at the same time. \end_layout \begin_layout Standard A batch processing script is included (osync-batch.sh) to launch sequential sync tasks. Failed sync tasks are rerun if every other task has completed and there's still some time left in a given timespan. \end_layout \begin_layout Standard osync can use rsync or ssh tunnel compression to gain bandwidth. Bandwidth can also be limited for slow link sharing. \end_layout \begin_layout Standard It can be run in quicksync mode for the impatient (nothing to configure except the replica paths), or with a full blown config file. \end_layout \begin_layout Standard You may run osync manually, schedule it with cron, or have it monitor a directory as a daemon and launch sync tasks on file modifications. \end_layout \begin_layout Standard osync has been deeply tested on RHEL / CentOS 5, 6 & 7, Fedora 23, 24 & 25, Debian 6, 7, & 8, Linux Mint 14, 17 & 18, FreeBSD 8.3, 10.3 & 11, pfSense 2.x, Mac OS X, Android using termux, Windows 10 bash, and msys / msys2 & cygwin environments. \end_layout \begin_layout Subsubsection What osync cannot do \end_layout \begin_layout Standard Contrary to most sync tools, osync is stateful, which means that it doesn't need an agent installed on computers it synchronizes to. This makes it gain some degree of fault tolerance with WAN links. \end_layout \begin_layout Standard But this also has some drawbacks, there is a cornercase when a user deletes a file while the synchronisation is happening, the file won't get deleted on the other side because the sync is actually updating the directory states at the same time they get changed. Even if this happens only on very big deployments in some rare occasions because of the time osync takes to sync, there is still a chance that your deletion wouldn't make it. Generally speaking, the easiest way is to setup osync as a crontask and launch it by night in order to avoid any possible \end_layout \begin_layout Standard The good point is, osync is built from ground up to keep your data safe, and there's no chance it will delete anything you didn't want. \end_layout \begin_layout Standard osync is a simple bash script that relies on other tools like rsync. \end_layout \begin_layout Standard Hence, it has some advantages and disavantages: \end_layout \begin_layout Standard Advantages: \end_layout \begin_layout Standard - Easily customisable \end_layout \begin_layout Standard - Fast \end_layout \begin_layout Standard - Agentless \end_layout \begin_layout Standard Disavantages: \end_layout \begin_layout Standard - There's no way to detect file moves. If you move a directory on replica A, osync will soft delete the directory on replica B and copy the new directory from replica A. \end_layout \begin_layout Standard - There's no multi-master replication in osync V1. Hence, if you want to sync replicas A, B, and C using osync, you'll have to use one of the following schemas: \end_layout \begin_layout Paragraph Replicate 3-way with A as master \end_layout \begin_layout Standard Run the following tasks sequentially where A is the initiatior: \end_layout \begin_layout Standard Replicate A & B \end_layout \begin_layout Standard Replicate A & C \end_layout \begin_layout Standard Replicate A & B \end_layout \begin_layout Paragraph Replicate 3-way with A & B as masters \end_layout \begin_layout Standard Run the following tasks from each system. Be sure they won't run concurrently (osync will detect that another replication is still running, and abort the current one): \end_layout \begin_layout Standard Replicate A & B where A is the initiator \end_layout \begin_layout Standard Replicate B & C where B is the initiator \end_layout \begin_layout Standard Replicate C & A where C is the initiator \end_layout \begin_layout Subsection Why use osync \end_layout \begin_layout Standard There are a lot of file sync tools out there, some probably better than osync depending on the use case. \end_layout \begin_layout Standard osync has been basically written to be low bandwidth friendly, with resume options for unstable internet links (but it will also do great on a fiber link :)). \end_layout \begin_layout Standard It has also been written to be a setup and forget tool, without any user interaction like manual conflict resolution. \end_layout \begin_layout Standard osync also is one of the few tools that \series bold support ACL synchronization \series default . \end_layout \begin_layout Standard At last, osync consumes very little RAM and CPU ressources and is suitable from lower en hardware up to serveres. \end_layout \begin_layout Subsection How osync tries to resolve sync issues \end_layout \begin_layout Standard Let's get back to the example above, where osync is used to sync two remote offices with users' home directories. \end_layout \begin_layout Standard Now imagine a user uploaded 100GB of data, and the WAN link between local and remote systems can only handle 6GB/hour of data transfer. \end_layout \begin_layout Standard Now if osync is scheduled every night at 10:00 pm, and it's configured to run for maximum 10 hours, it would stop at 6am, after having transferred 60GB. \end_layout \begin_layout Standard Then, on the next day, it would transfer the remaining 40GB from 10:00 pm to about 3:30am. \end_layout \begin_layout Standard Also, if you run sequential instances of osync (with osync-batch), one per user directory for example, you can decide how much time osync should spend per user. So if a user uploads too much data, and osync cannot finish the synchronization task for that user directory in a given timespan, it will stop that sync task and run next user synchronization task so every user sync task gets run, regardless of the amount of data. If there's time left, osync-batch reprograms the user sync task that has been stopped. \end_layout \begin_layout Subsection Naming in this document \end_layout \begin_layout Standard osync's goal is to synchronize two directories, that can be hosted on the same computer or two different ones. \end_layout \begin_layout Standard The computer that runs osync must have at least one of these two directories mounted, and will be called the \emph on local system. \end_layout \begin_layout Standard The first directory to sync on the local system is called the \emph on initiator replica \emph default . \end_layout \begin_layout Standard The other directory, called the \emph on target replica \emph default can be hosted on the \emph on local system \emph default , or on another computer which will be called the \emph on remote system \emph default . In that case, the \emph on local system \emph default will connect to the \emph on remote system \emph default through an ssh tunnel and synchronize both \emph on initiator \emph default and \emph on target replicas \emph default . \end_layout \begin_layout Standard Any osync configuration file that gets executed is called an \emph on osync task \emph default . \end_layout \begin_layout Standard In the following examples, a special user called \emph on syncuser \emph default is used for osync. \end_layout \begin_layout Subsection How osync solves sync conflicts \end_layout \begin_layout Standard Conflict resolution is done automatically. When a file is modified on both replicas, osync compares the timestamps on both replicas and keeps the most recent file. \end_layout \begin_layout Standard In the highly uncommon case where both files have the exact same timestamp, a configuration value called CONFLICT_PREVALANCE choses the which replica's file will be kept. By default, Initiator is chosen, unless specified otherwise in the config file. \end_layout \begin_layout Standard The file that isn't kept is copied to \begin_inset Quotes sld \end_inset .osync_wordir/backups \begin_inset Quotes srd \end_inset directory of the replica by default, with its full path relative to the replica, unless specified otherwise by the CONFLICT_BACKUP configuration value. \end_layout \begin_layout Standard After an amount of days set by CONFLICT_BACKUP_DAYS, which is 30 by default, the backed up file is deleted. \end_layout \begin_layout Standard If CONFLICT_BACKUP_MULTIPLE is set to YES (disabled by default), multiple versions of the backed up files are kept, with a timestamp suffix like \begin_inset Quotes sld \end_inset 2016.12.31-12.00.01 \begin_inset Quotes srd \end_inset . \end_layout \begin_layout Section Prerequisites \end_layout \begin_layout Subsection General packages \end_layout \begin_layout Standard osync is a bash script that will work with \series bold bash version 3.2 or better \series default , but will not run with ksh / tsh / csh or other shells. Usually bash comes with most linux distributions. \end_layout \begin_layout Standard Here's a list of commands to install basic prerequisites on both local and remote machines. \end_layout \begin_layout Standard Note that openssh-clients are only needed on local machine if connecting remotely and sshpass is only needed if dealing with password files instead of RSA keys. \end_layout \begin_layout Standard The sudo package is only needed if you plan to run osync remotely with a standard user that gets sudo privileges. \end_layout \begin_layout Standard Also, inotify-tools / fswatch is only needed if you plan to run osync in daemon mode (wherever inotify-tools is available). \end_layout \begin_layout Itemize RHEL / CentOS \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout yum install rsync coreutils openssh-clients sshpass inotify-tools \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize Debian / Ubuntu / Mint \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout apt-get install rsync openssh-client sshpass inotify-tools \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize FreeBSD \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout pkg install bash rsync sshpass inotify-tools sudo \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize MacOS X \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout brew install rsync fswatch \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize MSYS (use management interface to install tools) \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout msys-base msys-coreutils-ext msys-rsync procps openssh-client \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize MSYS2 \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout pacman -S rsync procps openssh-client \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize Cygwin (use management interface to install tools) \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout rsync procps-ng openssh wget \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize termux (Android / Busybox) \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout apk --no-cache add bash rsync openssh-client \end_layout \end_inset \end_layout \end_deeper \begin_layout Standard Additionnaly, on termux, shebangs need to be modified to work with the system root. Example on how to get osync.sh working: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout termux-fix-shebang osync.sh \end_layout \end_inset \end_layout \begin_layout Subsubsection Mail alert prerequisites \end_layout \begin_layout Standard osync can send alert emails on warnings / errors. \end_layout \begin_layout Itemize On standard linux distros, BSD, MacOSX and Windows 10 bash, osync relies on your system-wide configured mail sending support in order to send alerts. It will try to use mail / mutt & sendmail. Please make sure that your system is configured to allow outgoing mail. You may check for osync mail in your mail logs (example: /var/log/maillog on RHEL / CentOS). \end_layout \begin_layout Itemize On android / busybox, you will need sendmail and openssl-tools if you plan to use mail encryption (TLS/SSL) \end_layout \begin_deeper \begin_layout Itemize \begin_inset listings inline false status open \begin_layout Plain Layout apt --no-cache add sendmail openssl-tools \end_layout \end_inset \end_layout \end_deeper \begin_layout Itemize On Cygwin / MSYS2, mail support is done via mailsend (https://github.com/muquit/m ailsend) or sendmail (http://caspian.dotconf.net/menu/Software/SendEmail/) which have to be executable path (%PATH% variable, or just place them in system32). \end_layout \begin_layout Subsubsection Daemon mode prerequisites \end_layout \begin_layout Standard osync can run in daermon mode in order to detect file modifications. This mode works on Linux / BSD and MacOSX. \end_layout \begin_layout Standard On Linux and BSD, please install the following package: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout inotify-tools \end_layout \end_inset \end_layout \begin_layout Standard On FreeBSD, you might want to increase kern.maxfiles tunable if you plan to use inotify-tools for applications that need to monitor activity of a lot of files. Modify /boot/loader.conf and add \begin_inset listings inline false status open \begin_layout Plain Layout kern.maxfiles="25000" \end_layout \end_inset \end_layout \begin_layout Standard On MacOS X, you will need to install fswatch with brew: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout brew install fswatch \end_layout \end_inset \end_layout \begin_layout Subsection Downloading osync \end_layout \begin_layout Standard osync can be downloaded on the author's site (stable version) or on github (stable or latest dev snapshot). \end_layout \begin_layout Standard Getting osync via author's site \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ wget http://netpower.fr/projects/osync/osync.v1.2.tar.gz \end_layout \begin_layout Plain Layout $ tar xvf osync.v1.2.tar.gz \end_layout \end_inset \end_layout \begin_layout Standard Getting osync via github (remove the -b \begin_inset Quotes sld \end_inset stable \begin_inset Quotes srd \end_inset if you want latest dev snapshot) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ git clone -b "stable" https://github.com/deajan/osync \end_layout \end_inset \end_layout \begin_layout Standard On Linux / BSD / Windows 10 bash, Once you downloaded osync, enter into the newly created folder and run the install script \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ bash ./install.sh \end_layout \end_inset \end_layout \begin_layout Standard This will copy osync to /usr/local/bin and create /etc/osync with a test sync.conf file. \end_layout \begin_layout Standard It will also copy daemon required files to /etc/init.d or /usr/lib/systemd/system and /etc/systemd/user depending on your distribution. \end_layout \begin_layout Standard On MacOS X, msys, Cygwin and termux (Android), you may directly use osync.sh script. \end_layout \begin_layout Standard Uninsalling is done by using \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ bash ./install.sh --remove \end_layout \end_inset \end_layout \begin_layout Standard There is also an RPM package available for Fedora & CentOS, which will install all binaries to /usr/bin instead of /usr/local/bin in order to enforce good practices. \end_layout \begin_layout Subparagraph Please note that when using the RPM packages, the binaries paths in this document shall be read as /usr/bin instead of /usr/local/bin. \end_layout \begin_layout Subsection File synchronization \end_layout \begin_layout Standard File sync tasks don't need any special configurations. You only have to worry about your sync user having the filesystem permissions to read / write on both replicas. \end_layout \begin_layout Standard A good way is to make your user member of the files' group that has full permissions. \end_layout \begin_layout Standard Another way to achieve this is using ACLs if your filesystem supports them. You can add the following permissions for user "syncuser" on directory "/home/web". Setting a default rule will add rights on new files. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # setfacl -dRm u:syncuser::r-x /home/web \end_layout \end_inset \end_layout \begin_layout Standard Be aware that ACLs are tricky and default unix permissions serve as mask for ACLs. \end_layout \begin_layout Standard Make always sure you can read /write to both replicas with your sync user: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # su syncuser \end_layout \begin_layout Plain Layout $ cat /initiator/replica/test.file \end_layout \begin_layout Plain Layout $ touch /initiator/replica/othertest.file \end_layout \end_inset \end_layout \begin_layout Standard Repeat that step for the target replica. \end_layout \begin_layout Subsection Time setup \end_layout \begin_layout Standard WARNING: osync's conflict resolution relies on timestamps, so it is very important for all systems osync runs / syncs to, to have a reliable and common timesource. \end_layout \begin_layout Standard Please consider setting up NTPD first before you plan to run osync. \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "subsec:Performing-superuser-backups" \end_inset Performing superuser sync \end_layout \begin_layout Standard osync can be run as superuser on the remote side, which should always be avoided by granting the read / write permissions to a dedicated sync user to both replicas. \end_layout \begin_layout Standard There are still some cases where osync needs to be run as superuser, especially when syncing system files. \end_layout \begin_layout Standard In those cases, osync can be run as dedicated sync user and ask for sudo permissions (if system permits it) for specific commands, or directly run as root (which isn't recommended). \end_layout \begin_layout Standard In order to be able to use the sudo command without having to enter a password, you’ll need to modify remote system to allow the following commands to be run as superuser. \end_layout \begin_layout Standard Use visudo to edit the sudoers file (or carefully edit /etc/sudoers yourself) and add the following \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout syncuser ALL= NOPASSWD:SETENV:/usr/bin/rsync,/usr/bin/bash \end_layout \end_inset \end_layout \begin_layout Standard Please note that the SETENV parameter is required so osync can send it's local variables as remote environment variables (equals sudo -E). \end_layout \begin_layout Standard You might check the right paths to your commands, especially on some FreeBSD environments where rsync might be in /usr/local/bin instead of /usr/bin (example to get path for rsync executable): \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ type rsync \end_layout \end_inset \end_layout \begin_layout Standard You'll also need to disable requiretty in /etc/sudoers by adding the following line: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout Defaults:syncuser !requiretty \end_layout \end_inset \end_layout \begin_layout Standard Once your standard sync user is granted to run what osync needs, you can enable sudo in osync's config file: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SUDO_EXEC=yes \end_layout \end_inset \end_layout \begin_layout Standard You should be aware that there is a risk with having rsync command run as superuser. A user who can run rsync command as superuser can upload any file he wants to the system, including a tweaked /etc/sudoers or /etc/passwd file. Please read chapter \begin_inset CommandInset ref LatexCommand ref reference "subsec:Enhancing-remote-backup" \end_inset to secure your installation. \end_layout \begin_layout Subsection Remote sync \end_layout \begin_layout Standard osync can perform local or remote synchronization tasks. For local sync, pelease refer to chapters \begin_inset CommandInset ref LatexCommand ref reference "sec:Running-Osync-in" \end_inset , \begin_inset CommandInset ref LatexCommand ref reference "sec:Running-Osync-with-config-file" \end_inset and \begin_inset CommandInset ref LatexCommand ref reference "sec:Running-Osync-as-daemon" \end_inset . \end_layout \begin_layout Standard Remote synchronization is done through an SSH tunnel. To be able to establish such a tunnel without having to enter a password, you’ll have to generate a pair of private and public RSA keys. \end_layout \begin_layout Standard The private part is kept by the computer that initiates the connection, the local system. The public part is kept on the remote system. \end_layout \begin_layout Standard The following steps will be required to generate a ssh key: \end_layout \begin_layout Standard Create a dedicated sync user and log in as that user on the local system to perform the following actions. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ssh-keygen -t rsa \end_layout \end_inset \end_layout \begin_layout Standard This should create two files named ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub \end_layout \begin_layout Standard You should also create a dedicated sync user on the remote system. \end_layout \begin_layout Standard Copy the public part of the RSA pair to the remote system with scp (replace 22 with your ssh port number if needed). \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ scp -p 22 ~/.ssh/id_rsa syncuser@remotesystem.tld:/home/syncuser/.ssh/authorized_ keys \end_layout \end_inset \end_layout \begin_layout Standard Make sure the file is only readable and owned by the syncuser on the remote system. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ chmod 600 /home/syncuser/.ssh/authorized_keys \end_layout \begin_layout Plain Layout $ chown syncuser:root /home/syncuser/.ssh/authorized_keys \end_layout \end_inset \end_layout \begin_layout Standard Now you should be able to login as "syncuser" on the remote system without any password. You can try to remotely login by entering the following on the local system: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ssh -p 22 syncuser@remotesystem.tld \end_layout \end_inset \end_layout \begin_layout Standard Be aware that only the user that generated the ssh key can remotely log in. \end_layout \begin_layout Standard You may optionnaly enhance remote login security by applying chapter \begin_inset CommandInset ref LatexCommand ref reference "subsec:Enhancing-remote-backup" \end_inset methods. \end_layout \begin_layout Subsection Mail transport agent \end_layout \begin_layout Standard You should make sure your system can send emails so osync can warn you if something bad happens. osync will use mutt or mail command. Please make sure you can send a test mail with at least one of the following commands run by your sync user: \end_layout \begin_layout Standard On Linux / BSD / MacOS / Windows 10 Bash \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ echo "your test message" | mutt -x -s "This is a test message" your@mail.tld \end_layout \begin_layout Plain Layout $ echo "your test message" | mail -s "This is a test message" your@mail.tld \end_layout \end_inset \end_layout \begin_layout Standard On termux (Android) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ echo -e"Subject:subject \backslash r \backslash nyour test message" |sendmail -f "sender@mail.tld" -S "smtp.yourisp.tld:25" -au"MySMTPUser" -ap"MySMTPPassword" "destination@mail.tld" \end_layout \end_inset \end_layout \begin_layout Standard Check your antispam if you don't get your message. If you still don't get your message, check your distributions documentation about the mail command. \end_layout \begin_layout Standard If you run on windows cygwin / msys environment, please make sure you can launch mailsend.exe / sendemail.exe by adding it to the %PATH% variable (found \begin_inset CommandInset href LatexCommand href name "here" target "http://github.com/muquit/mailsend" \end_inset and \begin_inset CommandInset href LatexCommand href name "here" target "http://caspian.dotconf.net/menu/Software/SendEmail/" \end_inset ). \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "subsec:Enhancing-remote-backup" \end_inset Enhancing remote system security \end_layout \begin_layout Standard You may want to secure a password-less ssh access by removing non necessary services offered by SSH. Edit the file ~/.ssh/authorized_keys created earlier on the remote system and add the following line in the beginning of the file: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty \end_layout \end_inset \end_layout \begin_layout Standard Also, we may want to prevent any host except of our initiator replica system to passwordless connect. Add the following line: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout from=*.my.initiator.replica.server.domain.tld \end_layout \end_inset \end_layout \begin_layout Standard Your authorized_keys file should look like this: \end_layout \begin_layout Standard \begin_inset listings lstparams "breaklines=true" inline false status open \begin_layout Plain Layout from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no- pty ssh-rsa yourkey== syncuser@host.tld \end_layout \end_inset \end_layout \begin_layout Subsection More security enhancing (not recommended unless you know what you are doing) \end_layout \begin_layout Standard We may also restrict the ssh session to only a couple of commands we'll need. osync comes with a script called \emph on ssh_filter.sh \emph default that will only allow execution of commands osync needs. Once again edit your authorized_keys file and add the following. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout command="/usr/local/bin/ssh_filter.sh SomeAlphaNumericToken9" \end_layout \end_inset \end_layout \begin_layout Standard Your file should then look like this: \end_layout \begin_layout Standard \begin_inset listings lstparams "breaklines=true,showstringspaces=false" inline false status open \begin_layout Plain Layout from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no- pty,command="/usr/local/bin/ssh_filter.sh SomeAlphaNumericToken9" ssh-rsa yourkey== syncuser@remotesystem.tld \end_layout \end_inset \end_layout \begin_layout Standard Copy then the script ssh_filter.sh to /usr/local/bin on the remote system. Don't forget to make it executable and make it owned by root \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # chmod 755 /usr/local/bin/ssh_filter.sh \end_layout \begin_layout Plain Layout # chown root:root /usr/local/bin/ssh_filter.sh \end_layout \end_inset \end_layout \begin_layout Standard Now, only the commands send by osync including a specific remote token may be executed via the ssh tunnel. You may enable / disable the usage of sudo command by editing the following value in the ssh_filter.sh script as well as in the osync config file: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SUDO_EXEC=yes \end_layout \end_inset \end_layout \begin_layout Subparagraph The remote token can be set in the config file and must match the one setup in authorized_keys file. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout _REMOTE_TOKEN=SomeAlphaNumericToken9 \end_layout \end_inset \end_layout \begin_layout Standard You may also set it on the fly in quicksync mode with the parameter \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ osync.sh --remote-token=SomeAlphaNumericToken9 \end_layout \end_inset \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "subsec:More-security-(or" \end_inset Security for the paranoid (not recommended unless you really know what you are doing) \end_layout \begin_layout Standard Executing rsync as superuser is a security risk. A way to prevent rsync usage allowing only a symlink to be executed. Thus, a attacker script using rsync would not work. This kind of security is called \begin_inset Quotes eld \end_inset security by obscurity \begin_inset Quotes erd \end_inset and should generally not be the only security process, but makes any attack harder. First, let's create a symlink to rsync called let's say o_rsync, on both local and remote systems. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # ln -s $(type rsync) $(dirname $(type rsync))/o_rsync \end_layout \end_inset \end_layout \begin_layout Standard Now edit the osync config file and change the following value: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_EXECUTABLE=o_rsync \end_layout \end_inset \end_layout \begin_layout Section Running osync \end_layout \begin_layout Subsection Basics \end_layout \begin_layout Standard Everytime osync is run, it runs with an INSTANCE_ID. \end_layout \begin_layout Standard This INSTANCE_ID helps identifying successive runs on replicas, and helps differentiate concurrent runs of the same initiator replica to different targets. \end_layout \begin_layout Standard By default, the INSTANCE_ID in quicksync mode is \begin_inset Quotes sld \end_inset quicksync_task \begin_inset Quotes srd \end_inset . This might be overriden via command-line or by using a configuration file. \end_layout \begin_layout Standard Keep in mind that two osync instances should never share the same INSTANCE_ID. \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "sec:Running-Osync-in" \end_inset Running osync in quicksync mode \end_layout \begin_layout Standard You just osync to sync two local dirs like this: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh --initiator=/path/to/dir1 --target=/path/to/dir2 \end_layout \end_inset \end_layout \begin_layout Standard You also may want to sync a remote directory. \end_layout \begin_layout Standard You may specify an alternate SSH port directly in the URI. When ommited, SSH port 22 is used. \end_layout \begin_layout Standard Also, if not set, the default RSA key will be read from ~/.ssh/id_rsa \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh --initiator=/path/to/dir1 --target=ssh://remoteuser@remotehost.com//pa th/to/dir2 \end_layout \begin_layout Plain Layout $ ./osync.sh --initiator=/path/to/dir2 --target=ssh://remoteuser@remotehost.com:22/ /path/to/dir2 --rsakey=/home/user/.ssh/other_key \end_layout \end_inset \end_layout \begin_layout Standard The following command-line options are quicksync specific: \end_layout \begin_layout Standard Use an RSA key to connect remotely \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --rsakey=/path/to/rsa.key \end_layout \end_inset \end_layout \begin_layout Standard Use a password file instead of an RSA key (needs sshpass utility installed) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --password-file=/path/to/passwd \end_layout \end_inset \end_layout \begin_layout Standard Set INSTANCE_ID \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --instance-id="some-instance-name" \end_layout \end_inset \end_layout \begin_layout Standard Skip deletions on initiator, target or both (valid values are: initiator or target or initiator,target) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --skip-deletion=initiator,target \end_layout \end_inset \end_layout \begin_layout Standard Send alert emails (only works with Linux / BSD / Windows 10 bash / MacOSX in quicksync mode) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --destination-mails="my@mail.tld myother@mail.tld" \end_layout \end_inset \end_layout \begin_layout Subsection Setting environment variables \end_layout \begin_layout Subsubsection General environment variables \end_layout \begin_layout Standard Quicksync mode actually can make use of most options available in config files. \end_layout \begin_layout Standard You may set them on the fly as environment variable like \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ PRESERVE_ACL=yes ./osync.sh --initiator=/path/to/dir1 --target=/path/to/dir2 \end_layout \end_inset \end_layout \begin_layout Standard A list of config values is found in appendix. \end_layout \begin_layout Subsubsection Special environment variables \end_layout \begin_layout Standard There is a set of special environment variables used mainly for debugging purposes. \end_layout \begin_layout Standard Valid ones are: \end_layout \begin_layout Standard Enable deubgging \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout _DEBUG=yes \end_layout \end_inset \end_layout \begin_layout Standard Enable paranoid debugging (only works on debug_osync.sh build or bootstrap.sh) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout _PARANOIA_DEBUG=yes \end_layout \end_inset \end_layout \begin_layout Standard Increase / decrease sleeping time between process checks when osync waits for a process. Default value is 0.05. (x is an integer or floating point number) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SLEEP_TIME=x \end_layout \end_inset \end_layout \begin_layout Standard Ignore unknown OS and execute regardless with default options. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout IGNORE_OS_TYPE=yes \end_layout \end_inset \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "sec:Running-Osync-with-config-file" \end_inset Running osync with a full blown configuration file \end_layout \begin_layout Standard Running osync with a configuration will do just the same as in quicksync mode, except that you have much more control of what's going on. \end_layout \begin_layout Standard A sample configuration file is called sync.conf and is included with osync. You may edit this file to fit your needs. Basically configuration files should go to /etc/osync. \end_layout \begin_layout Standard Every option of the configuration file is explained in the appendix. \end_layout \begin_layout Standard Once you've setup a file according to your needs, you may go for a test run. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh /etc/osync/my_sync.conf --dry --verbose \end_layout \end_inset \end_layout \begin_layout Standard osync should enumerate which changes will be done on both sides. \end_layout \begin_layout Standard If everything worked out right, you might process the actual sync process. \end_layout \begin_layout Standard A full configuration file specifies a maximum execution delay. Initial sync tasks can take a huge amount of time depending on bandwidth between replicas, in that case you might add parameter –no-maxtime to your first sync run so execution time won't be enforced. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh /etc/osync/my_sync.conf --no-maxtime \end_layout \end_inset \end_layout \begin_layout Standard Creating a regular sync scenario is quite simple as long as you don't schedule twice the same sync task in a shorter time span than your HARD_MAX_EXEC_TIME_TO TAL value. Just create a crontab entry and add parameter –silent so your local mailbox won't get filled up. Example, having a sync scheduled every hour in /etc/crontab \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout 00 * * * * syncuser /usr/local/bin/osync.sh /etc/osync/your_sync.conf --silent \end_layout \end_inset \end_layout \begin_layout Subparagraph Please note that this syntax works on RedHat / CentOS. On Debian you might need to remove the usename \begin_inset Quotes sld \end_inset ie syncuser \begin_inset Quotes srd \end_inset in order to make the crontab line work. \end_layout \begin_layout Standard You may find the sync log under /var/log/osync-your_sync.log or under the current directory if /var/log is not writable. \end_layout \begin_layout Subsection \begin_inset CommandInset label LatexCommand label name "sec:Running-Osync-as-daemon" \end_inset Running osync as deamon \end_layout \begin_layout Subsubsection Manually \end_layout \begin_layout Standard osync may also run in file monitor mode. In this mode, osync checks the initiator replica, and runs a synchronization as soon as there is file activity on initiator replica. With this mode, you do not need a schedule anymore. Be aware that only initiator replica is monitored, and target replica sync updates only occur when initiator replica modifications happen. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh /etc/osync/my_sync.conf --on-changes \end_layout \end_inset \end_layout \begin_layout Subsubsection As a system service \end_layout \begin_layout Standard If you plan to run osync on a regular basis in file monitor mode, you might consider installing it as a system service. \end_layout \begin_layout Standard From the directory you downloaded osync, run the install.sh script and enable the service. \end_layout \begin_layout Standard Remark: When exiting osync daemon, the process will continue to run for up to a minute to unlock replicas and termine sub processes. \end_layout \begin_layout Paragraph init.d service files \end_layout \begin_layout Standard For init.d systems, syntax is: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # service osync-srv start \end_layout \begin_layout Plain Layout # chkconfig osync-srv on \end_layout \end_inset \end_layout \begin_layout Standard osync then scans for *.conf files in /etc/osync and will run an instance per configuration file. \end_layout \begin_layout Standard Service control just works like with standard system services. \end_layout \begin_layout Paragraph systemd service files \end_layout \begin_layout Standard For systemd systems, syntax is: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # systemctl start osync-srv@config_file \end_layout \begin_layout Plain Layout # systemctl enable osync-srv@config_file \end_layout \end_inset \end_layout \begin_layout Standard With systemd, every config file found in /etc/osync can be controlled as a separate service. \end_layout \begin_layout Standard Note that on FreeBSD, rsync exclusion patterns won't work because if inotifywait 's limited –exclude implementation that can only make use of one exclude parameter. \end_layout \begin_layout Subsection Modifying osync behavior \end_layout \begin_layout Subsubsection Output / log behavior \end_layout \begin_layout Standard On most runs, you may only want to output file modifications or errors. You may achieve this (in quicksync / conf / daemon modes) by adding the following parameters: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ./osync.sh /path/to/sync.conf --errors-only --summary --no-prefix \end_layout \end_inset \end_layout \begin_layout Standard The following is a list of all command line switches that work with all three modes: \end_layout \begin_layout Standard Will run a simulation only \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --dry \end_layout \end_inset \end_layout \begin_layout Standard Silents osync totally, to be used in a cron schedule \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --silent \end_layout \end_inset \end_layout \begin_layout Standard Show a summary of updated / deleted files \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --summary \end_layout \end_inset \end_layout \begin_layout Standard Suppress non error state messages \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --errors-only \end_layout \end_inset \end_layout \begin_layout Standard More detailled output (debug) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --verbose \end_layout \end_inset \end_layout \begin_layout Standard Suppress log time prefix \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --no-prefix \end_layout \end_inset \end_layout \begin_layout Standard Add rsync stats to output \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --stats \end_layout \end_inset \end_layout \begin_layout Subsubsection Execution behavior \end_layout \begin_layout Standard Disable HARD_MAX_EXEC_TIME checks, so that a synchronization can take as long as it wants. This is used to perfom initial sync operations on big sets of files \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --no-maxtime \end_layout \end_inset \end_layout \begin_layout Standard Override any existing active or dead locks on replicas. Be careful if there is another running sync on any of both replicas \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --force-unlock \end_layout \end_inset \end_layout \begin_layout Standard Invoke daemon like mode. Shouldn't actually be used unless for debugging. Use daemon service files instead. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --on-changes \end_layout \end_inset \end_layout \begin_layout Standard Show help \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --help \end_layout \end_inset \end_layout \begin_layout Subsection Running osync batch \end_layout \begin_layout Standard If you have multiple configuration files in /etc/osync that you would like to run sequentially, and re-run failed sync tasks, osync comes with a tool called osync-batch. \end_layout \begin_layout Standard It will execute all osync conf files in a row, in a given timespan. \end_layout \begin_layout Standard osync-batch passes all its parameters to osync itself, except for the following which modifiy batch behavior itself: \end_layout \begin_layout Standard Path to directory containing osync configuration files \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --path=/etc/osync \end_layout \end_inset \end_layout \begin_layout Standard Maximum number of runs for failing synchronisation tasks (x is an integer). Defaults to 3. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --max-runs=x \end_layout \end_inset \end_layout \begin_layout Standard Maximum execution time for whole batch process in seconds (x is an integer). Defaults to 36000. Setting to 0 disables max execution time check. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout --max-exec-time=x \end_layout \end_inset \end_layout \begin_layout Standard You may program a cron task for osync-batch.sh like \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout 00 * * * * syncuser /usr/local/bin/osync-batch.sh --path=/etc/osync --silent \end_layout \end_inset \end_layout \begin_layout Section System & FS Caveats \end_layout \begin_layout Standard Depending on which configuration osync runs, not all functions may work. \end_layout \begin_layout Standard Especially, ACL and entended attributes preservation won't work on MacOSX, Cygwin, MSYS, Android and BusyBox local or remote environments. \end_layout \begin_layout Standard In order to work on most systems, ACL preservation and extended system attribute preservarion has been disabled in quicksync mode. \end_layout \begin_layout Standard To enable them, please run osync with the following environment variables \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ PRESERVE_ACL=yes PRESERVE_XATTR=yes ./osync.sh \end_layout \end_inset \end_layout \begin_layout Standard Those settings are also valid if set in config files. \end_layout \begin_layout Standard Also, setting up a sync between different file systems may prevent certain functionnality to work (mainly ACL preservation, extended attribute preservatio n and symlink / hardlink support). \end_layout \begin_layout Standard At last, MSYS2 has symlink support disabled by default. It can be enabled by uncommenting MSYS=winsymlinks:nativestrict in msys2-shell.c md, but doesn't behave exactly like other implementations. \end_layout \begin_layout Section Configuration file appendix \end_layout \begin_layout Subsection Full list of configuration file parameters \end_layout \begin_layout Standard Set this to whatever you want to identify your sync task. Two different osync runs should never share the same INSTANCE_ID. This value also determines the log filename and appears in the warning / error mails. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout INSTANCE_ID=name_of_your_sync \end_layout \end_inset \end_layout \begin_layout Standard Initiator directory to sync (initiator replica), must be on the system you're running osync on. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout INITIATOR_SYNC_DIR="/some/path" \end_layout \end_inset \end_layout \begin_layout Standard Target directory to sync (target replica), can be on the same system you're running osync on or another remote system, reachable via an SSH tunnel. \end_layout \begin_layout Standard Target directory can be a SSH uri like \begin_inset Quotes sld \end_inset ssh://user@host.com:1234//some/other/path \begin_inset Quotes srd \end_inset where 1234 is an optional port, and the first slash is a separator, meaning that the full path is /some/other/path. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout TARGET_SYNC_DIR="/some/other/path" \end_layout \end_inset \end_layout \begin_layout Standard Location of the private RSA key. If left empty, the default path \begin_inset Quotes sld \end_inset ~/.ssh/id_rsa \begin_inset Quotes srd \end_inset will be used. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SSH_RSA_PRIVATE_KEY=~/.ssh/id_rsa \end_layout \end_inset \end_layout \begin_layout Standard Alternate auth method by using a password file. This needs sshpass to be installed. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SSH_PASSWORD_FILE=/home/syncuser/path/to/passwd \end_layout \end_inset \end_layout \begin_layout Standard When using ssh_filter security, you need to specify a remote token matching the one setup in remote authorized_keys file \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout _REMOTE_TOKEN=SomeAlphaNumericToken9 \end_layout \end_inset \end_layout \begin_layout Standard Tells osync to create initiator or target directories if they don't exist. Default is no. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout CREATE_DIRS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard By default, leaving this empty sets the log file to /var/log/osync_INSTANCE_ID.lo g or ./osync_INSTANCE_ID.log if /var/log is not writable. You might change this to specify a personalized log file. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout LOGFILE="" \end_layout \end_inset \end_layout \begin_layout Standard Generate an alert if initiator or target replicas have less space than the following given value in kilobytes. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout MINIMUM_SPACE=10240 \end_layout \end_inset \end_layout \begin_layout Standard Bandwidth limit in kilobytes / second. Leave this to zero to disable limitation. Note that bandwidth is enforced only when running file updates. All other sync steps don't enforce that setting as they generally need very little bandwidth. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout BANDWIDTH=0 \end_layout \end_inset \end_layout \begin_layout Standard Synchronization tasks may be executed as root if you enable the following parameter. See prerequisites in chapter \begin_inset CommandInset ref LatexCommand ref reference "subsec:Performing-superuser-backups" \end_inset . \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SUDO_EXEC=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Paranoia option. Don't change this unless you read chapter \begin_inset CommandInset ref LatexCommand ref reference "subsec:More-security-(or" \end_inset and understand what you are doing. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_EXECUTABLE=rsync \end_layout \end_inset \end_layout \begin_layout Standard Remote Rsync Executable path. Don't change this unless your remote rsync binary isn't in the execution path. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_REMOTE_PATH="" \end_layout \end_inset \end_layout \begin_layout Standard Rsync include / exclude order. If set to include, includes will be processed before excludes, and vice-versa. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_PATTERN_FIRST=include|exclude \end_layout \end_inset \end_layout \begin_layout Standard List of files / directories to include / exclude from both replicas (see rsync patterns for more explanations, wildcards won't work). \end_layout \begin_layout Standard Poaths are relative to both replicas. List is separated by PATH_SEPARATOR_CHAR defined below. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_INCLUDE_PATTERN="" \end_layout \begin_layout Plain Layout RSYNC_EXCLUDE_PATTERN="tmp;archives;somepath" \end_layout \end_inset \end_layout \begin_layout Standard File that contains the list of files /directories to include / exclude from both replicas (see rsync pattern files for more explanations). Leave this empty if you don't want to use an exclusion file. This file has to be in the same directory as the config file. Paths are relative to sync dirs. One element per line. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_INCLUDE_FROM="" \end_layout \begin_layout Plain Layout RSYNC_EXCLUDE_FROM="exclude.list" \end_layout \end_inset \end_layout \begin_layout Standard Path separator char for RSYNC_EXCLUDE_PATTERN, you might change this in the unholy case that your filenames contains semicolons. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PATH_SEPARATOR_CHAR=";" \end_layout \end_inset \end_layout \begin_layout Standard Enable / disable ssh compression. Leave this enabled unless your connection to remote system is high speed (LAN) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SSH_COMPRESSION=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Tell ssh to not check the remote computer ssh fingerprint. DANGER WILL ROBINSON ! This should generally lead to security issues. Only enable this if you know exactly what you are doing. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SSH_IGNORE_KNOWN_HOSTS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Ping remote host before launching synchronization. Be sure the host is responding to ping. Failing to ping will skip current task. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout REMOTE_HOST_PING=yes|no \end_layout \end_inset \end_layout \begin_layout Itemize Check for internet access by pinging one or more hosts before launching remote sync task. Leave this empty do disable the checks. Failed checks won't abort current task, just trigger a warning. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout REMOTE_3RD_PARTY_HOST="www.kernel.org www.google.fr" \end_layout \end_inset \end_layout \begin_layout Itemize Misc settings \end_layout \begin_layout Standard Optional arguments to pass to rsync. Do not use already managed parameters by rsync (-r -l -p -t -g -o -D -E - u- i- n --executability -A -X -L -K -H -8 -zz –skip-compress –checksum –bwlimit –partial –partial-dir –no-whole-file –whole-file –backup –backup-dir –suffix --exclude --exclude-from --include --include-from --list-only --stats) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_OPTIONAL_ARGS="" \end_layout \end_inset \end_layout \begin_layout Standard Should osync preserve standard unix permissions \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_PERMISSIONS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Should osync preserve file owner \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_OWNER=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Should osync preserve group owner \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_GROUP=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Should osync preserve files executability status \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_EXECUTABILITY=yes|no \end_layout \end_inset Preserve ACLs. Please check that your filesystem supports ACLs and is mounted with it's support or rsync will get you loads of errors. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_ACL=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Preserve Xattr. The same applies as for ACLs \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_XATTR=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Transforms symlinks into referent files/dirs when syncing replicas. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout COPY_SYMLINKS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Treat symlinked dirs as dirs. CAUTION: This also follows symlinks outside of the replica root. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout KEEP_DIRLINKS=no \end_layout \end_inset \end_layout \begin_layout Standard Preserve hard links. Make sure source and target FS can manage hard links or you will lose them. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PRESERVE_HARDLINKS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Do a full checksum on files instead of comparing file sizes and modification times. Enabling this will make sync tasks longer. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout CHECKSUM=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Use rsync compression for file transfers. Leave this disabled unless your're not using SSH compression. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RSYNC_COMPRESS=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Maximum execution time (in seconds) for sync process. Soft value generates a warning only. Hard value generates a warning and stops sync task. \end_layout \begin_layout Standard You may set this to 0 to disable time checks. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SOFT_MAX_EXEC_TIME_FILE_TASK=7200 \end_layout \begin_layout Plain Layout HARD_MAX_EXEC_TIME_FILE_TASK=10600 \end_layout \end_inset \end_layout \begin_layout Standard Log a status message every KEEP_LOGGING seconds. Set this to zero to disable status messages. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout KEEP_LOGGING=1801 \end_layout \end_inset \end_layout \begin_layout Standard Minimum time (in seconds) in file monitor /daemon mode between modification detection and sync task in order to let copy operations finish. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout MIN_WAIT=60 \end_layout \end_inset \end_layout \begin_layout Standard Maximum time (in seconds) in file monitor / daemon mode. After this amount of time, a sync operation is forced. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout MAX_WAIT=7200 \end_layout \end_inset \end_layout \begin_layout Itemize Conflict and deletion option \end_layout \begin_layout Standard Enabling this option will keep a backup of a file on the target replica if it gets updated from the source replica. Backups will be made to .osync_workdir/backups \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout CONFLICT_BACKUP=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Keep multiple backup versions of the same file. Warning, This can be very space consuming. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout CONFLICT_BACKUP_MULTIPLE=yes|no \end_layout \end_inset \end_layout \begin_layout Standard osync will clean backup files after a given number of days. Setting this to 0 will disable cleaning and keep backups forever. Warning: This can be very space consuming. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout CONFLICT_BACKUP_DAYS=30 \end_layout \end_inset \end_layout \begin_layout Standard If the same file exists on both replicas, newer version will be synced. However, if both files have the same timestamp but differ, CONFLICT_PREVALANCE sets winner replica. \begin_inset listings inline false status open \begin_layout Plain Layout CONFLICT_PREVALANCE=initiator|target \end_layout \end_inset \end_layout \begin_layout Standard On deletition propagation to the target replica, a backup of the deleted files can be kept. Deletions will be kept in .osync_workdir/deleted \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SOFT_DELETE=yes|no \end_layout \end_inset \end_layout \begin_layout Standard osync will clean deleted files after a given number of days. Setting this to 0 will disable cleaning and keep deleted files forever. Warning: This can be very space consuming. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SOFT_DELETE_DAYS=30 \end_layout \end_inset \end_layout \begin_layout Standard Deletion propagation (not soft deletion) can be skipped on one or both replicas. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SKIP_DELETION=initiator|target|initiator,target \end_layout \end_inset \end_layout \begin_layout Itemize Resuming options \end_layout \begin_layout Standard Try to resume an aborted sync task \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RESUME_SYNC=yes|no \end_layout \end_inset Number maximum resume tries before initating a fresh sync. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RESUME_TRY=2 \end_layout \end_inset \end_layout \begin_layout Standard When a pidlock exists on target replica that does not correspond to initiator's instance-id, force pidlock removal. Be carefull with this option if you have multiple initiators. \end_layout \begin_layout Standard \begin_inset listings lstparams "breaklines=true" inline false status open \begin_layout Plain Layout FORCE_STRANGER_LOCK_RESUME=no \end_layout \end_inset \end_layout \begin_layout Standard Keep partial uploads that can be resumed on next run. This can be very useful if big files must get updated though slow links. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout PARTIAL=no \end_layout \end_inset \end_layout \begin_layout Standard Use rsync delta copy algorithm. Disabling this is useful to reduce CPU usage and raise bandwidth which might be valuable on local-local syncs or LAN syncs. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout DELTA_COPIES=yes \end_layout \end_inset \end_layout \begin_layout Itemize Alert Options \end_layout \begin_layout Standard Optional mail body encoding (using iconv). \end_layout \begin_layout Standard By default, all mails are sent in UTF-8 format without header (because of maximum compatibility of all platforms). You may specify an optional encoding here (like "ISO-8859-1" or whatever iconv can handle) \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout MAIL_BODY_CHARSET="" \end_layout \end_inset \end_layout \begin_layout Standard List of alert mails separated by spaces \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout DESTINATION_MAILS="your@alert.tld" \end_layout \end_inset \end_layout \begin_layout Standard MSYS / Cygwin / Android / Busybox only mail options \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout SENDER_MAIL="alert@your.system.tld" \end_layout \begin_layout Plain Layout SMTP_SERVER=smtp.your.isp.tld \end_layout \begin_layout Plain Layout SMTP_PORT=25 \end_layout \begin_layout Plain Layout SMTP_ENCRYPTION=tls|ssl|none \end_layout \begin_layout Plain Layout SMTP_USER=optional_smtp_user \end_layout \begin_layout Plain Layout SMTP_PASSWORD=optional_smtp_password \end_layout \end_inset \end_layout \begin_layout Itemize Execution hooks \end_layout \begin_layout Standard Commands can will be run before and / or after sync process (remote execution will only happen if REMOTE_SYNC is set). Multiple commands can be semicolon separated. \end_layout \begin_layout Standard Command(s) to run locally before sync process starts. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout LOCAL_RUN_BEFORE_CMD="" \end_layout \end_inset \end_layout \begin_layout Standard Command(s) to run locally if sync process finishes. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout LOCAL_RUN_AFTER_CMD="" \end_layout \end_inset \end_layout \begin_layout Standard Command(s) to run on remote system before sync process starts. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout REMOTE_RUN_BEFORE_CMD="" \end_layout \end_inset \end_layout \begin_layout Standard Command(s) to run on remote system if sync process finishes. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout REMOTE_RUN_AFTER_CMD="" \end_layout \end_inset \end_layout \begin_layout Standard Max execution time of commands before they get force killed. Leave 0 if you don't wan't this to happen. Time is specified in seconds. \begin_inset listings inline false status open \begin_layout Plain Layout MAX_EXEC_TIME_PER_CMD_BEFORE=0 \end_layout \begin_layout Plain Layout MAX_EXEC_TIME_PER_CMD_AFTER=0 \end_layout \end_inset \end_layout \begin_layout Standard Stops osync execution if one of the above commands fail \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout STOP_ON_CMD_ERROR=yes|no \end_layout \end_inset \end_layout \begin_layout Standard Run local and remote commands after a sync task even if it failed. \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout RUN_AFTER_CMD_ON_ERROR=yes|no \end_layout \end_inset \end_layout \begin_layout Subsection Upgrades from v1.0x and v1.1x \end_layout \begin_layout Standard Upgrades from osync v1.0x require to rewrite osync config file and state filenames. Master & slave replica are now called initiator and target replicas. SYNC-ID is now called INSTANCE-ID. \end_layout \begin_layout Standard There's an update script that does this job. \end_layout \begin_layout Standard For quicksync tasks: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ upgrade-v1.0x-v1.2x.sh --master=/path/to/master/replica --slave=/path/to/slave/re plica --sync-id=[INSTANCE_ID] \end_layout \end_inset \end_layout \begin_layout Standard For config file tasks: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ upgrade-v1.0x-v1.2x.sh /etc/osync/sync.conf \end_layout \end_inset \end_layout \begin_layout Standard Upgrades from osync v1.1x require to add osync configuration values into config file. \end_layout \begin_layout Standard Just use the upgrade script the same way you would to upgrade from v1.0x. \end_layout \begin_layout Section Troubleshooting \end_layout \begin_layout Standard osync has been tested successfully on multiple systems for a wide variety of sync plans. Please check the following steps before requesting help. \end_layout \begin_layout Standard If no error message is shown in console, please check the log files. \end_layout \begin_layout Subsection Log files \end_layout \begin_layout Standard Default [log_path] is /var/log, /home, ./ and /tmp depending if the directories are writable. \end_layout \begin_layout Standard Everything that is printed on screen via stdout or stderr is always included in log file which is named osync.[INSTANCE_ID].log \end_layout \begin_layout Standard Disabling switches like –errors-only or –silent may help diagnostic the problem. \end_layout \begin_layout Standard Log inludes a prefix (normally TIME) which indicates since how much seconds osync runs. \end_layout \begin_layout Standard When some commands get run remotely it will indicate how much seconds osync runs a specific remote task, beginning from 0 at every remote task. \end_layout \begin_layout Standard Prefix can be disabled via a command line argument. \end_layout \begin_layout Subsection Warnings, errors and alerts \end_layout \begin_layout Standard osync has two levels of alerts. \end_layout \begin_layout Standard Warnings are raised when something minor goes wrong, like checking for minimum available space or checking for internet connectivity, which will not affect the synchronization task directly. \end_layout \begin_layout Standard Errors are raised when something happens that could possibily stop sync from happening. \end_layout \begin_layout Standard Both are logged, and if email support is set, an alert email will be sent. \end_layout \begin_layout Subsection Local-local sync \end_layout \begin_layout Standard osync logs every of it's actions to [log_path]/osync-version.instance_id.log. \end_layout \begin_layout Standard Please check the log file if something went wrong. \end_layout \begin_layout Standard You might try running osync as root to check if your problem is filesystem permission related. \end_layout \begin_layout Standard You might add –verbose option to see what actually happens. \end_layout \begin_layout Standard Also, running osync with the following command will give the exact commands that actually happen: \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ DEBUG=yes /usr/local/bin/osync.sh /etc/osync/my_sync.conf --verbose \end_layout \end_inset \end_layout \begin_layout Subsection Local-remote sync \end_layout \begin_layout Standard Remote synchronization is a bit more tricky. \end_layout \begin_layout Standard You might check that you can log in remotely with the command \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ssh -p 22 remotesyncuser@remotehost.tld \end_layout \end_inset Also, you might check that you can use rsync command remotely \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout $ ssh -p 22 remotesyncuser@remotehost.tld rsync --help \end_layout \end_inset You can temporarily disable ssh security by removing lines you added in chapter \begin_inset CommandInset ref LatexCommand ref reference "subsec:Enhancing-remote-backup" \end_inset . Additionnaly, you can check ssh_filter log in ~/.ssh/ssh_filter.log on the remote system. You might try running osync with SUDO_EXEC to check if your problem is user permission related. \end_layout \begin_layout Subsection File monitor mode \end_layout \begin_layout Standard In file monitor mode, osync will still log it's execution to /var/log/osync.insta nce_id.log (or current directory if /var/log is not writable). \end_layout \begin_layout Standard Also, standard systemd log method is used if available. You may check an execution with \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # systemctl status osync-srv@configfile \end_layout \end_inset \end_layout \begin_layout Standard You mai also see osync logs in journalctl with \end_layout \begin_layout Standard \begin_inset listings inline false status open \begin_layout Plain Layout # journalctl -xn \end_layout \end_inset \end_layout \begin_layout Section Final words \end_layout \begin_layout Standard The idea of osync came in a discussion around a beer one evening. It began as a project for a friend, whose company I was working for as a consultant. \end_layout \begin_layout Standard Today, osync is still used by this company, and a lot of others around the globe. \end_layout \begin_layout Standard I do provide technical help and support in my spare time, and will appreciate every contribution on Github :) \end_layout \begin_layout Standard \begin_inset CommandInset line LatexCommand rule offset "0.5ex" width "100col%" height "1pt" \end_inset \end_layout \end_body \end_document