My application generates text files that need to be synced with remote servers, which may be windows or linux. Sync has to happen without user’s intervention.
I tried with rsync but windows doesn’t come with rsync by default. Also it is not possible to supply password in the command line for rsync.
Currently I’m going with ftp. But that seems like an inefficient way.
Is there a way to rsync without user intervention? What are the ways to sync with a remote server programmatically? App is on nodejs.
4
Rather than using FTP which is insecure (compared to SFTP) and takes a lot of time and effort (relatively) I suggest you use SSH to upload the files to the remote server through SCP with Cron jobs (if this is a routine operation), or by adding an alias in your ~/.bash_profile for something like push text-files.
Because you’ve said this app runs with Node you can use something called exec to run commands from the script. For example you could say something like this:
exec = require('child_process').exec;
exec('scp -r /path/to/the/files server:/path/they/should/upload/to');
Here’s a great article on setting up SSH with a Unix-based server if you’re unfamiliar: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys–2
You could also write jobs in your app with something like node-schedule or another job package. This approach means you can skip writing Cron jobs on the server. The Rails docs are completely unrelated to Node, but the introduction will give you an idea as to what jobs are and what they do: http://edgeguides.rubyonrails.org/active_job_basics.html
I think you should accomplish two tasks:
- watch source filesystem for change. This could be done by either cron job (task scheduler for windows) to check for changes, FTP watcher that checks changes in FTP server. I am sure there are other options too.
- You have to copy source files to target servers. This you can achieve in many ways trough ftp or ssh or even upload files through http.
Your best setup depends on what is already installed there on servers or what is easier for you to set up. Also do you need to sync back if files have changes in remote servers.
1
Synching entire files has the tendency to create more traffic than you want. It may be a better option to do this on a finer-grained level, using your own RPC service.
For example, you could have a look at Apache Thrift. It basically allows you to create a remote service to do next to anything what you want. It works cross-language, cross-platform out of the box, designed to be efficient and based on a very simple IDL.
While creating your own service may involve some work, you are no longer bound to transferring entire files (but you can of course do that as well), you will also be able to extend the service in the future easily.
If you still want to exchange whole files, consider compressing them beforehand, and using services like dropbox, or the like. Depending on the “disclosure-level” of your data, you may additionally want to encrypt them during transfer.