Copying files to multiple servers

Now I have 20 servers + 1 main one. With the main one, I use scp to drop files on all the others.

How can I automate the process so that one file can be uploaded to my list of servers?


Now I do this:

$ scp package.rpm user@server2:/tmp/

And so with 20 servers.

Author: aleksandr barakin, 2016-06-30

5 answers

The issue of synchronization between hosts is not solved in one line. I will describe the general principles, since the specific setting will be extensive in the description.

There were several ways that I tried on hosts (3,000+): write scripts manually, create synchronization daemons, use ready-made solutions.

For otpraki used ssh, http[s] and torrent protocols, the latter allows you to create your own fault-tolerant network of content transfers.

From simple to confusing

You have a local network, without any restrictions. Hosts and server on Linux. You can raise the web server. with lsftp for file distribution. The pxe+apache+lsftp bundle has been performing its task perfectly for two years now. Tseklichno run. on each

~$ wget file_server:/file 

In this option, you do not need keys for hosts and passwords for ftp at all, it is enough to have remote access to each one. The pull{[4] paradigm]}

You have disparate hosts that are accessed via a shared network (intrnet, WAN, MAN, or VPN). In the presence of an ssh server-client on each of the hosts and the main server. In this option, it is more profitable to use rsync and expect if there are no RSA keys, then we write a small script on the server for expect

#!/usr/bin/env bash

hosts="$1"
send_file="$2"

for host in "$hosts"; do
    expect -c "
         spawn rsync -az --append \"$send_file\" \"$host\"
         expect \".assword.+\"
         send \"my_password\r\"
    " &
done

With keys and in one line

~# while read -r host; do rsync -az --append /path/to/file "$host" & done < hosts

An interesting point is to send each "hand" to the background process &.

You want to synchronize thousands and more hosts or cluster nodes. Here already scripts can not cope with ready-made and free solutions can be noted:

  • Ansible-copy and synchronize modules. ssh protocol
  • Lsyncd (Live Syncing Daemon) syncs local directories with remote hosts. rsync wrapper daemonized on Lua
  • Resilient is a solution for distributed file delivery.
  • glusterfs distributed file system
  • libtorrent is an open library in c++ see applications using it as a base, on the official website.
 1
Author: Hellseher, 2017-07-07 21:52:19

Every cricket praises its hearth. I will praise Ansible. It uses a normal SSH connection with the same settings as the normal ssh hostname

Let's assume that you have already configured SSH key access.

We list the hosts in the file hosts (any name). You can also set the parameter, but I assume that the list of hosts you have is more or less constant.

host1.dc1
host2.dc1
# все номера с 1 по 10
host[1:10].dc2 

Run the command anywhere.

ansible all -i path/to/hosts -m copy -a 'src=path/to/example.txt dest=/tmp/example.txt' 

And that's it, the files are copied, and you get report on each host:

example.host | SUCCESS => {
    "changed": true, 
    "checksum": "c3499c2729730a7f807efb8676a92dcb6f8a3f8f", 
    "dest": "/tmp/example.txt", 
    "gid": 1002, 
    "group": "root", 
    "md5sum": "1a79a4d60de6718e8e5b326e338ae533", 
    "mode": "0664", 
    "owner": "root", 
    "size": 7, 
    "src": "/home/username/.ansible/tmp/ansible-tmp-1467312219.54-140783443033936/source", 
    "state": "file", 
    "uid": 1002
}

Additional parameters here: http://docs.ansible.com/ansible/copy_module.html

If you have not configured SSH keys, then add to the start line:

-u <имя-пользователя> --ask-pass

Then the connection will be established by the specified username, the password will be requested at the command line.

 10
Author: Nick Volynkin, 2016-07-03 09:25:24

With keys

I would recommend using the key authorization option.

  1. Generate a public/private key pair for yourself:

    $ ssh-keygen
    

    All questions can be answered " yes " (press enter). if the key already exists, nothing terrible will happen, the program will inform you about it and offer to overwrite the files (which you can refuse by pressing n and enter).

  2. Copy it the public part of the key to all the necessary servers:

    $ ssh-copy-id user@host
    
  3. install the program parallel-scp (in debian-major distributions, it is usually included in the package pssh and renamed from the original name pscp to avoid a name conflict with the program from the package putty-tools, which has completely different functionality).
  4. Copy the file:

    $ parallel-scp -H user1@host1 -H user2@host2 /локальный/файл /путь/на/удалённой/машине
    [1] 18:40:23 [SUCCESS] user2@host2
    [2] 18:40:24 [SUCCESS] user1@host1
    

    /путь/на/удалённой/машине must be absolute, and the path to the local file is optional (can be relative).

    There can be any (reasonable) number of options -H user@host.

    Moreover , it makes sense to specify the -h /путь/к/списку option instead, where /путь/к/списку is a file containing strings of the [user@]host[:port] format (in some implementations - host[:port] [user], check in the help for your version: man parallel-scp).

If it is not possible to install the mentioned program, use the advice from the answer of @sercxjo.

Without keys

If for some reason you don't you can / want to use access by key, then, in order not to enter a password to the request of the program ssh/scp, you can use, for example, the program sshpass (usually-from the package of the same name):

$ sshpass -p 'пароль' scp параметры-программы-scp
$ sshpass -p 'пароль' ssh параметры-программы-ssh

I think it will not be difficult to modify the script proposed by @sercxjo accordingly.


Useful information: Simultaneous management of multiple servers

 9
Author: aleksandr barakin, 2017-04-13 12:53:30

Configure ssh authorization using the key.

Example of a script for parallel download, if the user and the download paths on all servers are the same ($1 - the first argument when running the script):

#!/bin/bash
serverlist=server1 server2 server3
for server in $serverlist ; do
    scp "$1" user@$server:/tmp/ &
done
wait
 3
Author: sercxjo, 2017-04-13 12:53:25

Here, I look, while there is no specific help, I sketched something (like working) on the knee:

#!/bin/bash

echo Test scp-expect
User=avp
PASS=XXXXXX
RDIR=/tmp
FILE=$1
if [ "X$FILE" == "X" ]; then
    echo usage: ./tscp FILE
    exit 1
fi

echo Go...

#rm ~/.ssh/known_hosts  вместо этого -o 'UserKnownHostsFile /dev/null' в вызове scp

for HOST in nas
do
    expect -c "
spawn scp -o UserKnownHostsFile\\ /dev/null $FILE $User@$HOST:$RDIR
expect \".*no)? \"
send \"yes\r\"
expect \".*assword:\"
send \"$PASS\r\"
expect eof
"
done

In this case, a single file is forwarded to a single host (named nas) in the table of contents /tmp.

Note that to simplify the script, expect I make all local information ssh about hosts unavailable (by default it is located in~/. ssh/known_hosts), because at startup it is not known whether ssh/scp has already visited this host or not yet (and then scp requires input "yes").

Well, I think how expect works here is clear in principle, and you will write the Shell binding (what, where, where, where to get the password, etc.) for your own task.

 3
Author: avp, 2016-07-01 14:28:20