rsync for data transfer from master to client
bash for scripts and templating
unionfs and restricted bash for noop operations
plaintext files in a structured directory tree for node classification and configuration

DISCO is in the very early alpha stages of development. At this point, I am fairly confident that it won't destroy the system you run it on, but I still only run it on disposable virtual machines until it reaches a higher level of maturity.

Server Side Setup

The only server side setup required for DISCO is to setup an rsyncd and sshd server. This is outside the purview of this README.

We would recommend setting up the rsync server to allow your DISCO clients (which MUST run as root), to come in on a non-priveleged, non-root account. You can still use rsync's module definitions with non-root users by setting up ~/.rsyncd in that user's home directory, and adding "--rsh 'ssh -l USER_NAME'" to your /disco/client/cmds/rsync parameter on the clients. This will allow you to specify your rsync locations in your module definitions as USER@HOST::MODULE_NAME instead of having to specify a filesystem path, will give you all the benefits of an SSH key trust relationship, and no concern of incoming root access to the server. (Note that this also prevents the often mysterious and troublesome SSL certificate issues associated with other CI systems.)

Performance Metrics

DISCO stores performance metrics for pretty much everything it does. Use 'disco report' to see them. The report generated will represent the times and statistics for the most recent disco dance.

The Gory Details ("how does it work?")

DISCO is a work in progress so not all of it is complete, but the general idea is this:

- DISCO client rsyncs its node configuration parameters from the server
- DISCO client performs topological sort of required modules, and for each one:
    - fetch all files, templates and scripts
    - resolve all templates
        - resolve all templates
    - execute all scripts
    - report all differences
- report overall success or failure (any piece of any module failing indicates failure)

DISCO is able to easily report all differences by executing all scripts and templates inside a restricted bash chroot, and on top of a read-only unionfs with a scratchpad on the top, some custom twiddly bits in the middle, and the existing running filesystem at the bottom (read-only). The scratchpad is not merged if there is a failure during live (non-NOOP) execution, to prevent from locking the system in a non-functioning state.

If the NOOP flag is set, then all the same operations are performed, but the restricted environment stops all potentially dangerous commands at the reporting level (presumably), and the fetched files are not merged out of the scratchpad onto the live filesystem.

See the client disco-fs-* and disco-exec-* scripts for more information on how this is done.

A complete example

Presume we have a server with an incoming user, "disco", who has a home directory like this:

disco@server:~$ cat rsyncd.conf
[parameters]
path = /home/disco/parameters
read only = true
comment = DISCO Parameters
list = yes
use chroot = false

[testmodule-1.0]
path = /home/disco/modules/testmodule-1.0
read only = true
comment = v1.0 of the Test module
list = yes
use chroot = false

[othermodule-3.2]
path = /home/disco/modules/othermodule-3.2
read only = true
comment = v3.2 of othermodule
list = yes
use chroot = false 

disco@server:~$ find parameters
parameters
parameters/localhost.localdomain
parameters/localhost.localdomain/parameters
parameters/localhost.localdomain/parameters/something
parameters/localhost.localdomain/modules
parameters/localhost.localdomain/modules/othermodule-3.2
parameters/localhost.localdomain/modules/testmodule-1.0

disco@server:~$ cat parameters/localhost.localdomain/parameters/something
LOLTHISKEYMEANSNOTHING

disco@server:~$ find modules
modules
modules/othermodule-3.2
modules/othermodule-3.2/requires
modules/othermodule-3.2/parameters
modules/othermodule-3.2/parameters/othermodule-3.2
modules/othermodule-3.2/scripts
modules/othermodule-3.2/templates
modules/othermodule-3.2/templates/etc
modules/othermodule-3.2/templates/etc/othermodule
modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
modules/othermodule-3.2/files
modules/testmodule-1.0
modules/testmodule-1.0/requires
modules/testmodule-1.0/parameters
modules/testmodule-1.0/parameters/testmodule-1.0
modules/testmodule-1.0/scripts
modules/testmodule-1.0/scripts/00-hello.sh
modules/testmodule-1.0/scripts/10-service_stop.sh
modules/testmodule-1.0/templates
modules/testmodule-1.0/files

disco@server:~$ cat modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
echo HOST=$(hostname)
echo KEY_VALUE=$(cat /var/disco/parameters/$(hostname)/parameters/something)

disco@server:~$ cat modules/testmodule-1.0/scripts/00-hello.sh
#!/bin/bash

echo "Hello, disco"

disco@server:~$ cat modules/testmodule-1.0/scripts/10-service_stop.sh
#!/bin/bash

service postgresql stop

... and that we have, on our client, a disco parameters tree set up like this:

[disco@client disco]$ disco-param dump
disco = {}
disco/client = {}
disco/client/cmds = {}
disco/client/cmds/rsync = rsync -qaWHe "ssh -i /home/disco/.ssh/id_rsa_disco"
disco/server = {}
disco/server/uri = disco@aklabs.net

... Then we can use disco to configure our host.

First we need to mount and initialize disco's testing/noop filesystem as root on the client.

[root@localhost disco]$ NOOP=true disco-fs-mount
[root@localhost disco]$ NOOP=true disco-fs-init

This will take a minute or two, the init does a lot of work. (But you only have to run the init once at system start, no matter how many times you run disco.) Now we can do our noop run:

[disco@localhost disco]$ NOOP=true disco dance
error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
info: Processing testmodule-1.0
Hello, disco
warning: Would execute : service postgresql stop
info: Processing othermodule-3.2
info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
0a1,2
> HOST=localhost.localdomain
> KEY_VALUE=LOLTHISKEYMEANSNOTHING

Here we can see the noop at work; it is preventing potentially destructive commands like 'service' from running, while allowing other harmless commands to operate in the noop context so that script logic is not affected. We can also see the highly detailed statistics and diffs returned for file modifications. But none of the files actually wind up present on the system, and no running processes were affected:

[root@client ~]$ ps ax | grep -i postgresql
15595 pts/1    S+     0:00 grep -i postgresql
24457 ?        S      0:12 /usr/lib/postgresql/8.4/bin/postgres -D /var/lib/postgresql/8.4/main -c config_file=/etc/postgresql/8.4/main/postgresql.conf
[root@client ~]$ ls -l /etc/othermodule/stuff.cfg
ls: cannot access /etc/othermodule/stuff.cfg: No such file or directory

If we were to turn the NOOP flag off, this would all happen for real:

[root@client disco]$ disco dance
error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
info: Processing testmodule-1.0
Hello, disco
info: Processing othermodule-3.2
info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
0a1,2
> HOST=localhost.localdomain
> KEY_VALUE=LOLTHISKEYMEANSNOTHING

... And we will see that the config file has been installed:

[root@client ~]$ cat /etc/othermodule/stuff.cfg
HOST=localhost.localdomain
KEY_VALUE=LOLTHISKEYMEANSNOTHING

... And that postgres has been stopped:

[root@client ~]# ps ax | grep -i postgresql
28394 pts/1    S+     0:00 grep -i postgresql

Hooray!

Disco will report other types of file modifications, as well. If you were to open an interactive shell in the disco chroot, and perform some more interesting operations, representing what a more advanced sort of script might do:

[disco@client disco]$ NOOP=true disco-sh-shell
[root@client /]# rm -f /etc/passwd
[root@client /]# grep -v root /etc/shadow | tee tmpfile
bin:*:15240:0:99999:7:::
daemon:*:15240:0:99999:7:::
adm:*:15240:0:99999:7:::
lp:*:15240:0:99999:7:::
sync:*:15240:0:99999:7:::
shutdown:*:15240:0:99999:7:::
halt:*:15240:0:99999:7:::
mail:*:15240:0:99999:7:::
uucp:*:15240:0:99999:7:::
operator:*:15240:0:99999:7:::
games:*:15240:0:99999:7:::
gopher:*:15240:0:99999:7:::
ftp:*:15240:0:99999:7:::
nobody:*:15240:0:99999:7:::
dbus:!!:15324::::::
usbmuxd:!!:15324::::::
avahi-autoipd:!!:15324::::::
vcsa:!!:15324::::::
rtkit:!!:15324::::::
rpc:!!:15324:0:99999:7:::
pulse:!!:15324::::::
haldaemon:!!:15324::::::
avahi:!!:15324::::::
saslauth:!!:15324::::::
postfix:!!:15324::::::
apache:!!:15324::::::
ntp:!!:15324::::::
rpcuser:!!:15324::::::
nfsnobody:!!:15324::::::
gdm:!!:15324::::::
sshd:!!:15324::::::
tcpdump:!!:15324::::::
disco:$6$Hv67bVi.$d/EolMfURGTMbq1hBr1QL2HdYMYxAXvruq550Qqgu2HCOKWQ1YptMghLKvOAgr3h0NwzXZwHpXQ6fVLdpYe.9.:15533:0:99999:7:::
discostu:!!:15558:0:99999:7:::
[root@client /]# mv tmpfile /etc/shadow
mv: overwrite `/etc/shadow'? y
[root@client /]# echo LOL > /var/lib/p0wnt
bash: /var/lib/p0wnt: restricted: cannot redirect output
[root@client /]# echo LOL | tee /var/lib/p0wnt
LOL
[root@client /]# echo > /bin/myhotbash
bash: /bin/myhotbash: restricted: cannot redirect output
[root@client /]# touch /bin/myhotbash
[root@client /]# exit

... Since that was done inside of the noop shell (where all the scripts and templates run during noop), we can easily report on these activities:

[disco@client disco]$ NOOP=true disco-fs-diff
info: File: file: /etc/passwd : Deleted
1,35d0
< root:x:0:0:root:/root:/bin/bash
< bin:x:1:1:bin:/bin:/sbin/nologin
< daemon:x:2:2:daemon:/sbin:/sbin/nologin
< adm:x:3:4:adm:/var/adm:/sbin/nologin
< lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
< sync:x:5:0:sync:/sbin:/bin/sync
< shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
< halt:x:7:0:halt:/sbin:/sbin/halt
< mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
< uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
< operator:x:11:0:operator:/root:/sbin/nologin
< games:x:12:100:games:/usr/games:/sbin/nologin
< gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
< ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
< nobody:x:99:99:Nobody:/:/sbin/nologin
< dbus:x:81:81:System message bus:/:/sbin/nologin
< usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
< avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
< vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
< rtkit:x:499:496:RealtimeKit:/proc:/sbin/nologin
< rpc:x:32:32:Rpcbind Daemon:/var/cache/rpcbind:/sbin/nologin
< pulse:x:498:495:PulseAudio System Daemon:/var/run/pulse:/sbin/nologin
< haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
< avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
< saslauth:x:497:76:"Saslauthd user":/var/empty/saslauth:/sbin/nologin
< postfix:x:89:89::/var/spool/postfix:/sbin/nologin
< apache:x:48:48:Apache:/var/www:/sbin/nologin
< ntp:x:38:38::/etc/ntp:/sbin/nologin
< rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
< nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
< gdm:x:42:42::/var/lib/gdm:/sbin/nologin
< sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
< tcpdump:x:72:72::/:/sbin/nologin
< disco:x:500:10::/home/disco:/bin/bash
< discostu:x:501:501::/home/discostu:/bin/bash
info: File: file: /bin/myhotbash : Created : type=[regular empty file] device=[fd00] mode=[81a4] selinux=[?] md5=[d41d8cd98f00b204e9800998ecf8427e] perms=[root:root 644]
info: File: file: /etc/shadow : Modified : md5=[8b02f6d00dbcd622f869216bb1dbbbf4 => 336d0b913c8f8cd029964afd00357952] perms=[root:root 0 => root:root 644] mode=[8000 => 81a4]
1d0
< root:$6$57kBYzwRrygFb5op$vghIbLjxmkzTznSbN4kA5fdxFsd1ye7WWe/HFtwMJSTBlDuBcOZISgLKNg/xlA4uAFIBi82yAnW/JajgwhCXY.:15517:0:99999:7:::
info: File: file: /root/.bash_history : Modified : md5=[ead812e487da32cb99cebd09ad7f773b => cb0138b7f3c4f48639cafe9f7147413f] selinux=[unconfined_u:object_r:admin_home_t:s0 => ?]
228a229,237
> exit
> rm -f /etc/passwd
> grep -v root /etc/shadow | tee tmpfile
> mv tmpfile /etc/shadow
> echo LOL > /var/lib/p0wnt
> echo LOL | tee /var/lib/p0wnt
> echo > /bin/myhotbash
> touch /bin/myhotbash
> exit
info: File: file: /var/lib/p0wnt : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[5732edd7e4e1240b868e15bc95d36339] perms=[root:root 644]
0a1
> LOL

And here we see some more of Disco's rather extensive noop reporting capabilities.

But let's say that this run took longer than we thought it should. What was taking so much time? Disco will tell us.

[root@disco ~]# disco report
report: _internal: diff
report:    time_real 0.82 : time_user 0.14 : time_sys 0.66
report:    mem_avg 0 : mem_max 5184 : mem_faults_major 0 : mem_faults_minor 18218
report:    io_fsin 0 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0
report: _internal: fetch_params
report:    time_real 1.26 : time_user 0.02 : time_sys 0.08
report:    mem_avg 0 : mem_max 11136 : mem_faults_major 0 : mem_faults_minor 1728
report:    io_fsin 0 : io_fsout 0 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0
report: othermodule-3.2: diff
report:    time_real 0.80 : time_user 0.13 : time_sys 0.65
report:    mem_avg 0 : mem_max 4816 : mem_faults_major 0 : mem_faults_minor 16448
report:    io_fsin 0 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0
report: othermodule-3.2: fetch
report:    time_real 2.28 : time_user 0.05 : time_sys 0.14
report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2843
report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0
report: othermodule-3.2: template
report:    etc/othermodule/stuff.cfg :
report:        time_real 0.56 : time_user 0.04 : time_sys 0.37
report:        mem_avg 0 : mem_max 4592 : mem_faults_major 122 : mem_faults_minor 4885
report:        io_fsin 25536 : io_fsout 16 : io_sockin 0 : io_sockout 0 : io_signals 0
report:        exit: 0
report: testmodule-1.0: diff
report:    time_real 2.56 : time_user 0.42 : time_sys 2.10
report:    mem_avg 0 : mem_max 5184 : mem_faults_major 0 : mem_faults_minor 57661
report:    io_fsin 0 : io_fsout 32 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0
report: testmodule-1.0: exec
report:    00-hello.sh :
report:        time_real 0.27 : time_user 0.03 : time_sys 0.18
report:        mem_avg 0 : mem_max 4672 : mem_faults_major 32 : mem_faults_minor 3783
report:        io_fsin 6640 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
report:        exit: 0
report:    10-service_stop.sh :
report:        time_real 0.58 : time_user 0.03 : time_sys 0.41
report:        mem_avg 0 : mem_max 4960 : mem_faults_major 122 : mem_faults_minor 5462
report:        io_fsin 25656 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
report:        exit: 1
report: testmodule-1.0: fetch
report:    time_real 2.72 : time_user 0.06 : time_sys 0.15
report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2996
report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
report:    exit: 0

Happy dancing!