disco/README.md

disco
=====

Dead Simple COnfiguration management and continuous integration for linux like systems

DISCO uses simple, proven technologies to achieve the goal of configuration management.

* rsync for data transfer from master to client
* bash for scripts and templating
* unionfs and restricted bash for noop operations
* plaintext files in a structured directory tree for node classification and configuration

DISCO is in the very early alpha stages of development. At this point, I am fairly confident
that it won't destroy the system you run it on, but I still only run it on disposable virtual
machines until it reaches a higher level of maturity.

Server Side Setup
=====

The only server side setup required for DISCO is to setup an rsyncd and sshd server. This is outside
the purview of this README. 

We would recommend setting up the rsync server to allow your DISCO clients (which MUST run as root),
to come in on a non-priveleged, non-root account. You can still use rsync's module definitions with
non-root users by setting up ~/.rsyncd in that user's home directory, and adding
"--rsh 'ssh -l USER_NAME'" to your /disco/client/cmds/rsync parameter on the clients. This will
allow you to specify your rsync locations in your module definitions as USER@HOST::MODULE_NAME instead
of having to specify a filesystem path, will give you all the benefits of an SSH key trust relationship,
and no concern of incoming root access to the server. (Note that this also prevents the often mysterious
and troublesome SSL certificate issues associated with other CI systems.)

Performance Metrics
=====

DISCO stores performance metrics for pretty much everything it does. Use 'disco report' to see them. 
The report generated will represent the times and statistics for the most recent disco dance.

The Gory Details ("how does it work?")
=====

DISCO is a work in progress so not all of it is complete, but the general idea is this:

    - DISCO client rsyncs its node configuration parameters from the server
    - DISCO client performs topological sort of required modules, and for each one:
        - fetch all files, templates and scripts
        - resolve all templates
            - resolve all templates
        - execute all scripts
        - report all differences
    - report overall success or failure (any piece of any module failing indicates failure)

DISCO is able to easily report all differences by executing all scripts and templates inside a 
restricted bash chroot, and on top of a read-only unionfs with a scratchpad on the top,
some custom twiddly bits in the middle, and the existing running filesystem at the bottom (read-only). 
The scratchpad is not merged if there is a failure during live (non-NOOP) execution, to prevent
from locking the system in a non-functioning state.

If the NOOP flag is set, then all the same operations are performed, but the restricted
environment stops all potentially dangerous commands at the reporting level (presumably), and
the fetched files are not merged out of the scratchpad onto the live filesystem. 

See the client disco-fs-* and disco-exec-* scripts for more information on how this is done.

A complete example
=====

Presume we have a server with an incoming user, "disco", who has a home directory like this:

    disco@server:~$ cat rsyncd.conf
    [parameters]
    path = /home/disco/parameters
    read only = true
    comment = DISCO Parameters
    list = yes
    use chroot = false
   
    [testmodule-1.0]
    path = /home/disco/modules/testmodule-1.0
    read only = true
    comment = v1.0 of the Test module
    list = yes
    use chroot = false
   
    [othermodule-3.2]
    path = /home/disco/modules/othermodule-3.2
    read only = true
    comment = v3.2 of othermodule
    list = yes
    use chroot = false 

    disco@server:~$ find parameters
    parameters
    parameters/localhost.localdomain
    parameters/localhost.localdomain/parameters
    parameters/localhost.localdomain/parameters/something
    parameters/localhost.localdomain/modules
    parameters/localhost.localdomain/modules/othermodule-3.2
    parameters/localhost.localdomain/modules/testmodule-1.0
  
    disco@server:~$ cat parameters/localhost.localdomain/parameters/something
    LOLTHISKEYMEANSNOTHING
  
    disco@server:~$ find modules
    modules
    modules/othermodule-3.2
    modules/othermodule-3.2/requires
    modules/othermodule-3.2/parameters
    modules/othermodule-3.2/parameters/othermodule-3.2
    modules/othermodule-3.2/scripts
    modules/othermodule-3.2/templates
    modules/othermodule-3.2/templates/etc
    modules/othermodule-3.2/templates/etc/othermodule
    modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
    modules/othermodule-3.2/files
    modules/testmodule-1.0
    modules/testmodule-1.0/requires
    modules/testmodule-1.0/parameters
    modules/testmodule-1.0/parameters/testmodule-1.0
    modules/testmodule-1.0/scripts
    modules/testmodule-1.0/scripts/00-hello.sh
    modules/testmodule-1.0/scripts/10-service_stop.sh
    modules/testmodule-1.0/templates
    modules/testmodule-1.0/files
   
    disco@server:~$ cat modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
    echo HOST=$(hostname)
    echo KEY_VALUE=$(cat /var/disco/parameters/$(hostname)/parameters/something)
  
    disco@server:~$ cat modules/testmodule-1.0/scripts/00-hello.sh
    #!/bin/bash
  
    echo "Hello, disco"
  
    disco@server:~$ cat modules/testmodule-1.0/scripts/10-service_stop.sh
    #!/bin/bash
  
    service postgresql stop

... and that we have, on our client, a disco parameters tree set up like this:

    [disco@client disco]$ disco-param dump
    disco = {}
    disco/client = {}
    disco/client/cmds = {}
    disco/client/cmds/rsync = rsync -qaWHe "ssh -i /home/disco/.ssh/id_rsa_disco"
    disco/server = {}
    disco/server/uri = disco@aklabs.net

... Then we can use disco to configure our host.

First we need to mount and initialize disco's testing/noop filesystem as
root on the client.

    [root@localhost disco]$ NOOP=true disco-fs-mount
    [root@localhost disco]$ NOOP=true disco-fs-init

This will take a minute or two, the init does a lot of work. (But you only
have to run the init once at system start, no matter how many times you 
run disco.) Now we can do our noop run:

    [disco@localhost disco]$ NOOP=true disco dance
    error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
    error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
    info: Processing testmodule-1.0
    Hello, disco
    warning: Would execute : service postgresql stop
    info: Processing othermodule-3.2
    info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
    0a1,2
    > HOST=localhost.localdomain
    > KEY_VALUE=LOLTHISKEYMEANSNOTHING

Here we can see the noop at work; it is preventing potentially destructive
commands like 'service' from running, while allowing other harmless commands
to operate in the noop context so that script logic is not affected. We can
also see the highly detailed statistics and diffs returned for file 
modifications. But none of the files actually wind up present on the 
system, and no running processes were affected:

    [root@client ~]$ ps ax | grep -i postgresql
    15595 pts/1    S+     0:00 grep -i postgresql
    24457 ?        S      0:12 /usr/lib/postgresql/8.4/bin/postgres -D /var/lib/postgresql/8.4/main -c config_file=/etc/postgresql/8.4/main/postgresql.conf
    [root@client ~]$ ls -l /etc/othermodule/stuff.cfg
    ls: cannot access /etc/othermodule/stuff.cfg: No such file or directory

If we were to turn the NOOP flag off, this would all happen for real:

    [root@client disco]$ disco dance
    error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
    error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
    info: Processing testmodule-1.0
    Hello, disco
    info: Processing othermodule-3.2
    info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
    0a1,2
    > HOST=localhost.localdomain
    > KEY_VALUE=LOLTHISKEYMEANSNOTHING

... And we will see that the config file has been installed:

    [root@client ~]$ cat /etc/othermodule/stuff.cfg
    HOST=localhost.localdomain
    KEY_VALUE=LOLTHISKEYMEANSNOTHING

... And that postgres has been stopped:

    [root@client ~]# ps ax | grep -i postgresql
    28394 pts/1    S+     0:00 grep -i postgresql

Hooray!

Disco will report other types of file modifications, as well. If you were to 
open an interactive shell in the disco chroot, and perform some more interesting
operations, representing what a more advanced sort of script might do:

    [disco@client disco]$ NOOP=true disco-sh-shell
    [root@client /]# rm -f /etc/passwd
    [root@client /]# grep -v root /etc/shadow | tee tmpfile
    bin:*:15240:0:99999:7:::
    daemon:*:15240:0:99999:7:::
    adm:*:15240:0:99999:7:::
    lp:*:15240:0:99999:7:::
    sync:*:15240:0:99999:7:::
    shutdown:*:15240:0:99999:7:::
    halt:*:15240:0:99999:7:::
    mail:*:15240:0:99999:7:::
    uucp:*:15240:0:99999:7:::
    operator:*:15240:0:99999:7:::
    games:*:15240:0:99999:7:::
    gopher:*:15240:0:99999:7:::
    ftp:*:15240:0:99999:7:::
    nobody:*:15240:0:99999:7:::
    dbus:!!:15324::::::
    usbmuxd:!!:15324::::::
    avahi-autoipd:!!:15324::::::
    vcsa:!!:15324::::::
    rtkit:!!:15324::::::
    rpc:!!:15324:0:99999:7:::
    pulse:!!:15324::::::
    haldaemon:!!:15324::::::
    avahi:!!:15324::::::
    saslauth:!!:15324::::::
    postfix:!!:15324::::::
    apache:!!:15324::::::
    ntp:!!:15324::::::
    rpcuser:!!:15324::::::
    nfsnobody:!!:15324::::::
    gdm:!!:15324::::::
    sshd:!!:15324::::::
    tcpdump:!!:15324::::::
    disco:$6$Hv67bVi.$d/EolMfURGTMbq1hBr1QL2HdYMYxAXvruq550Qqgu2HCOKWQ1YptMghLKvOAgr3h0NwzXZwHpXQ6fVLdpYe.9.:15533:0:99999:7:::
    discostu:!!:15558:0:99999:7:::
    [root@client /]# mv tmpfile /etc/shadow
    mv: overwrite `/etc/shadow'? y
    [root@client /]# echo LOL > /var/lib/p0wnt
    bash: /var/lib/p0wnt: restricted: cannot redirect output
    [root@client /]# echo LOL | tee /var/lib/p0wnt
    LOL
    [root@client /]# echo > /bin/myhotbash
    bash: /bin/myhotbash: restricted: cannot redirect output
    [root@client /]# touch /bin/myhotbash
    [root@client /]# exit

... Since that was done inside of the noop shell (where all the scripts and 
templates run during noop), we can easily report on these activities:

    [disco@client disco]$ NOOP=true disco-fs-diff
    info: File: file: /etc/passwd : Deleted
    1,35d0
    < root:x:0:0:root:/root:/bin/bash
    < bin:x:1:1:bin:/bin:/sbin/nologin
    < daemon:x:2:2:daemon:/sbin:/sbin/nologin
    < adm:x:3:4:adm:/var/adm:/sbin/nologin
    < lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
    < sync:x:5:0:sync:/sbin:/bin/sync
    < shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
    < halt:x:7:0:halt:/sbin:/sbin/halt
    < mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
    < uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
    < operator:x:11:0:operator:/root:/sbin/nologin
    < games:x:12:100:games:/usr/games:/sbin/nologin
    < gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
    < ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
    < nobody:x:99:99:Nobody:/:/sbin/nologin
    < dbus:x:81:81:System message bus:/:/sbin/nologin
    < usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
    < avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
    < vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
    < rtkit:x:499:496:RealtimeKit:/proc:/sbin/nologin
    < rpc:x:32:32:Rpcbind Daemon:/var/cache/rpcbind:/sbin/nologin
    < pulse:x:498:495:PulseAudio System Daemon:/var/run/pulse:/sbin/nologin
    < haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
    < avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
    < saslauth:x:497:76:"Saslauthd user":/var/empty/saslauth:/sbin/nologin
    < postfix:x:89:89::/var/spool/postfix:/sbin/nologin
    < apache:x:48:48:Apache:/var/www:/sbin/nologin
    < ntp:x:38:38::/etc/ntp:/sbin/nologin
    < rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
    < nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
    < gdm:x:42:42::/var/lib/gdm:/sbin/nologin
    < sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
    < tcpdump:x:72:72::/:/sbin/nologin
    < disco:x:500:10::/home/disco:/bin/bash
    < discostu:x:501:501::/home/discostu:/bin/bash
    info: File: file: /bin/myhotbash : Created : type=[regular empty file] device=[fd00] mode=[81a4] selinux=[?] md5=[d41d8cd98f00b204e9800998ecf8427e] perms=[root:root 644]
    info: File: file: /etc/shadow : Modified : md5=[8b02f6d00dbcd622f869216bb1dbbbf4 => 336d0b913c8f8cd029964afd00357952] perms=[root:root 0 => root:root 644] mode=[8000 => 81a4]
    1d0
    < root:$6$57kBYzwRrygFb5op$vghIbLjxmkzTznSbN4kA5fdxFsd1ye7WWe/HFtwMJSTBlDuBcOZISgLKNg/xlA4uAFIBi82yAnW/JajgwhCXY.:15517:0:99999:7:::
    info: File: file: /root/.bash_history : Modified : md5=[ead812e487da32cb99cebd09ad7f773b => cb0138b7f3c4f48639cafe9f7147413f] selinux=[unconfined_u:object_r:admin_home_t:s0 => ?]
    228a229,237
    > exit
    > rm -f /etc/passwd
    > grep -v root /etc/shadow | tee tmpfile
    > mv tmpfile /etc/shadow
    > echo LOL > /var/lib/p0wnt
    > echo LOL | tee /var/lib/p0wnt
    > echo > /bin/myhotbash
    > touch /bin/myhotbash
    > exit
    info: File: file: /var/lib/p0wnt : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[5732edd7e4e1240b868e15bc95d36339] perms=[root:root 644]
    0a1
    > LOL

And here we see some more of Disco's rather extensive noop reporting capabilities.

But let's say that this run took longer than we thought it should. What was taking
so much time? Disco will tell us.

    [root@disco ~]# disco report
    report: _internal: diff
    report:    time_real 0.82 : time_user 0.14 : time_sys 0.66
    report:    mem_avg 0 : mem_max 5184 : mem_faults_major 0 : mem_faults_minor 18218
    report:    io_fsin 0 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0
    report: _internal: fetch_params
    report:    time_real 1.26 : time_user 0.02 : time_sys 0.08
    report:    mem_avg 0 : mem_max 11136 : mem_faults_major 0 : mem_faults_minor 1728
    report:    io_fsin 0 : io_fsout 0 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0
    report: othermodule-3.2: diff
    report:    time_real 0.80 : time_user 0.13 : time_sys 0.65
    report:    mem_avg 0 : mem_max 4816 : mem_faults_major 0 : mem_faults_minor 16448
    report:    io_fsin 0 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0
    report: othermodule-3.2: fetch
    report:    time_real 2.28 : time_user 0.05 : time_sys 0.14
    report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2843
    report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0
    report: othermodule-3.2: template
    report:    etc/othermodule/stuff.cfg :
    report:        time_real 0.56 : time_user 0.04 : time_sys 0.37
    report:        mem_avg 0 : mem_max 4592 : mem_faults_major 122 : mem_faults_minor 4885
    report:        io_fsin 25536 : io_fsout 16 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:        exit: 0
    report: testmodule-1.0: diff
    report:    time_real 2.56 : time_user 0.42 : time_sys 2.10
    report:    mem_avg 0 : mem_max 5184 : mem_faults_major 0 : mem_faults_minor 57661
    report:    io_fsin 0 : io_fsout 32 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0
    report: testmodule-1.0: exec
    report:    00-hello.sh :
    report:        time_real 0.27 : time_user 0.03 : time_sys 0.18
    report:        mem_avg 0 : mem_max 4672 : mem_faults_major 32 : mem_faults_minor 3783
    report:        io_fsin 6640 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:        exit: 0
    report:    10-service_stop.sh :
    report:        time_real 0.58 : time_user 0.03 : time_sys 0.41
    report:        mem_avg 0 : mem_max 4960 : mem_faults_major 122 : mem_faults_minor 5462
    report:        io_fsin 25656 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:        exit: 1
    report: testmodule-1.0: fetch
    report:    time_real 2.72 : time_user 0.06 : time_sys 0.15
    report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2996
    report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
    report:    exit: 0

Happy dancing!