Added real life example to the README, minor fixups to disco and disco-fs-diff

README.md

@@ -24,8 +24,8 @@ Requirements
 
 DISCO assumes that you:
 
-    - have at least one server capable of running rsyncd, sshd and bash 4+
-    - have one or more clients capable of running bash 4+, ssh, rsync, and fuse-unionfs
+    - have at least one server capable of running rsyncd, sshd and a recent GNU
+    - have one or more clients capable of running ssh, rsync, fuse-unionfs, and recent GNU
 
 While that's a very simple requirements list, it currently restricts it to recent Linux
 systems. You may or may not be able to use this tool on FreeBSD or Mac OS X, I haven't
@@ -88,8 +88,9 @@ From the server perspective, the parameters tree looks like:
     ___ ___ ___ client
     ___ ___ ___ ___ cmds
     ___ ___ ___ ___ ___ rsync
-    ___ ___ server
-    ___ ___ ___ uri
+    ___ ___ disco
+    ___ ___ ___ server
+    ___ ___ ___ ___ uri
     ___ ___ NODE_NAME
     ___ ___ ___ modules
     ___ ___ ___ ___ ...
@@ -111,18 +112,18 @@ operation, and is accessible as a filesystem tree (or the disco-param command wh
 bash wrapper). These parameters appear in /var/disco/parameters on the client and server, and 
 default values can be found there in the client/server install before the first run of the client.
 
-    /disco/client/cmds/rsync : The rsync command to use when synching
+    disco/client/cmds/rsync : The rsync command to use when synching
         files.
-    /disco/server/uri : The rsync URI from which to fetch module definitions.
-    /disco/NODE_NAME/modules : This list defines the modules to install 
+    disco/server/uri : The rsync URI from which to fetch module definitions.
+    disco/NODE_NAME/modules : This list defines the modules to install 
         on a given node. 
-    /disco/NODE_NAME/parameters : This tree defines all configuration 
+    disco/NODE_NAME/parameters : This tree defines all configuration 
         parameters for the node not related to any module in particular.
 
 Some special parameters are provided to the client, that do not exist on the paramters tree until
 runtime:
 
-    /disco/NODE_NAME/current_module : This parameter defines the full 
+    disco/NODE_NAME/current_module : This parameter defines the full 
         name of the current module, such that a module definition file 
         can access its personal parameters via without knowing its name, e.g.:
         $(disco-param get $(hostname)/classes/$(disco-param get $(hostname)/current_module)/some/module/specific/path)
@@ -135,10 +136,9 @@ rudimentary dependency mechanism implemented via a topological sort.
 
 Essentially, to deploy something, you need 3 things:
 
-    - some files and templates on an rsync server
+    - some static files on an rsync server
+    - some bash templates (scripts that output what their content should be) on an rsync server
     - some scripts that may or may not do something with those files and templates
-    - a definition file saying where to get those files, templates and scripts, and which
-      order to apply them in, as well as what other things you need deployed before this thing
 
 Scripts
 =====
@@ -168,18 +168,6 @@ NOOP execution environment (file modifications will be discarded, and only rudim
 are enabled). Templates have access to all client parameters via the disco-param command. Templates
 will end up with the same permissions that rsync gives them.
 
-Definition Files
-=====
-
-Definition files are just a series of files that say what files on the disk should be templated,
-or executed as scripts, for this module; as well as defining module-level parameters, and dependency
-requirements, for this module.
-
-Definition files can use node parameters via the $(disco-param /path/to/node/parameter) syntax.
-This interpolation is done on the client side, so the server does not execute any code for this.
-This is useful for when a module needs to pull different files or whatever depending on its branch,
-release name, whatever.
-
 Module Layout
 =====
 
@@ -188,7 +176,6 @@ A disco module (also called a "disco ball" for fun) looks like this:
     MODULE
     ___ requires
     ___ parameters
-    ___ steps
     ├── files/
     ├── scripts/
     └── templates/
@@ -260,3 +247,315 @@ environment stops all potentially dangerous commands at the reporting level (pre
 the fetched files are not merged out of the scratchpad onto the live filesystem. 
 
 See the client disco-fs-* and disco-exec-* scripts for more information on how this is done.
+
+A simple example
+=====
+
+Presume we have a server with an incoming user, "disco", who has a home directory like this:
+
+ disco@server:~$ cat rsyncd.conf
+ [parameters]
+   path = /home/disco/parameters
+   read only = true
+   comment = DISCO Parameters
+   list = yes
+   use chroot = false
+ 
+ [testmodule-1.0]
+   path = /home/disco/modules/testmodule-1.0
+   read only = true
+   comment = v1.0 of the Test module
+   list = yes
+   use chroot = false
+ 
+ [othermodule-3.2]
+   path = /home/disco/modules/othermodule-3.2
+   read only = true
+   comment = v3.2 of othermodule
+   list = yes
+   use chroot = false
+
+... and that we have, on our client, a disco parameters tree set up like this:
+
+ [disco@client disco]$ disco-param dump
+ disco = {}
+ disco/client = {}
+ disco/client/cmds = {}
+ disco/client/cmds/rsync = rsync -qaWHe "ssh -i /home/disco/.ssh/id_rsa_disco"
+ disco/server = {}
+ disco/server/uri = disco@aklabs.net
+ localhost.localdomain = {}
+ localhost.localdomain/modules = {}
+ localhost.localdomain/modules/othermodule-3.2 = {}
+ localhost.localdomain/modules/testmodule-1.0 = {}
+ localhost.localdomain/parameters = {}
+ localhost.localdomain/parameters/something = LOLTHISKEYMEANSNOTHING
+
+... And that the parameters/modules on our rsync server look like this:
+
+ disco@server:~$ find parameters
+ parameters
+ parameters/localhost.localdomain
+ parameters/localhost.localdomain/parameters
+ parameters/localhost.localdomain/parameters/something
+ parameters/localhost.localdomain/modules
+ parameters/localhost.localdomain/modules/othermodule-3.2
+ parameters/localhost.localdomain/modules/testmodule-1.0
+ 
+ disco@server:~$ cat parameters/localhost.localdomain/parameters/something
+ LOLTHISKEYMEANSNOTHING
+ 
+ disco@server:~$ find modules
+ modules
+ modules/othermodule-3.2
+ modules/othermodule-3.2/requires
+ modules/othermodule-3.2/parameters
+ modules/othermodule-3.2/parameters/othermodule-3.2
+ modules/othermodule-3.2/scripts
+ modules/othermodule-3.2/templates
+ modules/othermodule-3.2/templates/etc
+ modules/othermodule-3.2/templates/etc/othermodule
+ modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
+ modules/othermodule-3.2/files
+ modules/testmodule-1.0
+ modules/testmodule-1.0/requires
+ modules/testmodule-1.0/parameters
+ modules/testmodule-1.0/parameters/testmodule-1.0
+ modules/testmodule-1.0/scripts
+ modules/testmodule-1.0/scripts/00-hello.sh
+ modules/testmodule-1.0/scripts/10-service_stop.sh
+ modules/testmodule-1.0/templates
+ modules/testmodule-1.0/files
+ 
+ disco@server:~$ cat modules/othermodule-3.2/templates/etc/othermodule/stuff.cfg
+ echo HOST=$(hostname)
+ echo KEY_VALUE=$(cat /var/disco/parameters/$(hostname)/parameters/something)
+ 
+ disco@server:~$ cat modules/testmodule-1.0/scripts/00-hello.sh
+ #!/bin/bash
+ 
+ echo "Hello, disco"
+ 
+ disco@server:~$ cat modules/testmodule-1.0/scripts/10-service_stop.sh
+ #!/bin/bash
+ 
+ service postgresql stop
+
+... Then we can use disco to configure our host.
+
+First we need to mount and initialize disco's testing/noop filesystem as
+root on the client.
+
+ [root@localhost disco]$ NOOP=true disco-fs-mount
+ [root@localhost disco]$ NOOP=true disco-fs-init
+
+This will take a minute or two, the init does a lot of work. Now we can 
+do our noop run:
+
+ [disco@localhost disco]$ NOOP=true disco dance
+ error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
+ error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
+ info: Processing testmodule-1.0
+ Hello, disco
+ warning: Would execute : service postgresql stop
+ info: Processing othermodule-3.2
+ info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
+ 0a1,2
+ > HOST=localhost.localdomain
+ > KEY_VALUE=LOLTHISKEYMEANSNOTHING
+
+Here we can see the noop at work; it is preventing potentially destructive
+commands like 'service' from running, while allowing other harmless commands
+to operate in the noop context so that script logic is not affected. We can
+also see the highly detailed statistics and diffs returned for file 
+modifications. But none of the files actually wind up present on the 
+system, and no running processes were affected:
+
+ [root@client ~]$ ps ax | grep -i postgresql
+ 15595 pts/1    S+     0:00 grep -i postgresql
+ 24457 ?        S      0:12 /usr/lib/postgresql/8.4/bin/postgres -D /var/lib/postgresql/8.4/main -c config_file=/etc/postgresql/8.4/main/postgresql.conf
+ [root@client ~]$ ls -l /etc/othermodule/stuff.cfg
+ ls: cannot access /etc/othermodule/stuff.cfg: No such file or directory
+
+If we were to turn the NOOP flag off, this would all happen for real:
+
+ [root@client disco]$ disco dance
+ error: othermodule-3.2: rsync: link_stat "/files/*" (in othermodule-3.2) failed: No such file or directory (2)
+ error: testmodule-1.0: rsync: link_stat "/files/*" (in testmodule-1.0) failed: No such file or directory (2)
+ info: Processing testmodule-1.0
+ Hello, disco
+ info: Processing othermodule-3.2
+ info: File: file: /etc/othermodule/stuff.cfg : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[77b20e4840b1be13a577e152edc6b443] perms=[root:root 644]
+ 0a1,2
+ > HOST=localhost.localdomain
+ > KEY_VALUE=LOLTHISKEYMEANSNOTHING
+
+... And we will see that the config file has been installed:
+
+ [root@client ~]$ cat /etc/othermodule/stuff.cfg
+ HOST=localhost.localdomain
+ KEY_VALUE=LOLTHISKEYMEANSNOTHING
+
+... And that postgres has been stopped:
+
+ [root@client ~]# ps ax | grep -i postgresql
+ 28394 pts/1    S+     0:00 grep -i postgresql
+
+Hooray!
+
+Disco will report other types of file modifications, as well. If you were to 
+open an interactive shell in the disco chroot, and perform some more interesting
+operations, representing what a more advanced sort of script might do:
+
+ [disco@client disco]$ sudo NOOP=true ./client/bin/disco-sh-shell
+ [root@client /]# rm -f /etc/passwd
+ [root@client /]# grep -v root /etc/shadow | tee tmpfile
+ bin:*:15240:0:99999:7:::
+ daemon:*:15240:0:99999:7:::
+ adm:*:15240:0:99999:7:::
+ lp:*:15240:0:99999:7:::
+ sync:*:15240:0:99999:7:::
+ shutdown:*:15240:0:99999:7:::
+ halt:*:15240:0:99999:7:::
+ mail:*:15240:0:99999:7:::
+ uucp:*:15240:0:99999:7:::
+ operator:*:15240:0:99999:7:::
+ games:*:15240:0:99999:7:::
+ gopher:*:15240:0:99999:7:::
+ ftp:*:15240:0:99999:7:::
+ nobody:*:15240:0:99999:7:::
+ dbus:!!:15324::::::
+ usbmuxd:!!:15324::::::
+ avahi-autoipd:!!:15324::::::
+ vcsa:!!:15324::::::
+ rtkit:!!:15324::::::
+ rpc:!!:15324:0:99999:7:::
+ pulse:!!:15324::::::
+ haldaemon:!!:15324::::::
+ avahi:!!:15324::::::
+ saslauth:!!:15324::::::
+ postfix:!!:15324::::::
+ apache:!!:15324::::::
+ ntp:!!:15324::::::
+ rpcuser:!!:15324::::::
+ nfsnobody:!!:15324::::::
+ gdm:!!:15324::::::
+ sshd:!!:15324::::::
+ tcpdump:!!:15324::::::
+ disco:$6$Hv67bVi.$d/EolMfURGTMbq1hBr1QL2HdYMYxAXvruq550Qqgu2HCOKWQ1YptMghLKvOAgr3h0NwzXZwHpXQ6fVLdpYe.9.:15533:0:99999:7:::
+ discostu:!!:15558:0:99999:7:::
+ [root@client /]# mv tmpfile /etc/shadow
+ mv: overwrite `/etc/shadow'? y
+ [root@client /]# echo LOL > /var/lib/p0wnt
+ bash: /var/lib/p0wnt: restricted: cannot redirect output
+ [root@client /]# echo LOL | tee /var/lib/p0wnt
+ LOL
+ [root@client /]# echo > /bin/myhotbash
+ bash: /bin/myhotbash: restricted: cannot redirect output
+ [root@client /]# touch /bin/myhotbash
+ [root@client /]# exit
+
+... Since that was done inside of the noop shell (where all the scripts and 
+templates run during noop), we can easily report on these activities:
+
+ [disco@client disco]$ sudo NOOP=true ./client/bin/disco-fs-diff
+ info: File: file: /etc/passwd : Deleted
+ 1,35d0
+ < root:x:0:0:root:/root:/bin/bash
+ < bin:x:1:1:bin:/bin:/sbin/nologin
+ < daemon:x:2:2:daemon:/sbin:/sbin/nologin
+ < adm:x:3:4:adm:/var/adm:/sbin/nologin
+ < lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
+ < sync:x:5:0:sync:/sbin:/bin/sync
+ < shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
+ < halt:x:7:0:halt:/sbin:/sbin/halt
+ < mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
+ < uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
+ < operator:x:11:0:operator:/root:/sbin/nologin
+ < games:x:12:100:games:/usr/games:/sbin/nologin
+ < gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
+ < ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
+ < nobody:x:99:99:Nobody:/:/sbin/nologin
+ < dbus:x:81:81:System message bus:/:/sbin/nologin
+ < usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
+ < avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
+ < vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
+ < rtkit:x:499:496:RealtimeKit:/proc:/sbin/nologin
+ < rpc:x:32:32:Rpcbind Daemon:/var/cache/rpcbind:/sbin/nologin
+ < pulse:x:498:495:PulseAudio System Daemon:/var/run/pulse:/sbin/nologin
+ < haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
+ < avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
+ < saslauth:x:497:76:"Saslauthd user":/var/empty/saslauth:/sbin/nologin
+ < postfix:x:89:89::/var/spool/postfix:/sbin/nologin
+ < apache:x:48:48:Apache:/var/www:/sbin/nologin
+ < ntp:x:38:38::/etc/ntp:/sbin/nologin
+ < rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
+ < nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
+ < gdm:x:42:42::/var/lib/gdm:/sbin/nologin
+ < sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
+ < tcpdump:x:72:72::/:/sbin/nologin
+ < disco:x:500:10::/home/disco:/bin/bash
+ < discostu:x:501:501::/home/discostu:/bin/bash
+ info: File: file: /bin/myhotbash : Created : type=[regular empty file] device=[fd00] mode=[81a4] selinux=[?] md5=[d41d8cd98f00b204e9800998ecf8427e] perms=[root:root 644]
+ info: File: file: /etc/shadow : Modified : md5=[8b02f6d00dbcd622f869216bb1dbbbf4 => 336d0b913c8f8cd029964afd00357952] perms=[root:root 0 => root:root 644] mode=[8000 => 81a4]
+ 1d0
+ < root:$6$57kBYzwRrygFb5op$vghIbLjxmkzTznSbN4kA5fdxFsd1ye7WWe/HFtwMJSTBlDuBcOZISgLKNg/xlA4uAFIBi82yAnW/JajgwhCXY.:15517:0:99999:7:::
+ info: File: file: /root/.bash_history : Modified : md5=[ead812e487da32cb99cebd09ad7f773b => cb0138b7f3c4f48639cafe9f7147413f] selinux=[unconfined_u:object_r:admin_home_t:s0 => ?]
+ 228a229,237
+ > exit
+ > rm -f /etc/passwd
+ > grep -v root /etc/shadow | tee tmpfile
+ > mv tmpfile /etc/shadow
+ > echo LOL > /var/lib/p0wnt
+ > echo LOL | tee /var/lib/p0wnt
+ > echo > /bin/myhotbash
+ > touch /bin/myhotbash
+ > exit
+ info: File: file: /var/lib/p0wnt : Created : type=[regular file] device=[fd00] mode=[81a4] selinux=[?] md5=[5732edd7e4e1240b868e15bc95d36339] perms=[root:root 644]
+ 0a1
+ > LOL
+
+And here we see some more of Disco's rather extensive noop reporting capabilities.
+
+But let's say that this run took longer than we thought it should. What was taking
+so much time? Disco will tell us.
+
+ report: _internal: diff
+ report:    time_real 0.82 : time_user 0.14 : time_sys 0.66
+ report:    mem_avg 0 : mem_max 5184 : mem_faults_major 0 : mem_faults_minor 18218
+ report:    io_fsin 0 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:    exit: 0
+ report: _internal: fetch_params
+ report:    time_real 1.17 : time_user 0.03 : time_sys 0.08
+ report:    mem_avg 0 : mem_max 11136 : mem_faults_major 0 : mem_faults_minor 1718
+ report:    io_fsin 0 : io_fsout 0 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:    exit: 0
+ report: othermodule-3.2: fetch
+ report:    time_real 2.43 : time_user 0.05 : time_sys 0.14
+ report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2881
+ report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:    exit: 0
+ report: othermodule-3.2: template
+ report:    etc/othermodule/stuff.cfg :
+ report:        time_real 0.56 : time_user 0.03 : time_sys 0.37
+ report:        mem_avg 0 : mem_max 4608 : mem_faults_major 122 : mem_faults_minor 4880
+ report:        io_fsin 25536 : io_fsout 16 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:        exit: 0
+ report: testmodule-1.0: exec
+ report:    00-hello.sh :
+ report:        time_real 0.26 : time_user 0.02 : time_sys 0.19
+ report:        mem_avg 0 : mem_max 4608 : mem_faults_major 34 : mem_faults_minor 3773
+ report:        io_fsin 6704 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:        exit: 0
+ report:    10-service_stop.sh :
+ report:        time_real 0.56 : time_user 0.03 : time_sys 0.38
+ report:        mem_avg 0 : mem_max 4608 : mem_faults_major 126 : mem_faults_minor 5233
+ report:        io_fsin 25776 : io_fsout 8 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:        exit: 1
+ report: testmodule-1.0: fetch
+ report:    time_real 2.41 : time_user 0.05 : time_sys 0.15
+ report:    mem_avg 0 : mem_max 11152 : mem_faults_major 0 : mem_faults_minor 2839
+ report:    io_fsin 0 : io_fsout 24 : io_sockin 0 : io_sockout 0 : io_signals 0
+ report:    exit: 0
+
+Happy dancing!

client/bin/disco

@@ -13,7 +13,8 @@ fi
 function colorize() {
     sed s/"^info:\(.*\)"/"${COLOR_CYAN}info:\1${COLOR_NORMAL}"/g |\
         sed s/"^warning: \(.*\)"/"${COLOR_YELLOW}warning: \1${COLOR_NORMAL}"/g |\
-        sed s/"^error: \(.*\)"/"${COLOR_RED}error: \1${COLOR_NORMAL}"/g
+        sed s/"^error: \(.*\)"/"${COLOR_RED}error: \1${COLOR_NORMAL}"/g |\
+        sed s/"^report: \(.*\)"/"${COLOR_CYAN}report: \1${COLOR_NORMAL}"/g
 }
 
 function report() {
@@ -52,9 +53,9 @@ function dance() {
     do
 	echo "info: Processing ${module}"
 	NOOP="true" disco-ball template $module
-	/usr/bin/time -f "$TIME" -o /var/disco/reports/_internal/diff disco-fs-diff
+	NOOP=true /usr/bin/time -f "$TIME" -o /var/disco/reports/_internal/diff disco-fs-diff
 	if [ "$NOOP" == "" ]; then
-	    rsync -aWH /var/disco/testfs/noop/* /
+	    rsync -aWH /var/disco/testfs/noop/scratchfs/. /.
 	fi
 	NOOP="$NOOP" disco-ball exec $module
 	RETVAL=$?

client/bin/disco-fs-diff

@@ -11,6 +11,10 @@ fi
 
 cd $DISCOROOT
 
+CREATED=0
+DELETED=0
+MODIFIED=0
+
 rsync --checksum --times --perms --owner --group -ani ./scratchfs/ ./rootfs/ |\
     grep -v ".unionfs.*/$" |\
     sed s/"\.\/scratchfs"/""/ |\