It's pretty slow with lots of :package:apt/... rules.

@enobrev saw the following error while running as root:

# [blueprint] /usr/local/bin/python2.7 caused EXDEV - try running as root

When a file Blueprint is interested in has a gid that doesn't map to an entry in /etc/group, KeyError is raised. This should be caught and the blueprint should be created with the numeric gid in place of the group name.

The same should happen for uid, too.

That's a waste of time and is very confusing to new users.

When mixing sudo and non-sudo invocations of blueprint(1), this happens:

error: insufficient permission for adding an object to repository database /home/vagrant/.blueprints.git/objects

error: .gitignore: failed to insert into database
fatal: Unable to process path .gitignore
Traceback (most recent call last):
  File "/home/vagrant/work/blueprint/bin/blueprint-create", line 54, in <module>
    b.commit(options.message or '')
  File "/home/vagrant/work/blueprint/blueprint/__init__.py", line 233, in commit
    git.git('update-index', '--add', os.path.abspath('.gitignore'))
  File "/home/vagrant/work/blueprint/blueprint/git.py", line 40, in git
    raise GitError(p.returncode)
blueprint.git.GitError: 128

The likely solution is to drop permissions to $SUDO_UID and $SUDO_GID when forking git(1) processes.

This goes for every package manager as interpreted by the shell code generator.

Default files, should be in packages and ignored.

Ran into this error when running the command 'blueprint-create test'

jcran@disko:~/Desktop/blueprint/blueprint$ blueprint-create test
# [blueprint] searching for APT packages to exclude
No packages found matching installation-report.
No packages found matching linux-generic-pae.
No packages found matching linux-server.
# [blueprint] caching excluded APT packages
# [blueprint] searching for Yum packages to exclude
# [blueprint] searching for APT packages
# [blueprint] searching for configuration files
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 53, in <module>
    b = blueprint.Blueprint(name=name, create=True)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/__init__.py", line 66, in __init__
    getattr(backend, funcname)(self)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/backend/files.py", line 90, in files
    for filename in filenames]]
OSError: [Errno 13] Permission denied: '/etc/ld.so.conf.d/rapid7-nexpose.conf'

In trying to access this file, i received a 'permission denied' error, despite it being world-readable:

jcran@disko:~/Desktop/blueprint$ ls -latr /etc/ld.so.conf.d/rapid7-nexpose.conf
'ls: cannot access /etc/ld.so.conf.d/rapid7-nexpose.conf: Permission denied

jcran@disko:~/Desktop/blueprint$ sudo ls -latr /etc/ld.so.conf.d/rapid7-nexpose.conf
-rw-r--r-- 1 root root 35 2010-12-21 12:45 /etc/ld.so.conf.d/rapid7-nexpose.conf

https://groups.google.com/forum/#!topic/blueprint-users/gSM3iO94uYY discusses using blueprint.Blueprint.__sub__ to implement diffing between two blueprints. The use cases described are driving towards more modular output. Rather than a single monolithic Puppet manifest, the diff represents the customizations on top of a base state.

Quoting Kal McFate, here's a sample workflow:

• kick and puppet a new system
• blueprint create base
• set up system
• blueprint create serverclass
• blueprint diff base serverclass

Did a "$ make clean" out of habit and turns out I lost all the man files. Really minor. It looks like this was intended to clean the gh-pages?

gist of how this behaves on my box: https://gist.github.com/920984

If you want it, can open a pull request for this guy: https://github.com/flores/blueprint/commit/219a7517bbf551ecf06dde81cee10c883dfa825e

Dope software. Excited. Thanks!

Hi,

First things first... this is an AMAZING project ! Not doubt it's about to become the very first package installed on my boxes from now on. Thanks a LOT for it.

Now the request :-P
As I already use etckeeper with Mercurial on my systems, because I tend to favor Python software over others where I can, and so Hg over other DVCSs, and because it would avoid me installing both Hg and Git on all my servers... It'd be great if Blueprint would also support Mercurial.

Hat down for this brilliant idea and thanks a lot again for coding and freeing it
Bests
JT

I get:

[blueprint] searching for software built from source

Traceback (most recent call last):
File "/usr/local/bin/blueprint-create", line 56, in
b = blueprint.Blueprint(name=name, create=True)
File "/opt/lib/python2.6/site-packages/blueprint/init.py", line 72, in init
getattr(backend, funcname)(self)
File "/opt/lib/python2.6/site-packages/blueprint/backend/sources.py", line 40, in sources
os.lchown(dirpath2, s.st_uid, s.st_gid)
OverflowError: signed integer is greater than maximum

When running sudo blueprint create. Presumably related to: http://bugs.python.org/issue1747858 as this is a 64-bit CentOS 4 system with python 2.6 built from source. I worked around this by catching the OverflowError and just calling os.lchown(dirpath2, s.st_uid, -1) directly, not sure what the right solution overall is, maybe nothing.

(And also the part of blueprint-create(1) that acts like blueprint-show(1).)

Both blueprint-create(1) and blueprint-show(1) need a command line option that causes generated code to relax the hard version requirements stored in the blueprint. In shell code, this will amount to removing the version number from the commands. In Puppet code, this will amount to ensure => installed, and in Chef, this will amount to omitting the version method calls.

The purpose of this feature is to enable blueprints to work properly across releases of the OS.

On a factory-fresh Debian install, /usr/bin/python may not exist but /usr/bin/python2.6 does. Look for all of

1. /usr/bin/python
2. /usr/bin/python2.6
3. /usr/bin/python26

before assuming Python's missing.

Many thanks to all the updates of last month! Blueprint seems to be working really good on CentOS 5.5!

On a blueprinted system i have the EPEL and RPMFORGE repositorys enabled. The repositorys are not being restored on a new empty server. Im getting errors when running the shell bootstrap regarding the GPG key and packages which cannot be found in the default CentOS repository.

It would be really nice if it is somehow possible to automate the blueprinting of other repositories as well because right now i have to manually install all the extra repositories and then run bootstrap.sh

The following command returns all available repositories: yum repolist

This is real output from the shell code generator, which will break horribly when executed because the $-prefixed things will be interpolated by the shell applying the blueprint, breaking eventual execution of the file.

cat >"/etc/acpi/actions/power.sh" <<EOF
#!/bin/sh

PATH=/sbin:/bin:/usr/bin

# Get the ID of the first active X11 session:
uid_session=$(
ck-list-sessions | \
awk '
/^Session[0-9]+:$/ { uid = active = x11 = "" ; next }
{ gsub(/'\''/, "", $3) }
$1 == "unix-user" { uid = $3 }
$1 == "active" { active = $3 }
$1 == "x11-display" { x11 = $3 }
active == "TRUE" && x11 != "" {
    print uid
    exit
}')

# Check that there is a power manager, otherwise shut down.
[ "$uid_session" ] &&
ps axo uid,cmd | \
awk '
    $1 == '$uid_session' &&
    ($2 ~ /gnome-power-manager/ || $2 ~ /kpowersave/ ||
     $2 ~ /kded4/ || $3 ~ /guidance-power-manager/) \
        { found = 1; exit }
    END { exit !found }
' ||
  shutdown -h now

EOF

The shell and Puppet code generators need to escape $ with \$. The Chef code generator needs to escape #{ with \#{.

[root@blueprintcentos blueprint]# cat /etc/redhat-release

CentOS release 5.5 (Final)
[root@blueprintcentos blueprint]# python -V

Python 2.4.3
[root@blueprintcentos blueprint]# make install

install -d /usr/local/bin
install
bin/blueprint
bin/blueprint-apply
bin/blueprint-create
bin/blueprint-destroy
bin/blueprint-list
bin/blueprint-show
/usr/local/bin/
install -d /usr/lib/python2.4/site-packages/blueprint/
install -m644
blueprint/chef.py
blueprint/context_managers.py
blueprint/git.py
blueprint/init.py
blueprint/manager.py
blueprint/puppet.py
blueprint/sh.py
/usr/lib/python2.4/site-packages/blueprint/
install -d /usr/lib/python2.4/site-packages/blueprint/backend/
install -m644
blueprint/backend/apt.py
blueprint/backend/files.py
blueprint/backend/gem.py
blueprint/backend/init.py
blueprint/backend/php.py
blueprint/backend/pypi.py
blueprint/backend/sources.py
/usr/lib/python2.4/site-packages/blueprint/backend/
PYTHONPATH=/usr/lib/python2.4/site-packages python -mcompileall
/usr/lib/python2.4/site-packages/blueprint
Listing /usr/lib/python2.4/site-packages/blueprint ...
Compiling /usr/lib/python2.4/site-packages/blueprint/init.py ...
Listing /usr/lib/python2.4/site-packages/blueprint/backend ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/init.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/apt.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/files.py ...
File "/usr/lib/python2.4/site-packages/blueprint/backend/files.py", line 265
return os.path.isdir(pathname) if dir_only else True
^
SyntaxError: invalid syntax

Compiling /usr/lib/python2.4/site-packages/blueprint/backend/gem.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/php.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/pypi.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/backend/sources.py ...
File "/usr/lib/python2.4/site-packages/blueprint/backend/sources.py", line 40
except OSError as e:
^
SyntaxError: invalid syntax

Compiling /usr/lib/python2.4/site-packages/blueprint/chef.py ...
File "/usr/lib/python2.4/site-packages/blueprint/chef.py", line 114
except OSError as e:
^
SyntaxError: invalid syntax

Compiling /usr/lib/python2.4/site-packages/blueprint/context_managers.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/git.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/manager.py ...
Compiling /usr/lib/python2.4/site-packages/blueprint/puppet.py ...
File "/usr/lib/python2.4/site-packages/blueprint/puppet.py", line 161
except OSError as e:
^
SyntaxError: invalid syntax

Compiling /usr/lib/python2.4/site-packages/blueprint/sh.py ...
make: *** [install-lib] Error 1

git --version
git version 1.5.5.6

blueprint create my-first-blueprint
[blueprint] using cached blueprintignore rules
[blueprint] searching for software built from source
[blueprint] /usr/local/bin/python2.7 caused EXDEV - try running as root // NOTE - i am
[blueprint] searching for Ruby gems
[blueprint] searching for Python packages
[blueprint] searching for APT packages
[blueprint] searching for Yum packages
[blueprint] searching for configuration files
[blueprint] searching for PEAR/PECL packages
Traceback (most recent call last):
File "/usr/local/bin/blueprint-create", line 60, in
b.commit(options.message or '')
File "/usr/local/lib/python2.7/site-packages/blueprint/init.py", line 238, in commit
git.init()
File "/usr/local/lib/python2.7/site-packages/blueprint/git.py", line 45, in init
raise GitError(p.returncode)
blueprint.git.GitError: 129

Upgrading Git resolved the issue. I didn't see the requirement anywhere, so I figured I'd announce my confusion.

Thanks for being awesome.

$ sudo make install
...

install -d /usr/local/share/man/man1
install -m644 \
        man/man1/blueprint.1 \
        man/man1/blueprint-apply.1 \
        man/man1/blueprint-create.1 \
        man/man1/blueprint-destroy.1 \
        man/man1/blueprint-list.1 \
        man/man1/blueprint-show.1 \
        /usr/local/share/man/man1/
install: man/man1/blueprint.1: No such file or directory
make: *** [install-man] Error 71

Analogous to #12.

This would entail creating service resources in Puppet/Chef and calling start servicename or /etc/init.d/servicename start in shell.

As reported by CarlFK, it seems like this file should be ignored by default.

I noticed in -S output:

pip install blueprint
pip install blueprint

Generated blueprints should be able to create users and groups based on those present on the system that ran blueprint-create(1).

I had an issue where I had created a fresh copy of a VM in our environment that I was going to blueprint, and I kept getting an GitError: 1 message. I changed the output of subprocess to go to stdout rather than stderr and saw the issue was that I hadn't set my git ident and git was throwing an error saying empty ident not allowed. Blueprint should capture and display this output on an error.

Two users now have reported confusion when using Blueprint along with sudo -i. Environment variables don't explicitly indicate how sudo was invoked but perhaps the presence of the SUDO_COMMAND environment variable can be used to decide the right thing to do.

If someone is running full-time as root, the path to the blueprints repository should be /root/.blueprints.git, not /home/$SUDO_USER/.blueprints.git.

Command : blueprint create -S plop
Output :

   [blueprint] searching for Python packages
   [blueprint] searching for software built from source
   [blueprint] /usr/local caused EPERM - try running as root
   [blueprint] searching for Ruby gems
   [blueprint] searching for PEAR/PECL packages
   [blueprint] searching for apt packages
   [blueprint] searching for apt packages to exclude
   [blueprint] caching excluded apt packages
   [blueprint] searching for configuration files
   Initialized empty Git repository in /home/paul/.blueprints.git/
   Traceback (most recent call last):
     File "/usr/local/bin/blueprint-create", line 58, in <module>
       filename = getattr(b, options.generate)().dumpf()
     File "/usr/local/lib/python2.6/dist-packages/blueprint/__init__.py", line 510, in sh
       s.add('{0} >"{1}" <<{2}', command, pathname, eof)
     File "/usr/local/lib/python2.6/dist-packages/blueprint/sh.py", line 29, in add
       self.out.append(u'{0}\n'.format(s).format(*args))
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 67: ordinal not in range(128)

Python 2.6.6 on Ubuntu 10.10, using latest master.

I installed blueprint on MacOS 10.6.7 and when I try blueprint create I get the following:

Fabian-MacBook-Pro:blueprint fabian$ blueprint create my-first-blueprint

[blueprint] searching for APT packages to exclude

[blueprint] searching for Yum packages to exclude

[blueprint] searching for APT packages

[blueprint] searching for configuration files

[blueprint] searching for Ruby gems

[blueprint] searching for PEAR/PECL packages

[blueprint] searching for Python packages

Traceback (most recent call last):
File "/usr/local/bin/blueprint-create", line 56, in
b = blueprint.Blueprint(name=name, create=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/blueprint/init.py", line 66, in init
getattr(backend, funcname)(self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/blueprint/backend/pypi.py", line 113, in pypi
stderr=subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Fabian-MacBook-Pro:blueprint fabian$

Requested by http://twitter.com/offbytwo/status/52719004506267649.

CentOS 5.6 with Git 1.7.4.1 fails with a strange GitError. Running Git commands directly works as expected.

# [blueprint] searching for software built from source
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for Yum packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for Ruby gems
# [blueprint] searching for Python packages
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 69, in <module>
    filename = getattr(b, options.generate)().dumpf()
  File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 541, in sh
    content = git.content(blob)
  File "/usr/lib/python2.6/site-packages/blueprint/git.py", line 122, in content
    status, stdout = git('show', blob)
  File "/usr/lib/python2.6/site-packages/blueprint/git.py", line 62, in git
    raise GitError(p.returncode)
blueprint.git.GitError: -9

This issue was reported by Sam Weiss in Campfire.

This is similar to the _dpkg_query_S function that finds files in Debian packages. This is the last thing necessary for full RPM-based distro support.

└[~]> sudo blueprint create plop -S
# [blueprint] searching for Python packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for software built from source
# [blueprint] searching for apt packages
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 53, in 
    b = blueprint.Blueprint(name=name, create=True)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/__init__.py", line 67, in __init__
    getattr(backend, funcname)(self)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/backend/apt.py", line 19, in apt
    s = exclusions()
  File "/usr/local/lib/python2.6/dist-packages/blueprint/backend/apt.py", line 50, in exclusions
    package, property = line.rstrip().split()
ValueError: need more than 1 value to unpack

└[~]> python --version             
Python 2.6.5

I ran into an issue where a large tar file included in a generated cookbook caused a chef "knife cookbook upload" to fail due to out of memory error (http://tickets.opscode.com/browse/CHEF-2390), which is not surprising when you consider that the chef server tries to hash the entire request contents including the file in order to authenticate it and we are using a micro instance as our chef server. In the long run though most organizations will probably want to move the tars generated by blueprint to a file host like s3 and link to them in their recipes as remote files, rather than checking them into version control.

Currently backticks (`) output within HereDocs in the shell output are not escaped. This causes the shell script to fail with "Unexpected newline".

Certain /etc/ files have backticks in them, most prevalently Apache conf files, which appear to use the backtick as an 'open quote'. Example:

#   SSL Engine Options:
        #   Set various options for the SSL engine.
        #   o FakeBasicAuth:
        #     Translate the client X.509 into a Basic Authorisation.  This means that
        #     the standard Auth/DBMAuth methods can be used for access control.  The
        #     user name is the `one line' version of the client's X.509 certificate.
        #     Note that no password is obtained from the user. Every entry in the user
        #     file needs this password: `xxdfdfMT32132kVA'.

Keep getting errors when it tries to dump the contents of a UTF-8 configuration file.

Reverting change in #9 makes errors go away.

Traceback (most recent call last):
File "/usr/local/bin/blueprint-create", line 58, in
filename = getattr(b, options.generate)().dumpf()
File "/usr/lib64/python2.7/site-packages/blueprint/sh.py", line 61, in dumpf
f.write(out)
File "/usr/lib64/python2.7/codecs.py", line 692, in write
return self.writer.write(data)
File "/usr/lib64/python2.7/codecs.py", line 352, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 9141: ordinal not in range(128)

Mongodb package installed via apt-get, but modified config file not detected by blueprint, see below:

rm /etc/init/mongodb.conf
cat > /etc/init/mongodb.conf << EOF
description "MongoDB Config Server"

pre-start script
mkdir -p /data/db/mongodb/
chown -R mongodb:mongodb /data
end script

start on startup
stop on shutdown

exec start-stop-daemon --start --quiet --chuid mongodb --exec /usr/bin/mongod -- --configsvr --dbpath /data/db/mongodb --logpath /data/db/mongodb.log --logappend
EOF

Currently, /usr/bin/python is the hard-coded path to the Python interpreter. Provide a means for install-time changes to the path to the interpreter.

caleban reports in Campfire:

root@vm4:~/blueprints# blueprint create tutorial
# [blueprint] searching for software built from source
# [blueprint] searching for apt packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for configuration files
# [blueprint] searching for yum packages
# [blueprint] searching for yum packages to exclude
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 53, in <module>
    b = blueprint.Blueprint(name=name, create=True)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/__init__.py", line 68, in __init__
    getattr(backend, funcname)(self)
  File "/usr/local/lib/python2.6/dist-packages/blueprint/backend/yum.py", line 26, in yum
    s = exclusions()
  File "/usr/local/lib/python2.6/dist-packages/blueprint/backend/yum.py", line 54, in exclusions
    close_fds=True, stdout=subprocess.PIPE)
  File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
sh: getcwd() failed: No such file or directory
sh: getcwd() failed: No such file or directory
root@vm4:~/blueprints#

This is because his server has RPM installed, which allows blueprint.backend.yum to call exclusions which doesn't properly protect its calls to yum and rpm.

This is repeated all over the place. Time to factor it out.

Not sure if this is a bug or simply needs to be documented, but using blueprint on a server with etckeeper installed creates a sizable amount of noise when the entire /etc/.git directory is dumped. It might be valuable to add that directory to the default blueprintignore.

when I try and build my server config for chef, the build fails, and towards the end goes well over a 1gb of memory consumption. any ideas?

app@ip-10-202-57-250:~$ sudo blueprint create -C phobos_server
[sudo] password for app: 
# [blueprint] searching for APT packages to exclude
No packages found matching installation-report.
No packages found matching linux-generic-pae.
No packages found matching language-pack-en.
No packages found matching linux-server.
No packages found matching ubuntu-desktop.
No packages found matching language-pack-gnome-en.
# [blueprint] caching excluded APT packages
# [blueprint] searching for Yum packages to exclude
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for software built from source
# [blueprint] searching for configuration files
# [blueprint] searching for Ruby gems
# [blueprint] searching for APT packages
# [blueprint] searching for Yum packages
# [blueprint] searching for Python packages
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 69, in <module>
    filename = getattr(b, options.generate)().dumpf()
  File "/usr/local/lib/python2.7/dist-packages/blueprint/__init__.py", line 442, in chef
    content = git.content(blob)
  File "/usr/local/lib/python2.7/dist-packages/blueprint/git.py", line 122, in content
    status, stdout = git('show', blob)
  File "/usr/local/lib/python2.7/dist-packages/blueprint/git.py", line 62, in git
    raise GitError(p.returncode)
blueprint.git.GitError: -9

Each blueprint-create(1) call stores the ~/.blueprintignore file that it used. Create a new command or a new output format for blueprint-show(1) that prints that file to standard output.

$ sudo blueprint create test -S
# [blueprint] searching for software built from source
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for configuration files
# [blueprint] searching for Python packages
# [blueprint] searching for apt packages
# [blueprint] searching for Ruby gems
Reinitialized existing Git repository in /home/eric/.blueprints.git/
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 58, in <module>
    filename = getattr(b, options.generate)().dumpf()
  File "/usr/local/lib/python2.6/dist-packages/blueprint/sh.py", line 57, in dumpf
    f2.write(content)
  File "/usr/lib/python2.6/codecs.py", line 691, in write
    return self.writer.write(data)
  File "/usr/lib/python2.6/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 74752: ordinal not in range(128)

Is there some file I can provide for easier debugging?

Graphite installs in /opt and there's basically nothing you can do about it. To properly include such installations, we need to be able to tell blueprint-create to grab that extra directory.

I loathe configuration and setup on CentOS and the like, so this would be excellent to have.

Are there plans for adding support for rpm based systems? Any pointers on where to start?

As Mxx reports in Campfire, these files should be ignored by default.

The content seems static so including the expected MD5 sum would be a good idea.