anishathalye / dotbot Goto Github PK

Would it be possible when creating a link, to add an option to create the directory path as well, if it doesn't exist?

Currently, dotbot uses os.path.expandvars.
However, in some cases, it may be desireable to follow proper shell parameter substitution. (see: the related manual)
Here is an example usage: I want my zshrc to work. Some systems may define $ZDOTDIR to be $XDG_DATA_HOME/zsh, some may leave it undefined.
The typical solution would be to use ${ZDOTDIR:-$HOME}/.zshrc as the install location. However, without parameter substitution, there is no way to achieve this.

Hey!

I wonder if there's an way to install a directory tree without having to symlink the whole destination directory, only the actual files or directories that exists on the source dir.

Something like this:

~/.config/terminator:
  create: true
  path: config/terminator/*

The problem I'm dealing with is that there are some unmanaged files being created at the directory after program runs, so I need to keep adding this to a .gitignore everytime.

Is it possible to do something like that, am I missing it somewhere?

As a new user I found the README seemed like it was written upside down. I wanted a full example first and I could guess at what most of the syntax meant and then a full explanation when I wanted to customize a bit more. I think the README could be reorganized, streamlining the integration process. I have a few more detailed ideas and can submit a PR with my copy edits. Thoughts?

It'd be nice if yaml version (and any other requirements) were given with precision. As there are alredy various things to think about (did i say mess ;) for dotfiles, git or dotbot's starters who read that dotbot has no external dependencies, so there is no extra work required when setting up new systems. ;)

$ cd ~/.dotfiles
$ git clone <dotbot>
$ vim {.gitignore,.gitmodules,.install.conf.yaml}
$ mv ~/<.dotfile> <dotfile>
$ cp .dotbot/tools/submodules/install .install
$ ./.install
Traceback (most recent call last):
  File "~/.dots/.dotbot/bin/dotbot", line 23, in <module>
    import dotbot
  File "~/.dots/.dotbot/dotbot/__init__.py", line 1, in <module>
    from .cli import main
  File "~/.dots/.dotbot/dotbot/cli.py", line 2, in <module>
    from .config import ConfigReader, ReadingError
  File "~/.dots/.dotbot/dotbot/config.py", line 1, in <module>
  1 # dotbot configuration (yaml)                                                    
    import yaml
ImportError: No module named 'yaml'

Ooh sh*t!

$ pacman -Qs yaml
libyaml 0.1.6-1
python2-yaml 3.11-1

Just like this guy

dotbot ./install did well after I installed python-yaml-3.11-1 :)

dotbot works great, except for that zsh isnt activated until I actually type zsh. Is there a way to get that to automatically be the default shell with dotbot?

(sorry if this is a silly question)

Setting up Dotbot with an existing dotfiles repository is pretty quick.

It would be cool to have a script to make it easier to set up a dotfiles repository with automatic Dotbot integration. The script could be an interactive program that the user guides in creating a git repository, moving existing dotfiles, and setting up an initial install.conf.yaml automatically based on the files that were moved.

The script should be mostly automated, basically only asking the user for confirmation when doing things.

Dotbot, at least with 1.x, will maintain compatibility with both Python 2 and Python 3.
This can sometimes be painful to do, and sometimes, it can be error-prone. In the past, there have been bugs with cross-version compatibility.

It would be good to modify the testing code so that tests are run with both Python 2 and Python 3. Currently, it only tests with Python 2.

Would it make sense to add a flag to only run the clean, link, or shell sections of the config? Sometimes I dont want the shell commands to run so I just comment them out. This could be implemented using flags (dotbot -c/--clean -l/--link -s/--shell), or possibly as "sub-commands" (dotbot {clean,link,shell} or dotbot run {clean,link,shell}) and the default would still be to perform all three tasks.

I also thought it might be useful to add another optional argument to shell configurations to prompt the user if they want the shell command to run or not, something like "confirm: true". Not really sure that would be useful if I could just tell dotbot to only clean and link.

So things like #66 don't happen.

Looks like it's possible: http://www.lewuathe.com/blog/2015/01/25/vagrant-in-travisci/

CC @bchretien

I am in the process of migrating my custom dotfiles install script to Dotbot, and my dotfiles from an old, defunct Mercurial repo to Github. This means that almost all dotfiles I want Dotbot to manage are already symlinked. Naturally I tell Dotbot to force creation of the new symlinks, but this doesn't seem to work for symlinked directories, like ~/.vim:

$ ls .vim
lrwxrwxrwx 1 daff daff 28 Jan  1  2013 .vim -> /home/daff/dotfiles.hg/.vim/

My install.conf.yaml looks like this:

- clean: ['~']

- link:
    ~/.dotfiles: ''
    ~/.tmux.conf:
      force: true
      path: .tmux.conf
    ~/.vim:
      force: true
      path: .vim/
    ~/.vimrc:
      force: true
      path: .vimrc

- shell:
  - [git submodule update --init --recursive, Installing submodules]

When running ./install I get this:

All targets have been cleaned
Creating link ~/.dotfiles -> /home/rz01/daff/dotfiles/
Removing ~/.vimrc
Creating link ~/.vimrc -> /home/rz01/daff/dotfiles/.vimrc
Removing ~/.tmux.conf
Creating link ~/.tmux.conf -> /home/rz01/daff/dotfiles/.tmux.conf
Failed to remove ~/.vim
Incorrect link ~/.vim -> /home/rz01/daff/configs/.vim
Some links were not successfully set up
Installing submodules [git submodule update --init --recursive]
All commands have been executed

==> Some tasks were not executed successfully

The exception linker.py throws is Cannot call rmtree on a symbolic link. It seems the code in https://github.com/anishathalye/dotbot/blob/master/dotbot/executor/linker.py#L74 tries to delete the symlink after dereferencing it and thus tries to delete the target directory, in this case /home/daff/dotfiles.hg/.vim/ which shutil.rmtree doesn't allow. Not sure if I interpreted that correctly.

A simple solution could look like this:

diff --git a/dotbot/executor/linker.py b/dotbot/executor/linker.py
index d94a9bc..9821fe7 100644
--- a/dotbot/executor/linker.py
+++ b/dotbot/executor/linker.py
@@ -78,7 +78,9 @@ class Linker(Executor):
                 (self._exists(path) and not self._is_link(path))):
             fullpath = os.path.expanduser(path)
             try:
-                if os.path.isdir(fullpath):
+                if os.path.islink(fullpath):
+                    os.unlink(fullpath)
+                elif os.path.isdir(fullpath):
                     shutil.rmtree(fullpath)
                 else:
                     os.remove(fullpath)

I'll submit a PR.

This issue is meant for tracking work on Dotbot v2.0, the first major release since v1.0 in March 2014. All features that are backwards compatible are being included in Dotbot v1.0 as they are added. We'll only switch to the name "v2.0" if/when we decide to add features that are not backwards compatible.

Completed features

• Directive plugins (see #65)
• Default options (see #34 and #77)

Accepted features / changes

• Improved error messages (see #73)
• Backup (see #90)

Features under consideration

• CLI flag to only run certain sections (see #44 and #49, 2nd item)
• Platform specific link paths (see #57)
• Be able to write to files/folders that require special privileges (see #49, 3rd item)
• Uninstall command to delete symlinks specified in install config (see #49, 4th item)
• Automatic backup / replacement of dotfiles (see #12 and #36)
• Support interactive execution (see #74)
• Make Dotbot usable as a Python library (see #75)

Rejected features

(none)

Design decisions

(nothing here yet)

To be discussed

• What file format do we want to use? There are a couple options I can think of: JSON/YAML file (declarative), other data format (e.g. HOCON), or Python DSL. Maybe there are other completely different ways to do it too.
• Because people use Dotbot as a submodule, we need to be careful when figuring out how to ship the upgrade. Should it be on the master branch? Should Dotbot 2.0 forever live in a 2.0 branch with master being reserved for bugfixes for 1.0?

Release information

The release date is currently unknown. Work on Dotbot v2.0 has not started yet.

(This top post will be edited as stuff happens. This issue will be used for discussing the design of v2.0.)

I started looking at the 491 dotbot configs I pulled, and I noticed that 6 of them failed to parse. 5 were the kinds of things you would expect, mostly due to the ":" character inside of bare strings, which YAML doesn't like.

One was kind of interesting though, it's from this config file: https://github.com/adambubenicek/dotfiles/blob/master/install.conf.json

The problem is that it looks like he's using tabs for indentation, and YAML doesn't allow tabs. This is pretty much the only significant way that the JSON subset of YAML differs from JSON proper.

It might be good to call out this one difference in the readme, since the readme says that JSON config files are allowed, which isn't 100% accurate, since they're still parsed as YAML. Another option would be to explicitly use a JSON parser for '.json' config files.

It will be good if there is an option that will make dotbot backup the existing files and write over it.

Let's say I would set up my dotfiles repo on another computer, from the config and console output it sounds like it links the files first, then runs the shell commands which would create directories.

It can't link when there's no folder, right? So shouldn't the folders be created first? Otherwise you'd just run the script twice...

Also, I had some additional questions:

• The script doesn't seem to like this - [mkdir -p ~/.local/{bin,share}, Creating local bin and share dirs]

It throws this because it thinks the { is the "end of line" character which should be ]

Could not read config file:
  while parsing a flow sequence
    in "install.conf.yaml", line 26, column 5
  expected ',' or ']', but got '{'
    in "install.conf.yaml", line 26, column 24

Other than that, I just setup my repo using dotbot, and it is awesome so far! Was just a pain in the butt to create folders, move files, write it in the config, run the config... 😛

• How does dotbot handle existing files? For example, when you have a newly setup OS, .bashrc will probably exist already, and dotbot would fail to symlink because it exists.

dotbot works even under windows "as is" if you are able to create symlinks. So using python3 and having the rights to create symlinks you'll be able to use this great tool.

But as "cmd.exe" doesn't support ANSI you will not get colored output. Should be easy to fix using "colorama".

Is output from scripts intentionally not written to the command line?

I might be trying to be too clever. I have a version of Bynens's .osx script that I want to run via shell but allow user input to determine if it should continue because even though it's idempotent, it kills running apps and is excessive to run often. I tried this at the top of the file:

read -p "Run OS X defaults script (Y/n)? " -n 1 -r

if [[ ! $REPLY =~ ^[Y]$ ]]; then
  exit
fi

The script waits for input and correctly receives it, but the message from read is not printed.

I am finally going to put my dotfiles on a git repo, mostly because of backups. I was gonna use your tool for it!

The first question is about the config, though. In the example config, and in other configs I've seen, there's this line:

- link:
    ~/.dotfiles: ''

This causes this output (removed useless lines):

~/.dotfiles already exists but is a regular file or directory
Some links were not successfully set up
==> Some tasks were not executed successfully

Can you explain this? I don't really get it.

Hi.
In the YAML configuration file, if you say force: true, does that currently override the desination no matter what it is? I would like it to overwrite any existing symbolic link, but still be safe in all other ways. This is because I keep different versions of files in my dotbot repository, and several YAML files to install different versions, and would like to switch between them on the fly.

I have a further feature request: Possibility to set the default behaviour so that one can limit the verbosity of the YAML file; i.e. set globally force: true or the like.

Thanks for your consideration.

As I've been switching to a dotfiles repo and the subsequently to dotbot, I ran into cases where I had symlinks to incorrect locations. Dotbot doesn't currently give me any way to resolve these, I just get incorrect link messages in the log.

It would be nice to have some type of force option to fix all of those links.

hi,
I'm not sure if you'd want to consider a new simple feature. Dotbot is fine (great!) the way it is, really.

But suppose you've got an extensive list of entries under - link:, and you'd want to run the install on a fresh, minimal distro. Wouldn't it be useful to check if a program is installed first? This check is also partly included in my amateur configs and I'm working to make them more modular and avoid broken functions and aliases. I'd like to link only the configs I need, no more. EDIT: specifying another config file in install is obviously the simple solution.

Maybe we could have dotbot skip the creation of directories and symlinks based of the condition that if an installed: $1 setting was specified in a list entry and which $1 &>/dev/null (or similar) evaluated false. A separate cleanuninstalled command could be used to also clean symlinks (that point to the dotfiles directory and aren't broken), when the specified program is not in $PATH anymore. And lastly, bin/dotbot could be given an option to ignore this feature.

Note that I don't mind branching out and removing entries from install.conf.yaml, but dotfiles aren't like typical repo's, and changes in this branch might not get merged into master soon. Not without much pain at least.

anyway, thank you for this wonderful tool, I love the installer and the submodule approach.

As I used Dotbot to create my dotfiles repository, I have seen other tools that allows to replace current "real" files by the new symlink and putting them in a backup folder instead of doing nothing is there is already a file. That would be really useful, as I had to it myself manually.

It could be something like :

$ ./install --replace

If you implement it, in the dotfiles_template repository you could add a backup folder with its content gitignored.

Currently, this is what happens when using ./install :

Installing submodules [git submodule update --init --recursive]
Installing vim plugins [vim +PluginInstall +qall]
Installing zsh bundles [source '~/.config/zsh/.zshrc']
Command [source '~/.config/zsh/.zshrc'] failed
Some commands were not successfully executed

==> Some tasks were not executed successfully

when trying out the command itself, no error message appears and the exit code is 0. It would be nice to know why a command fails.

Hello, I've been using dotbot to manage my dotfiles for some time now.
It has worked great thus far, since most applications read from a common configuration directory on all platforms. Sublime Text however, uses different directories depending on what platform you're on.

Would it be feasible to add support for different destination paths depending on your OS?
In Python there's sys.platform which contains a string representing the platform Python is running on.

As for configuration syntax, I'm not sure.
YAML has a construct where you can map keys to values, see example 2.6, but since dotbot maps links "the other way", the syntax wouldn't make sense (and is syntactically incorrect):

- link:
    {
     linux: ~/.config/sublime-text-3,
     darwin: ~/Library/Application Support/Sublime Text 3,
     win32: ~/Sublime Text 3
    }: Sublime Text 3/

If you flip the target and destination, I think it looks decent and you would still be able to configure links independently of the OS/platform. It would obviously break all existing configuration files, however.

- link:
    Sublime Text 3/:
      linux: ~/.config/sublime-text-3,
      darwin: ~/Library/Application Support/Sublime Text 3,
      win32: ~/Sublime Text 3
    vim/: ~/.vim

What do you think?

I got a Linking failed error because I attempted put a link at ~/.vim/plugins.vim, but the ~/.vim directory didn't exist.

Tested Workaround: mkdir ~/.vim

Read the yaml or json file from bash and use ln -s ... to create symlink for windows.
Demo: https://github.com/shivapoudel/symlink

A great way to extend the functionality of dotbot would be to allow user-written plugins. These would allow custom directives alongside link, clean, and shell. I thought of this while looking at what people were doing with dotbot in casey/dotbot-user-study. I had written a dir directive, and csivanich/dotbot adds a new meta directive. The meta directive is used to print headers which separate logically grouped sections of directives.

Adding functionality to support directive plugins would allow users to extend dotbot without maintaining a fork. (Which I and csivanich both currently do.)

They could maintain the plugins in their dotfile repos, or in separate repos which could be pulled in as submodules and used by several dotfile repos.

Probably more users would write custom directives with a clear framework in place to do so, and it's likely that some of these would be interesting or useful enough to be included in dotbot proper, or possibly in a "contrib" folder of the repo.

I think the implementation would be pretty easy, since python allows for dynamically loaded modules.

A good interface would be to allow users to modify their install script and add --plugin=path/to/plugin.py flags to the invocation of dotbot. Dotbot could then load these modules at runtime and allow them to register new directive names.

What do you think? I'd be happy to write a PR.

So I added a simple shell command, [cp ssh -af ~/.ssh, Copying SSH-folder] in order to copy it instead of symlinking. (Long story.)

However, the command fails. I've tried the command in the normal shell, and it works fine. What could be the problem?

I'm not sure how many people use this. It might be nice to add support for the XDG Base Directory Specification so files to link can be specified relative to $XDG_CONFIG_HOME.

I'm not sure what's the best interface for this -- perhaps doing a string replacement in the link target would suffice.

Here is my current install.conf.yaml file:

#!/usr/bin/env bash

- clean: ['~']

set -e

- link:
  ~/.tmux.conf: tmux.conf
  ~/.vim: vim/
  ~/.vimrc: vimrc
  ~/.zsh: zsh/
  ~/.zshrc: zshrc

- shell:
  - [git submodule update --init --recursive, Installing/updating submodules]

Here is my install script:

#!/usr/bin/env bash

CONFIG="install.conf.yaml"
DOTBOT_DIR="dotbot"

DOTBOT_BIN="bin/dotbot"
BASEDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

cd "${BASEDIR}"
git submodule update --init --recursive "${DOTBOT_DIR}"

"${BASEDIR}/${DOTBOT_DIR}/${DOTBOT_BIN}" -d "${BASEDIR}" -c "${CONFIG}" "${@}"

Here is my output:

All targets have been cleaned
An error was encountered while executing action link
Action link not handled
Action ~/.zsh not handled
Action ~/.vimrc not handled
Action ~/.vim not handled
Action ~/.zshrc not handled
Action ~/.tmux.conf not handled
Installing/updating submodules [git submodule update --init --recursive]
All commands have been executed

==> Some tasks were not executed successfully

This happens no matter what. I have created and destroyed multiple vimrc, zshrc, etc. files, I have completely removed Dotbot and re-downloaded it, I have tried using JSON formatting. Nothing I seem to try will let me create the links. I'm not sure what the problem is at this point; any suggestions will be appreciated.

git submodule update --init --recursive doesn't do what the README seems to be suggesting it for because the setup suggests a bare git submodule add for dotbot itself. submodule add checks out the repo with a detached HEAD, which makes subsequent updates do nothing because the submodule is not tracking any branch.

To get the desired behavior you could check out the submodule and point it to the master branch:

# add submodule to track master branch
git submodule add -b master https://github.com/anishathalye/dotbot

# update your submodule
git submodule update --remote

Detailed info: http://stackoverflow.com/a/9189815/368697

Could be great to have the option to specify shell commands to be executed after all other actions, normal shell execution, links and clean.

That way we can execute additional shell commands after we have all the files deployed. For example we could change permissions of some folders not staged on repository.. Compile some package source, and so on.

How would I get prezto going with dotbot? Not possible?

Hi Everyone,
I have just found dotbot. It looks very promising. Here are a few suggestions:

1 in the link section of .install.conf.yaml , it looks like that those infomation are already contained in the path of the configuration files. There is another dotfile manager called homesick (its bash script version called homeshick) that links all file in the dotfiles folder to the corresponding file in home directory. My suggestion is if there is no extra setting for a file or folder, make a default case to avoid those unecessary configurations and make it looks cleaner.

2 if we keep all those settings, is it possible to organise the .install.conf.yaml by applications, for example, you can put all configureion files for emacs in one section, so it is easier to pick setting from others. Better if we can select install or export dotfiles by application names or user defined category.

3 I want to manage all my customised configuration files in the whole system. One example is /etc/apt/sources.list, which can be changed only by root user. Is is possible to add a configuration keyword that defines the permissions for the files or folders? There is etckeeper, but it does too much, I only want to backup setting that I changed explicitly.

4 if the whole system is managed, scanning for dead symlink may take too much time, maybe add a uninstall command, so it clean the symlink and delete configuration in .install.conf.yaml.
Thank you!

Even though Dotbot is a fairly simple piece of software (currently ~400 SLOC), it would be nice to have some tests.

A set of shell scripts or something similar could work well to form a little test suite.

It would cool if you could specify an optional OS type for every entry in the yaml file. I would love to use the same installer for both Linux and OS X.

Would it be possible to add an option to forcibly link a file if the physical file already exists?

I see that the linker.py supports a relink option to update a symbolic link that is not pointing to the correct target. Could you provide a global way to enable relinking of every link in the dotbot config?

This would support a few edges cases when you know you want to let dotbot clobber valid symbolic links.

Was trying to run dotbot on a server where the most recent version of python available is 2.6. The major version check in dotbot fails to work with the following error.

AttributeError: 'tuple' object has no attribute 'major'

There's a nice way of setting up Dotbot with git submodules using the install script.

It would be good to include a similar script for Mercurial-based dotfiles as well.

Hey :)

In your latest commit 47ad7f4 you was adding a print and this breaks the execution on python 3.

It should be removed or improved that it runs on both versions.
https://github.com/anishathalye/dotbot/blob/master/dotbot/config.py#L14

So how do I go on about calling a script on the yaml file with sudo?

Let's say I have a directory and on the root is a directory called scripts, how do I run a script in that directory and prompt for sudo when needed or as an entire execution?

I was trying to set the installation of my vim plugins in the install.conf.yamlfile. Unfortunately, that does not work.

I am using vim-plug to handle vim plugins, so I added vim +PlugInstall +qall as a shell command, but it fails. I believe that is because of this line

ret = subprocess.call(cmd, shell=True, stdin=stdin, stdout=stdout,
                      stderr=stderr, cwd=self._base_directory)

in the commandrunner.py script. A possible solution would be setting stdin=subprocess.PIPE, but I'm not sure if that posses a problem in some scenario.

Any reason for the

- shell:
    - [git submodule update --init --recursive, Installing/updating submodules]

to fail when run? To my knowledge, the only submodule I have installed is Dotbot. It was working for a bit and then it recently started to complain. I did start using prezto for my zsh configuration, so perhaps they have submodules that I can't find when I dig through the paths. I'm just not really sure what would have caused it to fail.

@anishathalye Is this dotbot compatiable with windows 8 or not. If compatiable can you link the example repo.

Project: https://github.com/shivapoudel/dotfiles

I setup my dotbot config to include a trailing slash at the source directory.

Here's a snippet from my yaml config file:

- link:
    ~/.config/fish: fish/
    ~/.config/i3: i3/
    ~/.config/nvim: nvim/

However, after running the script, the links point to the directory with an extra slash.

Here's a snippet of the output from running ls -l ~/.config:

fish -> /home/user/lib/dotfiles/fish//
i3 -> /home/user/lib/dotfiles/i3//
nvim -> /home/user/lib/dotfiles/nvim//

This isn't breaking anything, but it'd be nice to not have extra slashes appear.

One workaround is to not put the trailing slash into the config file, but I like being able to see that I'm linking a directory at a glance from the config file.

See #11

Hello,
I am looking for a way to easily track my dotfiles and I am considering using dotbot. But I would like to know if it has a feature, important to me. In the linking step is it possible to make backup of the existing files? Imaging I change distro or software version and when I deploy my dotfile something happens. I would like to be able to vimdiff the previous (default) file and see what can be happening.
I have been reading the link section but I don't know if this is possible.

Thank you very much.

Hi,

Would it be possible to let dotbot figure out relative instead of absolute symlinks?

Instead of how it looks today:

.zshrc -> /Users/traal/dotfiles/zshrc

It would be much cleaner to have this:

.zshrc -> dotfiles/zshrc

This also makes the directory structure more reliable and portable.

Cheers,

Fredrik Mellström