Giter VIP home page Giter VIP logo

docker-gitlab-mirrors's Introduction

Docker Repository on Quay

gitlab-mirrors for Docker

This is samrocketman/gitlab-mirrors packaged inside a Docker container based on Alpine Linux.

Quick-start

Automated builds are available from quay.io

docker pull quay.io/klowner/gitlab-mirrors:latest

The /config volume serves as the $HOME for the container's user. A few files are required for this container to work.

  • /config/.ssh/config: any custom ssh configuration you need when connecting to the GitLab server. (See the ssh/config example further down the page).
  • /config/.ssh/id_rsa: an SSH private key associated with the GitLab user that will be pushing mirrors
  • /config/private_token: The private API token for the GitLab user, this can be obtained in your GitLab user profile in the web UI.
  • /config/.bashrc: optional add any pre-init for commands, such as starting ssh-agent or adding ssh keys.

You can initialize the /config volume by running the container with the config command.

Test your configuration to verify that SSH keys and SSH configuration is correct

docker run --rm -it \
  -v ${PWD}/config:/config \
  quay.io/klowner/gitlab-mirrors:latest \
  run ssh git.example.com

Add a repo to mirror (this example shows mirroring the pcre2 library)

docker run --rm -it \
  -v "${PWD}/config:/config" \
  -v "${PWD}/mirrors:/data/mirrors" \
  -e GITLAB_MIRROR_GITLAB_USER=mark \
  -e GITLAB_MIRROR_GITLAB_NAMESPACE=Mirrors \
  -e GITLAB_MIRROR_GITLAB_URL=http://git.example.com \
  -e GITLAB_MIRROR_SVN_OPTIONS='-r500:HEAD -T code/trunk/ -t code/tags/' \
  quay.io/klowner/gitlab-mirrors:latest \
  add --svn --project-name pcre2 --mirror svn://vcs.exim.org/pcre2

If all goes well, this will mirror svn://vcs.exim.org/pcre2 to http://git.example.com/Mirrors/pcre2.

I recommend making a short script such as this for running updates, etc. This example also exposes ssh-agent to the container for convenience.

# gitlab-mirror.sh
docker run --rm -i \
  -v $(dirname $SSH_AUTH_SOCK):$(dirname $SSH_AUTH_SOCK) \
  -v "${PWD}/config:/config" \
  -v "${PWD}/mirrors:/data/Mirrors" \
  -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK \
  -e GITLAB_MIRROR_UID=$(id -u) \
  -e GITLAB_MIRROR_GITLAB_USER=mark \
  -e GITLAB_MIRROR_GITLAB_NAMESPACE=Mirrors \
  -e GITLAB_MIRROR_GITLAB_URL=http://git.klowner.com \
  quay.io/klowner/gitlab-mirrors:latest ${@:1}

Then you can

./gitlab-mirror.sh update

Configuration Options

Required parameters

  • GITLAB_MIRROR_GITLAB_NAMESPACE: GitLab namespace that mirrors will be pushed to (default: 'Mirrors')
  • GITLAB_MIRROR_GITLAB_USER: GitLab username to use for pushing to gitlab (default: git)
  • GITLAB_MIRROR_GITLAB_URL: http(s) URL of the GitLab server you'll be pushing mirrors to

Optional parameters

  • GITLAB_MIRROR_USER: the system user that gitlab-mirrors will run as inside the container (default: git)
  • GITLAB_MIRROR_UID: the userid to use for the system user above (default: 1000)
  • GITLAB_MIRROR_HOME: location of system user's home volume, this contains the gitlab private_token and ~/.ssh (default: /config)
  • GITLAB_MIRROR_INSTALL_DIR: location of gitlab-mirrors (default: /opt/gitlab-mirrors)
  • GITLAB_MIRROR_REPO_DIR: location of repositories volume, this should be persistent so you can run update later (default: /data)
  • GITLAB_MIRROR_COLORS: enable color output (default: true)
  • GITLAB_MIRROR_SVN_OPTIONS: additional parameters passed to git-svn (default: '-s')
  • GITLAB_MIRROR_NO_CREATE_SET: Force gitlab-mirrors to not create the gitlab remote so a remote URL must be provided. (superceded by no_remote_set)
  • GITLAB_MIRROR_NO_REMOTE_SET: Force gitlab-mirrors to only allow local remotes only.
  • GITLAB_MIRROR_FORCE_UPDATE: Enable force fetching and pushing. Will overwrite references if upstream force pushed. Applies to git projects only.
  • GITLAB_MIRROR_PRUNE_MIRRORS: If a branch is deleted upstream then only that change will propagate into your GitLab mirror. Applies to git projects only (default: false)
  • GITLAB_MIRROR_HTTP_REMOTE: push to GitLab over http? Otherwise will push projects via SSH (default: false)

These options affect the configuration options for new mirror repositories

  • GITLAB_MIRROR_NEW_ISSUES_ENABLED: (default: false)
  • GITLAB_MIRROR_NEW_WALL_ENABLED: (default: false)
  • GITLAB_MIRROR_NEW_WIKI_ENABLED: (default: false)
  • GITLAB_MIRROR_NEW_SNIPPETS_ENABLED: (default: false)
  • GITLAB_MIRROR_NEW_MERGE_REQUESTS_ENABLED: (default: false)
  • GITLAB_MIRROR_NEW_PUBLIC: (default: false)

Example configuration using docker-compose

Here's an example docker-compose.yml to simplify interacting with docker-gitlab-mirrors.

version: '2'

services:
  gitlab-mirrors:
    image: quay.io/klowner/gitlab-mirrors:latest
    volumes:
      - /srv/gitlab-mirrors/config:/config
      - /srv/gitlab-mirrors/data:/data
    environment:
      - GITLAB_MIRROR_UID=1000
      - GITLAB_MIRROR_GITLAB_USER=mark
      - GITLAB_MIRROR_GITLAB_NAMESPACE=mirrors
      - GITLAB_MIRROR_GITLAB_URL=http://gitlab
    external_links:
      - gitlab:gitlab
    networks:
      - gitlab_front

networks:
  gitlab_front:
    external:
      name: gitlab_front
  • My /config and /data volumes are stored on the docker host's /srv/gitlab-mirrors directory.
  • The external_links specifies the name of my running gitlab container, this matches the GITLAB_MIRROR_GITLAB_URL=http://gitlab.
  • gitlab_front is a named Docker network, this is added so the docker-gitlab-mirrors container can connect to the gitlab container.

Using the above example, you can run the container via docker-compose without the necessity to provide configuration options repeatedly.

From the directory containing your docker-compose.yml

# docker-compose run --rm gitlab-mirrors
Available options:
 mirrors           - Update all mirrors
 add               - Add a new mirror
 delete            - Delete a mirror
 ls                - List registered mirrors
 update <project>  - Update a single mirror
 config            - Populate the /config volume with ~/.ssh and other things
 run <command>     - Run an arbitrary command inside the container

Run some commands!

Example .ssh/config

After running config your /config volume will be populated with some starter files. You can provide additional configuration settings by customizing the /config/.ssh/config. Here's something similar to what I use, you should be able to adapt it to your needs.

Host gitlab.example.com
  Hostname gitlab
  User git
  ForwardAgent yes
  StrictHostKeyChecking no
  IdentityFile ~/.ssh/id_rsa_for_mirror_user

If everything is set up properly, you can use the container's run command to try connecting to your gitlab server via ssh. eg. docker-compose run --rm gitlab-mirrors run ssh gitlab.example.com If all goes well, you should get a "Welcome to GitLab, <your user>!" repsonse.

docker-gitlab-mirrors's People

Contributors

klowner avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

docker-gitlab-mirrors's Issues

There was an unknown issue with manage_gitlab_project.py

when I add mirror ,meet this error, please help, thanks.

docker-compose run --rm gitlab-mirrors add --git --project-name Test --mirror [email protected]:/Test/test.git
Resolving gitlab remote.
Creating new project Test
Traceback (most recent call last):
File "lib/manage_gitlab_project.py", line 105, in
found_project=createproject(project_name)
File "lib/manage_gitlab_project.py", line 78, in createproject
git.add_project(pname,description=description,**project_options)
File "build/bdist.linux-x86_64/egg/gitlab3/init.py", line 184, in fn
File "build/bdist.linux-x86_64/egg/gitlab3/init.py", line 418, in _post
File "build/bdist.linux-x86_64/egg/gitlab3/init.py", line 435, in _request
File "build/bdist.linux-x86_64/egg/gitlab3/init.py", line 411, in _check_status_code
gitlab3.exceptions.MissingRequiredAttribute: URL: http://git.domain.cn/api/v3/projects, Data: {'wall_enabled': 'false', 'namespace_id': '4', 'snippets_enabled': 'false', 'merge_requests_enabled': 'false', 'name': 'Test', 'issues_enabled': 'false', 'wiki_enabled': 'false', 'public': 'false', 'description': 'Mirror of [email protected]:/Test/test.git'}
There was an unknown issue with manage_gitlab_project.py

I'm trying mirror project from remote gitlab to local gitlab
remote gitlab verson:8.8.5
local gitlab verson:8.8.5

Logfile should be exposable to a volume

Sometimes the command fails with
Error: ./update_mirror.sh buildroot (more information in /opt/gitlab-mirror/cron.log)

but now where to fetch the logs. A mount of the /opt/gitlab-mirror is not good since it contains essential stuff and thus the logfile will vanish on exit. So it would be nice to move the logfile to a different position and mount it with a volume.

Investigate GITLAB_MIRROR_NEW_PUBLIC option

In the middle of Ludum Dare this weekend, but just plopping this here as a reminder to check and see why GITLAB_MIRROR_NEW_PUBLIC doesn't work. I probably just missed it.

Second `docker run` always results in 'permission denied'

I've got Gitlab up and running ( meaning I was able to add a new Github remote to my Mirrors group using the gitlab-mirrors.sh script below ).
But every time I try to run another command I get:

ฯŸ ./gitlab-mirrors.sh ls
chown: /data/Mirrors/project-x/objects/pack/pack-dca3292bf5a38f06677b52995ebf594a2b3dd4d5.idx: Permission denied
chown: /data/Mirrors/project-x/objects/pack/pack-dca3292bf5a38f06677b52995ebf594a2b3dd4d5.pack: Permission denied

contents of gitlab-mirrors.sh:

docker run --rm -i \
  -v $(dirname $SSH_AUTH_SOCK):$(dirname $SSH_AUTH_SOCK) \
  -v "${PWD}/config:/config" \
  -v "${PWD}/mirrors:/data/Mirrors" \
  -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK \
  -e GITLAB_MIRROR_GITLAB_UID=1000 \
  -e GITLAB_MIRROR_GITLAB_USER=gitmirror \
  -e GITLAB_MIRROR_GITLAB_NAMESPACE=Mirrors \
  -e GITLAB_MIRROR_GITLAB_URL=http://my-gitlab.com \
  quay.io/klowner/gitlab-mirrors:latest ${@:1}

When I remove those files I can run for example update one more time, but the second run I will see the same problem again.

README.md: incorrect commands in quickstart section?

Hi,

I'm new to docker and I'm trying to use this.

I tried this command with docke 1.7.1

[root@vagrant gitlab]# docker run --rm -it \
   -v ${PWD}:/config:/config \
   quay.io/klowner/gitlab-mirrors:latest \
   run ssh git.example.com

and got "Error response from daemon: invalid mode for volumes-from: /config"

I checked docker engine docs and it really doesn't expect path as third part after last colon. It expects option. The format should be src:dest:options.

the example script doesn't work in cron

First, in the script we shouldn't rely on ${PWD}, we should use the absolute path.
Second, the -t option should be removed so it works in cron. When -t is there, I get the following error when run in cron:

docker run --rm -it -v /opt/gitlab/mirror/config:/config -v /opt/gitlab/mirror/mirrors:/data/Mirrors -e GITLAB_MIRROR_GITLAB_UID=0 -e GITLAB_MIRROR_GITLAB_USER=gitmirror -e GITLAB_MIRROR_GITLAB_NAMESPACE=Mirrors -e GITLAB_MIRROR_GITLAB_URL=**** quay.io/klowner/gitlab-mirrors:latest update test_repo
cannot enable tty mode on non tty input

Typo in Readme

You can provide additional configuration settings by customizing the /data/.ssh/config

should be

You can provide additional configuration settings by customizing the /config/.ssh/config

Publish it to the main docker registry?

Hi,

I noticed that most projects put their container images to the cetral docker registry so in docker-compose.yml you put something like

gitlabmirror:
   image: klowner/gitlab-mirrors:latest

instead of:

gitlabmirror:
  image: quay.io/klowner/gitlab-mirrors:latest

I'm relatively new to docker, so maybe this request doesn't make sense...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.