Giter VIP home page Giter VIP logo

gitlab-migrator's Introduction

🤘 Migrate your private gitlab projects 🤘

When using a private Gitlab instance, you can easily import gitlab projects hosted on gitlab.com.

However, if you are already using a private gitlab instance you are out of luck

This tool is here to help you migrate projects from one private gitlab instance to another.

Usage

This tool is based on fastlane, different actions are implemented in terms of lanes.

List Projects

You can print a list of all projects on the source gitlab with this command:

bundle exec fastlane list_projects

This will also show projects that have already been migrated.

Attention: Keep in mind that projects are matched by name, so if you have migrated a project already and changed its name, it will not show up as migrated!

Migrate a Project

You can migrate a specific project with this command:

bundle exec fastlane migrate project:<project_path>

The project_path is the full path to the project in gitlab (including the namespace), you can take the a path as printed in the list_projects action.

As an example, fastlane migrate project:mexx-uni/pue1 will migrate the project pue1 in the mexx-uni group.

Setup

All necessary dependencies can be installed via bundler.

To install bundler, run

gem install bundler

To install the dependencies via bundler, run

bundle install

Migrating several projects

If you need to migrate several projects at once, you can just create a small bash script that calls this tool for every project that should be migrated.

#!/bin/bash

setopt shwordsplit
PI="namespace/project2 namespace/project2"

for i in $PI; do
	echo "Migrate Project: $i"
	bundle exec fastlane migrate project:$i
done

Github API Credentials

In order to access the gitlab instances, you'll need to provide their endpoint and an API access-token.

For fastlane to be able to access those, you will need have to provide some information in the fastlane/.env file. This repo already contains a file named fastlane/env. Edit this file: enter the gitlab endpoints and API access-tokens there, remove the comments and change the filename to fastlane/.env.

Two pieces of information are necessary for each gitlab instance:

  • FL_GITLAB_ENDPOINT_SRC / FL_GITLAB_ENDPOINT_DST: API endpoint URL, e.g. https://example.net/api/v3
  • FL_GITLAB_TOKEN_SRC / FL_GITLAB_TOKEN_DST: User's private token or OAuth2 access token. How to get your private token

Git Protocol

Optionaly you can choose the protocol to access the git repo (clone, push, etc). Default is ssh.

  • FL_GIT_PROTOCOL_SRC / FL_GIT_PROTOCOL_DST: GIT Protocol, e.g. ssh

Self-Signed SSH Certificates

If you can not connect to your gitlab API because of problems with a self signed SSH Certificate, you can use this to disable certificate verification:

export GITLAB_API_HTTPARTY_OPTIONS="{verify: false}"

I've put this into my ~/.zshrc (because I did not find out how else to set this ENV variable so that it still works in the context of fastlane)

Whats being migrated

Projects

  • description
  • default_branch
  • group
  • namespace
  • wiki_enabled
  • wiki (if enabled)
  • wall_enabled
  • issues_enabled
  • snippets_enabled
  • merge_requests_enabled
  • public
  • deploy keys

Labels

  • name
  • color

Milestones

  • project
  • title
  • description
  • due_date

Issues

  • project
  • title
  • description
  • assignee (on a best effort base)
  • milestone
  • labels
  • notes (see caveats)

Snippets

  • project
  • name
  • filename
  • code
  • notes (see caveats)
  • ❗️author is not migrated
    • author of the new snippet is the user thats performing the migration
  • ❗️visibility level is not migrated
    • snippets will be created with visibility level = 'internal'

Whats not being migrated

  • Merge Requests + Notes
  • System Hooks
  • Users

Repositories

After creating the project in the new gitlab, the git repository from the original source will be cloned and pushed to the new gitlab project

  • all branches
  • all tags

Caveats

Users

Users are not created in the target instance. Where users are necessary (e.g. as group members or issue assignees), existing users are matched by name. It is thus recommended to first create users in the new gitlab instance manually with the same name as in the original gitlab instance.

For users to be matched sucessfully, one of those has to be the same in both githab instances

  • name
  • username

Groups

  • Projects will be created in the same Namespace and for the same Group as in the original Gitlab
  • Namespace and Group are matched by Name
    • Gitlab API is a little bit strange: When creating a group, a group_id and namespace_id are required. Existing project only has a namespace, but namespace are not actuall entities in the gitlab API. It is assumed that group and namespace have a 1:1 relation and group_id = namespace_id, also name and path for groups and namespaces are always equal.
  • Groups will be created when not yet existing
    • Groups will be populated with users on a 'best effort' basis (matched by name).
    • Due to the aforementioned strange behavior of the Gitlab API, only a group will be created (namespace can not be created) and the group_id will also be used as namespace_id for the new project

Issues

  • Issues are assigned to users on a 'best effort' (matched by name).

Notes

  • All notes are posted in the name of the authenticated use who is performing the migration. It is not possible to change the author of a note.
  • Notes are posted with a specific header that notes the original author and the original date of the note

gitlab-migrator's People

Contributors

chraibi avatar innovaptor avatar iv-mexx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

gitlab-migrator's Issues

GitlabCreateProjectAction: Mapping users

In this line

users from source who are found in destination are mapped.

But why is this necessary here? User x wants to merge its own project from source to destination. So why to we need to map user y which has nothing to do with that project?

gitlab (LoadError)

I want to migrate a Gitlab-server to another one. Both running on different Linux-machine.

From my laptop (OS X Yosemite 10.10.5) I try to use gitlab-migrator

I edited the file env as stated in the readme.

Running

fastlane list_projects

fails with the following errors:

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in `require': cannot load such file -- gitlab (LoadError)
        from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in `require'
        from /Users/chraibi/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:6:in `<module:Actions>'
        from /Users/chraibi/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:2:in `<module:Fastlane>'
        from /Users/chraibi/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:1:in `<top (required)>'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in `require'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:126:in `require'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/lib/fastlane/actions/actions_helper.rb:83:in `block in load_external_actions'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/lib/fastlane/actions/actions_helper.rb:82:in `each'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/lib/fastlane/actions/actions_helper.rb:82:in `load_external_actions'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/lib/fastlane.rb:29:in `<module:Fastlane>'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/lib/fastlane.rb:20:in `<top (required)>'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:73:in `require'
        from /Library/Ruby/Gems/2.0.0/gems/fastlane-1.48.0/bin/fastlane:6:in `<top (required)>'
        from /usr/bin/fastlane:23:in `load'
        from /usr/bin/fastlane:23:in `<main>'

My environment:

ruby -v; bundle -v; gem -v

ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
Bundler version 1.11.2
2.0.14

Before starting I run bundle install
Here is the output I get:

Rubygems 2.0.14 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.
Using i18n 0.7.0
Using json 1.8.3
Using minitest 5.8.1
Using thread_safe 0.3.5
Using addressable 2.3.8
Using extlib 0.9.16
Using multi_json 1.11.2
Using babosa 1.0.2
Using colored 1.2
Using highline 1.7.8
Using security 0.1.3
Using excon 0.45.4
Using plist 3.1.0
Using rubyzip 1.1.7
Using multipart-post 2.0.0
Using terminal-table 1.4.5
Using multi_xml 0.5.5
Using coderay 1.1.0
Using method_source 0.8.2
Using slop 3.6.0
Using claide 0.9.1
Using unf_ext 0.0.7.1
Using dotenv 2.0.2
Using mini_magick 4.0.4
Using rouge 1.10.1
Using net-ssh 3.0.1
Using pbxplorer 1.0.0
Using mime-types 2.6.2
Using netrc 0.7.8
Using slack-notifier 1.4.0
Using jwt 1.5.2
Using little-plugger 1.1.4
Using memoist 0.12.0
Using retriable 1.4.1
Using terminal-notifier 1.6.3
Using bundler 1.11.2
Using tzinfo 1.2.2
Using fastimage 1.6.8
Using launchy 2.4.3
Using autoparse 0.3.3
Using commander 4.3.5
Using credentials_manager 0.10.0
Using faraday 0.9.2
Using httparty 0.13.4
Using pry 0.10.3
Using unf 0.1.4
Using xcpretty 0.2.1
Using net-sftp 2.1.2
Using logging 2.0.0
Using activesupport 4.2.4
Using sentry-raven 0.15.2
Using faraday_middleware 0.10.0
Using signet 0.6.1
Using gitlab 3.4.0 from git://github.com/NARKOZ/gitlab.git (at master@ef41dfa)
Using domain_name 0.5.25
Using xcpretty-travis-formatter 0.0.4
Using xcodeproj 0.28.2
Using fastlane_core 0.26.3
Using spaceship 0.14.2
Using krausefx-shenzhen 0.14.6
Using googleauth 0.4.2
Using http-cookie 1.0.2
Using gym 1.1.5
Using scan 0.3.1
Using snapshot 1.1.1
Using cert 1.2.3
Using deliver 1.5.1
Using pem 1.0.1
Using pilot 1.0.1
Using produce 1.0.0
Using sigh 1.1.3
Using xcode-install 1.0.1
Using google-api-client 0.8.6
Using rest-client 1.8.0
Using frameit 2.3.0
Using supply 0.2.1
Using fastlane 1.40.0
Bundle complete! 2 Gemfile dependencies, 77 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

Any idea what went wrong here? In gitlab_create_project.rb:L6 there is a
require 'gitlab', But gitlab is already installed. See:

Using gitlab 3.4.0 from git://github.com/NARKOZ/gitlab.git (at master@ef41dfa)

Edit:
I solved the problem by explicitly installing gitlab:

sudo gem install gitlab

I thought this is already done by bundle install, is it not?

Import Wiki

This Code is amazing! I could migrate my repositories from Gitlab-Server1 to Gitlab-Server2. However, the wiki-pages are not imported. I know these are yet other Git-repos, but would be nice to migrate them too.

Issues are not migrated...

At the moment issues are not migrated (which is the point for me to use gitlab-migrator).

Since this is a feature that is advertised, I wonder what I did wrong...

Migrating to Gitlab.com

Hi,
just wanted to ask if you already have experience with migrating to Gitlab.com.

I just tried to migrate a dummy project. It seems that reading from Gitlab.com is ok, however creating projects hangs.

screen shot 2016-12-20 at 09 26 34

After some debugging I found that this LINE blocks

users_dst = gitlab_dst.users.auto_paginate

Makes sense somehow. Gitlab.com has >100K users. ^^

Therefore, looping over users and groups should be avoided.

Managed to migrate one project, but had to make API calls with curl to get e.g. group id on Gitlab.com and Users id, and do the user mapping manually.

Not so nice. Any thoughts?

Snippets

Any chance to migrate snippets too?

env

Could you please explain the difference between

FL_GITLAB_ENDPOINT_SRC

and

FL_GITLAB_TOKEN_SRC

in the file env?

User contribution Summary

I was just wondering if you already found a way to setup the contribution events for users correctly after merging the projects.

I understand that the activity of a user starts with the minute the project is migrated to the new server, however I could not find in the API documentation
any hints how to modify the activity calendar.

Any thoughts? Maybe when mapping the users copy also their contribution events too?

screen shot 2016-12-15 at 18 25 10

Migrating Snippets

Got the following error while migrating a project with snippets

/usr/local/lib/ruby/gems/2.3.0/gems/gitlab-3.7.0/lib/gitlab/request.rb:80:in `validate': [!] Server responded with code 400, 
message: 400 (Bad request) "file_name" not given. Request URI: https://gitlab.version.fz-juelich.de/api/v3/projects/251/snip$
ets (Gitlab::Error::BadRequest)
        from /usr/local/lib/ruby/gems/2.3.0/gems/gitlab-3.7.0/lib/gitlab/request.rb:49:in `post'
        from /usr/local/lib/ruby/gems/2.3.0/gems/gitlab-3.7.0/lib/gitlab/client/snippets.rb:44:in `create_snippet'
        from /Users/chraibi/Workspace/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:121:in `block in migrate_sni$
pets'
        from /Users/chraibi/Workspace/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:119:in `each'
        from /Users/chraibi/Workspace/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:119:in `migrate_snippets'
        from /Users/chraibi/Workspace/gitlab-migrator/fastlane/actions/gitlab_create_project.rb:54:in `run'
        from /usr/local/lib/ruby/gems/2.3.0/gems/fastlane-2.0.3/fastlane/lib/fastlane/runner.rb:225:in `block (2 levels) ine$ecute_action'

the API suggests the existence of the attribute file_name.

According to the documentation when API-creating snippets file_name is required, however
snippets (created using the web-interface for instance) may have no file_name.

Difference between namespaces and groups

While preparing for the D-day, I found the following problem:

in src_server I have user joe
in dst_server I have user joe1

These are the same user. I can make sure that is the case with the variable user_mapping

Now we want to migrate joe/foo to new server. However, ensure_group() will tell us that the group joe does not exist and will create a group joe.

The fact that the group does not exist may be correct, however I think here the namespace is of kind "user" and not "group".

Therefore, the case

group = gitlab_dst.namespaces.auto_paginate.select { |g| g.id == dst_owner_id and g.kind == 'user'}.first

with

owner_id = original_project.owner.id
dst_owner_id = user_mapping[owner_id]

should be inserted in the else-case.

This worked fine for me, at least.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.