swingbit / jira2gitlab Goto Github PK

I'm trying to import some issues of some projects into GitLab using your tool, because the default implementation of GitLab itself doesn't import comments etc. and seems pretty useless this way. The important thing to note is that I don't have JIRA on-prem, but in the Atlassian Cloud. When trying to read users I get the following error message:

root@potsdam:/opt/jira2gitlab# python jira-user-list.py


Get participants of GOERLITZ
[INFO] Loading Jira issues from project GOERLITZ ... 10

[INFO] #1/10 Looking at Jira issue GOERLITZ-1 ...   Traceback (most recent call last):
  File "/opt/jira2gitlab/jira-user-list.py", line 78, in <module>
    project_users(jira_project)
  File "/opt/jira2gitlab/jira-user-list.py", line 67, in project_users
    jira_users.add(issue['fields']['assignee']['name'])
KeyError: 'name'

Invoking the query in the browser works and provides the following object for one of the issues:

"assignee": {
	"self": "https://[...]/rest/api/2/user?accountId=60c[...]ac5",
	"accountId": "60c[...]ac5",
	"avatarUrls": {
		"48x48": "https://[...]FMM-1.png",
		"24x24": "https://[...]FMM-1.png",
		"16x16": "https://[...]FMM-1.png",
		"32x32": "https://[...]FMM-1.png"
	},
	"displayName": "Marek Maćkowiak",
	"active": true,
	"timeZone": "Europe/Berlin",
	"accountType": "atlassian"
},

The objects for author, reporter etc. look the same. So is your tool simply not compatible with that version of the API or Atlassian Cloud in general or ...? What should name contain, some username different from displayName and accountId?

I didn't find anything like such a username in the response. Though, the import dialog of GitLab seems to show displayName only as well and provides a mapping based on that to usernames in GitLab.

With a large number of labels being created, clarity is quickly lost. It would be helpful if the label colors could be customized. For example, one color can be defined for all priorities-labels or all components-labels. This would increase the clarity - especially for cases where you work without a prefix.

Hello! I am hoping to use your code to import our old JIRA iissues into GitLab. There are a few of these out there, but yours feels like it's the most mature and feature-rich one out there.

I have the project in gitlab set up, the access token set up- with "project owner" rights, and "admin" rights within gitlab.

But something is still preventing the script from actually creating the issue(s). Wondering if you have an idea and can assist?

This is how far I'm getting (below). It's reading all the jira issues, and even finding and mapping the admin rights of the two users found in the first issue I have in JIRA. But when it actually goes to CREATE the issue, I am getting a weird access denied issue:

[root@butler jira2gitlab-main]# ./jira2gitlab.py

Migrating ITS to kent-test-group/it-support
[INFO] Loading Jira issues from project ITS ... 2311 <==== successfully reads all the jira issues

[INFO] #1/2311 Migrating Jira issue ITS-1 ... data: {'created_at': '2007-09-20T10:58:15.000-0500', 'assignee_ids': ['btaylor'], 'title': '[ITS-1] Directory creation request for rgddev and rgd servers', 'description': 'We need a directory to publish web sites for labs.rgd.mcw.edu and labsdev.rgd.mcw.edu . May I suggest the creation of the following directories on rgddev.mcw.edu and rgd.mcw.edu machines for publishing these sites to: \n /usr/local/apache2/labs_htdocs \nowned by the user apache. \nthanks \nGeorge\n\n___\n\nImported from Jira issue ITS-1\n\n', 'milestone_id': None, 'labels': 'jira-import, T::task, P::trivial, unix request'} ...
Traceback (most recent call last):
File "./jira2gitlab.py", line 569, in migrate_project
gl_issue.raise_for_status()
File "/usr/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://gitlab.ctsi.mcw.edu/api/v4/projects/10/issues

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./jira2gitlab.py", line 882, in
migrate_project(jira_project, gitlab_project)
File "./jira2gitlab.py", line 572, in migrate_project
raise Exception(f"Unable to create Gitlab issue for Jira issue {issue['key']}\n{e}")
Exception: Unable to create Gitlab issue for Jira issue ITS-1
403 Client Error: Forbidden for url: https://gitlab.ctsi.mcw.edu/api/v4/projects/10/issues

Migration failed

Resetting user privileges..

• User gkowalsk was made admin during the import to set the correct timestamps. Turning it back to non-admin.
• User btaylor was made admin during the import to set the correct timestamps. Turning it back to non-admin.

Hi!

Many thanks for this project. In order to use the project in a legally compliant manner, a licence valid for the entire project is required. Could you please add a license? If ok for you, an open license like MIT or Apache License 2.0 would be helpful.

Best regards, Mirjan

First of all, thank you for the script. We just migrated 1900 issues from Jira to GitLab.

That said, we had to make some adjustments including bug fixes to it:

• connection problems: increased default retries and turned off keep alive

# increase the number of retry connections
requests.adapters.DEFAULT_RETRIES = 10

# close redundant connections
# requests uses the urllib3 library, the default http connection is keep-alive, requests set False to close.
s = requests.session()
s.keep_alive = False

• "Sudo" was not correct in headers. It must be "Sudo" and not "SUDO" (see here)

headers={'PRIVATE-TOKEN': GITLAB_TOKEN,'Sudo': resolve_login(author, "unknown", "file_info")},

• story points are configured in the config file but not handled in the script itself. hence, added:

Note: only add weight to data dict if

        weight = None
...
        # storypoints / weight
        if JIRA_STORY_POINTS_FIELD in issue['fields'] and issue['fields'][JIRA_STORY_POINTS_FIELD]:
            # print(f"weight: {issue['fields'][JIRA_STORY_POINTS_FIELD]}")
            weight = int(issue['fields'][JIRA_STORY_POINTS_FIELD])
...
            if weight is not None:
                data['weight'] = weight

• Jira titles can be longer than 255 characters but not in GitLab (it's poorly documented and I finally saw the error message in api_json.log)

            title = f"{issue['fields']['summary']}"
            original_title = ""

            if (len(title) > 255):
                # add full original title as a comment later on
                original_title = f"Full original title: {title}\n\n"
                title = title[:252] + '...'

            data={
                'created_at': issue['fields']['created'],
                'assignee_ids': [gl_assignee],
                'title': title,
                'description': original_title + gl_description,
                'milestone_id': milestone_id,
                'labels': ", ".join(labels),
            }
            if weight is not None:
                data['weight'] = weight

            gl_issue = requests.post(
                f"{GITLAB_API}/projects/{gitlab_project_id}/issues",
                headers={'PRIVATE-TOKEN': GITLAB_TOKEN,'Sudo': resolve_login(reporter, issue['key'], "reporter")},
                verify=VERIFY_SSL_CERTIFICATE,
                data=data
            )
            gl_issue.raise_for_status()

• set updated_at when closing "done" issues

            # Close "done" issues
            # status-category can only be "new" (To Do) / "indeterminate" (In Progress) / "done" (Done) / "undefined" (Undefined)
            if issue['fields']['status']['statusCategory']['key'] == "done":
                status = requests.put(
                    f"{GITLAB_API}/projects/{gitlab_project_id}/issues/{gl_issue['iid']}",
                    headers={'PRIVATE-TOKEN': GITLAB_TOKEN,'Sudo': resolve_login(author, issue['key'], "sudo_close")},
                    verify=VERIFY_SSL_CERTIFICATE,
                    data={
                        'state_event': 'close',
                        'updated_at': issue['fields']['resolutiondate'],
                    }
                )
                status.raise_for_status()

• handle "issue X causes issue Y"

       # Only "outward" links were collected.
        # I.e. we only need to process (a blocks b), as (b blocked by a) comes implicitly.
        if j_type in ['relates to', 'blocks', 'causes']:
            # Gitlab free only support "relates_to" links
            gl_type = 'relates_to'

            if GITLAB_PREMIUM and j_type in ['relates to', 'blocks']:
                gl_type = j_type.replace(' ', '_')

            try:
                gl_link = requests.post(
                    f"{GITLAB_API}/projects/{gl_from['project_id']}/issues/{gl_from['iid']}/links",
                    headers={'PRIVATE-TOKEN': GITLAB_TOKEN},
                    verify=VERIFY_SSL_CERTIFICATE,
                    data={
                        'target_project_id': gl_to['project_id'],
                        'target_issue_iid': gl_to['iid'],
                        'link_type': gl_type,
                    }
                )
                gl_link.raise_for_status()

If I run the queries that are being used in the script using Postman, I receive data.

I am testing this out on a small project in Jira that has only one issue. Just to see how things map across in GitLab before going onto my more involved projects.

[INFO] #1/1 Migrating Jira issue ACT-1 ...   Traceback (most recent call last):
  File "/home/jared/jira2gitlab-main/./jira2gitlab.py", line 914, in <module>
    migrate_project(jira_project, gitlab_project)
  File "/home/jared/jira2gitlab-main/./jira2gitlab.py", line 507, in migrate_project
    ).json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


Migration failed


Resetting user privileges..

The script as written (when allowed to) temporarily grants admin rights to users so that timestamps can be set properly. Clearly needed, and a cool way to handle this.

Unfortunately, the script does not check for what rights the user HAD prior to the import. If a user had admin rights beforehand, that user is incorrectly reset to non--admin when the script finishes.

Great work, thank you for sharing it!

Maybe I missed something but Request participants doesn't seem to be supported currently in the script.

This feature is not exactly equivalent on GitLab, but the /cc @username feature is probably the closest thing:
https://gitlab.com/gitlab-org/gitlab/-/issues/197288

Any thoughts about this? Either appending the cc's to the bottom of the description or adding as a comment might work, though I haven't looked at the traffic yet to see exactly what is happening when I CC a user.

Cheers!

if there is more than 20 results in a GET from a gitlab API you only get the 20 first results ...
you can extend the page up to 100 but after that you have to use the pages to get all results.

I get the following error when invoking jira-user-list.py, while the same URL works when manually copied and executed in a browser.

401 Client Error: Unauthorized for url: https://riff-systemhaus.atlassian.net/rest/api/2/search?jql=project=%22GOERLITZ%22%20ORDER%20BY%20key&fields=*navigable,attachment,comment,worklog&maxResults=100&startAt=0

Tested the same with curl -u '[...]:[...]' [...] and received the following output:

Basic authentication with passwords is deprecated. For more information, see: https://developer.atlassian.com/cloud/confluence/deprecation-notice-basic-auth/

That link contains the following sentence, perfectly fine corresponding to my first error:

For the following APIs and pages, all requests using basic authentication with a non-API token credential will return 401 (Unauthorized) after the deprecation period:

Looking at the code, basic auth with username+password seems to be sued, so the error makes sense. OTOH, the script seems to be created after the deprecation?

The deprecation period for this functionality has ended. From June 3rd, 2019, we will be progressively disabling the usage of this authentication method.

Would be great if you could have a look at this problem. Thanks!

I do not know how or even why this happened, but I have discovered some old issues we have in Jira somehow got changed to a REPORTER identifier of "Anonymous". And some of these issues were reported by user(s) that actually do still exist in our Jira environment.

When running the jira2gitlab import script and it hits such an issue (Reporter = "Anonymous"), I get the followwing error:

[INFO] #1006/2307 Migrating Jira issue ITS-1155 ... Traceback (most recent call last):
File "./jira2gitlab.py", line 884, in
migrate_project(jira_project, gitlab_project)
File "./jira2gitlab.py", line 446, in migrate_project
reporter = issue['fields']['reporter']['name']
TypeError: 'NoneType' object is not subscriptable

Migration failed