Thanks @Hironsan for this usefull repo.
I can't seem to find a way to add new users to an existing project form the dashboard.
If there's a quick way to do it programmatically pls let me know.

Other nice to have's:

• ui for creating new users, or instruction how to do it from shell
• always add superuser as a user when creating a project

1, what's the format of txt files?
2, after upload the dataset, but can't find it in this project.

any suggestion?

Thank you for this beautiful tools!

I'm attempting to delete a project, but it's always that the latest project was deleted.

Hi,

I would like to able to format e.g. document classification content e.g. by means of HTML syntax.
Currently all content gets escaped.

Is there any quick remedy to this?

I could not find any way to navigate between items using keyboard shortcuts. Maybe it is a good idea to add them? For example arrow keys for previous/next document on the list.

Admin user can create project and upload dataset for other normal user, but can not add labels in this project.

as title

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

System information

• Windows 10
• Python 3.6.5

Describe the problem

For text classification, after I uploaded the text documents and defined the labels per the instruction, I clicked the annotate data tab, but it only showed an empty page, any clues what went wrong?

Source code / logs

This is a snapshot of what is shown when I clicked annotate data tab

Hello,

We have a huge collection of documents to be annotated. We'd like to use doccano to work on this collaboratively. Is it possible to have a button "next unannotated document"?

Thanks,
Jose

System information

• ** Windows 10**:
• ** Python 3.6**:

I've cloned the repo, the code is running, but I'm struggling with adding data to my interface. Could you provide more infos on how to add data (format, labels, etc) and bind it to the template, please ?

Thank you.

I found some useful links:
prodi.gy
Chinese-Annotator

The prodigy using active learning structure, and automatic push some hard samples to be labeled (This is very difficult).
The Chinese-Annotator using the same structure of prodigy, but it's no longer development.

I am very interested in the active learning parts, is there any possible to write some todo list or development plan？

I'm attempting to add in linebreak filters to allow newlines to be rendered within documents to complement #17 however I can't seem to get them working.

I've identified the line which inserts the document text to be line 34 of document_classification.html
https://github.com/chakki-works/doccano/blob/master/app/server/templates/annotation/document_classification.html#L34

However modifying it to [[ docs[pageNumber].text|linebreaks ]] does not produce the desired effect.

On a side note I am not very familiar with Django, what is the significance of the double square-bracket syntax? I have not seen it in any Django examples. Aren't variables typically inserted with double curly-brackets in templates?

How to solve this?

Describe the problem

In order to facilitate smarter labeling, it would be great if there were methods for re-ordering the documents based on specific criteria. Some basic orders might be:

• max( abs(annotation.prob - 0.5)) asc
  • this would order by the least certain predicted annotations
• document insertion date
• most/least recently updated (manual) annotation

Source code / logs

https://github.com/chakki-works/doccano/blob/master/app/server/api.py#L102
Seems like if a few more things got checked in the get_queryset def here, we could potentially allow you to either

1. set sort-order at a project level
2. re-sort using a button

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10
• Python version: python 3.6

Describe the problem

I migrate on mysql cause an error:
django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'text' used in key specification without a key length")

Source code / logs

(venv) D:\project\doccano-master\app>python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, server, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying server.0001_initial...Traceback (most recent call last):
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\mysql\base.py", line 71, in execute
return self.cursor.execute(query, args)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\cursors.py", line 198, in execute
res = self._query(query)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\cursors.py", line 304, in _query
db.query(q)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\connections.py", line 217, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1170, "BLOB/TEXT column 'text' used in key specification without a key length")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management_init_.py", line 381, in execute_from_command_line
utility.execute()
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management_init_.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management\base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management\base.py", line 353, in execute
output = self.handle(*args, **options)
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management\base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "D:\project\doccano-master\venv\lib\site-packages\django\core\management\commands\migrate.py", line 203, in handle
fake_initial=fake_initial,
File "D:\project\doccano-master\venv\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\migrations\executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\migrations\migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\migrations\operations\models.py", line 514, in database_forwards
getattr(new_model._meta, self.option_name, set()),
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\base\schema.py", line 360, in alter_unique_together
self.execute(self._create_unique_sql(model, columns))
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\base\schema.py", line 133, in execute
cursor.execute(sql, params)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 100, in execute
return super().execute(sql, params)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "D:\project\doccano-master\venv\lib\site-packages\django\db\backends\mysql\base.py", line 71, in execute
return self.cursor.execute(query, args)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\cursors.py", line 198, in execute
res = self._query(query)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\cursors.py", line 304, in _query
db.query(q)
File "D:\project\doccano-master\venv\lib\site-packages\MySQLdb\connections.py", line 217, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'text' used in key specification without a key length")

The project does not work without internet connection.
Is it possible to have an argument so the project works offline?
Or is the code easily editable for that purpose?

Note:
I have not checked the source code yet.

System information

• Debian GNU/Linux 9.5:
• Python 3.6.7 :: Anaconda, Inc.:
• Mozilla Firefox 60.3.0
• Chromium 69.0.3497.92 built on Debian 9.5, running on Debian 9.5

Describe the problem

Internet connection on

The project works fine with Firefox and with Chromium.

Internet connection off

The project does not work any longer with Firefox and with Chromium.

According to your README.md, requirements are:

• Python 3.6+
• django 2.0.5+
• Google Chrome

Since this uses supposedly only Python, I don't see any reason why the project should target:

• unpkg.com
• cloudflare

while it is open in browser.

Source code / logs

I don't think logs will give any insight but just in case, here they are.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• Python version: 3.6

How to bind the ip address to 0.0.0.0 such that I call the application remotely within the same private network.

Thanks

I am going to use this tool, but I have some questions, I wanna edit the text during labeling , is it allowed in this tool ?

Hi,
First of all, I must say that doccano is pretty good.
But I come with problem: doccano becomes unusable, when I need to annotate any document with more than 26 labels (example task: classification of hand-written country names: englind -> england, usa -> united_states_of_america and so on), because label can't be created without shortcut, label shortcut can't be reused and shortcuts map contains only lowercased ascii characters.

Thanks.

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

System information

• ** Linux Ubuntu 16.04 **:
• anaconda 3.7.1:

Describe the problem

Bug: Can not annotations like live demo

Source code / logs

Internal Server Error: /api/projects/1/docs/1/annotations/
Traceback (most recent call last):
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: main.server_label__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/views.py", line 483, in dispatch
response = self.handle_exception(exc)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/views.py", line 443, in handle_exception
self.raise_uncaught_exception(exc)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/views.py", line 480, in dispatch
response = handler(request, *args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/generics.py", line 244, in post
return self.create(request, *args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/mixins.py", line 21, in create
self.perform_create(serializer)
File "/data1/user/quangnd/doccano/app/server/api.py", line 134, in perform_create
serializer.save(document=doc, user=self.request.user)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "/data1/user/quangnd/doccano/app/server/serializers.py", line 59, in create
annotation = SequenceAnnotation.objects.create(**validated_data)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/query.py", line 413, in create
obj.save(force_insert=True, using=self.db)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/base.py", line 718, in save
force_update=force_update, update_fields=update_fields)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/base.py", line 748, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/base.py", line 831, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/base.py", line 869, in _do_insert
using=using, raw=raw)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/query.py", line 1136, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
cursor.execute(sql, params)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/data1/user/quangnd/anaconda3/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: main.server_label__old

Is it possible to use doccano interactively, i.e. have it exchange data with another service, for example with a python program (in isolation or inside a jupyter notebook) through API.

The use case I am envisioning is to connect doccano to a spacy NLP pipeline, where the annotated sequences from spacy NLP processing are fed inside doccano for control and annotation, allowing the user to correct mistakes in the assignments (e.g. NER assignment or POS tagging) and whenever a new sentence is confirmed by the user, it is sent back to the pipeline controller which uses it as part of dynamic training of the internal model to improve it's efficiency.

In other words using doccano for a semi supervised learning with interactive machine teaching.

If yes, please explain a little how to proceed about integrating and data exchange. Right now I only could see it exporting the performed annotations as csv in IOB format.

django_heroku is not listed in requirements.txt
So, when running manage.py, script raises error django_heroku module not found.

Feature request

Thanks for a great tool, feels really promising and is fast to work with.

However for me it is really lacking a feature to go to a specific page when annotating a sequence. Say that I have 2000 sequences I want to annotate with entities, if I want to go back to sequence 1000 I have to manually go through each page. This is time consuming and somewhat error prone as one might miss to annotate some sequences.

System information

• Linux Ubuntu 18.04:
• Python version :: 3.6.7:

Describe the problem

Getting the attached exception when i try to install this in my local as per the steps
This exception i am getting when i try to annotate text.

Source code / logs

[25/Dec/2018 04:35:39] "GET /static/images/favicons/favicon-32x32.png HTTP/1.1" 200 2379
Internal Server Error: /api/projects/1/docs/1/annotations/
Traceback (most recent call last):
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: main.server_label__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/views.py", line 483, in dispatch
response = self.handle_exception(exc)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/views.py", line 443, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/views.py", line 480, in dispatch
response = handler(request, *args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/generics.py", line 244, in post
return self.create(request, *args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/mixins.py", line 21, in create
self.perform_create(serializer)
File "/home/vidya/Documents/py_workspace/doccano/app/server/api.py", line 134, in perform_create
serializer.save(document=doc, user=self.request.user)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "/home/vidya/Documents/py_workspace/doccano/app/server/serializers.py", line 59, in create
annotation = SequenceAnnotation.objects.create(**validated_data)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/query.py", line 413, in create
obj.save(force_insert=True, using=self.db)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
force_update=force_update, update_fields=update_fields)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
using=using, raw=raw)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
cursor.execute(sql, params)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/vidya/installs/anaconda3/envs/job-desc/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: main.server_label__old
[25/Dec/2018 04:35:46] "POST /api/projects/1/docs/1/annotations/ HTTP/1.1" 500 22875

Hi,

I want to make an api call to POST /projects/<project_id>/docs/create to create a document classification dataset, using Postman for example. However, I get an error saying that csrf verification failed. Can you please let me know how can I make a POST call if I need to make this call from some other program when doccano is running?

Thanks

Feature Request

When in a Sequence Labeling project, it would be nice to be able to export the data in a format like spacy's training format.

("Uber blew through $1 million a week", {'entities': [(0, 4, 'ORG')]}),
("Google rebrands its business apps", {'entities': [(0, 6, "ORG")]})

Describe the problem

This feature would make the file we're downloading a lot smaller and also make offline entity labeling pipelines a lot faster.

Describe the problem

It would be very useful to be able to delete a document from its annotation page. This would be great for datasets where you haven't/can't do robust validation before uploading.

Source code / logs

I assume this is more complicated than a DELETE to the Document endpoint, but I'm not sure. It might make sense if this was an admin-only button, but I could see a use case where part of an annotator's job was to clean the dataset.

I do notice the auto-labelling #18 is under developing. However, I want to quickly have this function for my own development.
I followed the auto-labelling branch and did some changes. I am not familiar with Django and I have a problem saving the prediction results to Database.
Here is my AutoLabeling API. Looking forward to any suggestion and help. Very much appreciate it!

Source code / logs

class AutoLabelling(generics.RetrieveUpdateDestroyAPIView):

permission_classes = (IsAuthenticated, IsProjectUser)

def get_serializer_class(self):
    project = get_object_or_404(Project, pk=self.kwargs['project_id'])
    self.serializer_class = project.get_annotation_serializer()
    return self.serializer_class

def get_annotation_class(self):
    project = get_object_or_404(Project, pk=self.kwargs['project_id'])
    self.annotation_class = project.get_annotation_class()
    return self.annotation_class

def get_queryset(self):
    '''
    Format:
        {'id': 47, 'prob': 0.0, 'manual': False, 'user_id': 1,
        'document_id': 1, 'label_id': 1, 'start_offset': 11, 'end_offset': 17}
    '''
    document = get_object_or_404(Document, pk=self.kwargs['doc_id'])
    queryset = document.get_annotations()
    queryset = queryset.filter(user=self.request.user)
    return queryset

def get(self, request, *args, **kwargs):
    """ Reinitiate with a list of predicted entities.

    1. Get the preded_annotations from NERModel
    2. Delete the old annotations in database
    3. Store the prediced results
    """
    project = get_object_or_404(Project, pk=self.kwargs['project_id'])
    document = project.documents.get(id=self.kwargs['doc_id'])

    # Delete the annotations
    queryset = self.get_queryset()
    queryset.delete()

    preded_annotations = [
        {"label": 1, "start_offset": 34, "end_offset": 41},
        {"label": 2, "start_offset": 3, "end_offset": 10}
    ]

    # Question:
    # how to save the database with new annotations
    # can I use the AnnotationList to save here?

    return Response(preded_annotations)

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): centos7
• Python version: python 3.6
  server ngnix,uwsgi

Describe the problem

I load 600w short text （nearly 100 MB）for classification，when I request the annotate page ,the response is slow,so that return a 404。and it comsume 100% cpu. and nearly 3g memory.how could I refine its performance, is optimizing database useful?

For the semantic similarity of two sentence. Is there any possible to label sentence pairs？
Using '，' to separate two sentence seems possible, but it not very beautiful. How to using two line to show the sentence ?

Hi,
I'm trying to annotate a sentence using different labels fro the same set of words.
For example, for the sentence:
"the dog jumps on the skateboard"
I want the following labels:
the dog -> subject
jumps -> verb
dog -> animal
the dog jumps -> action
on the skateboard -> where
skateboard -> object

I tried to create a sequence labeling project, but seems like when I select a label for a word, I can't label that word or a sentence that include that word again.
Is there a workaround to obtain what I need?

Hello, I just started using this code but I have a problem.
After I use Sequence Labeling and export data, it comes tagged character by character. Let me explain with an example.
Instead of this:
188721, Sodali, B-Person
188721, Ayran, I-Person
I get,
188721, S, B-Person
188721, o, I-Person
etc. etc.

Am I missing something?

http://doccano.herokuapp.com/demo/text-classification/

Uncaught ReferenceError: module is not defined

Clicking on Delete button under project name in the dashboard doesn't do anything.

Ubuntu 16.04, python 3.6

My documents need to have newlines? How can I create a dataset with multiple documents with newlines?

Feature Request: import labeled data sets in BIO format. Like:

SOCCER	O
-	O
JAPAN	B-LOC
GET	O
LUCKY	O
WIN	O
,	O
CHINA	B-PER
IN	O
SURPRISE	O
DEFEAT	O
.	O

Nadim	B-PER
Ladki	I-PER

AL-AIN	B-LOC
,	O
United	B-LOC
Arab	I-LOC
Emirates	I-LOC
1996-12-06 O

Btw, I love your tool, thanks for doing it open source

Currently in order to upload a dataset for annotation all of the documents in the dataset must be in a single txt file with each document separated by a newline character like so:

document 1 text
document 2 text

It would be very helpful if instead documents could be within separate files and a bulk upload could be performed. For example:

Document 1

Document
1
text

Document 2

Document
2
text

This would also allow documents to contain newline characters which was requested here: #12

Love the tool!

System information

• Linux Ubuntu 16.04
• Python version: 3.6

Describe the problem

Bug report.
I follow the tutorial and try to annotate few simple entities such as Location, Person, etc.
I upload the sample data(text format, each line is a sentence without line break).
Two bugs are both related to django database (sqlite3 in my case) issue.

1. I cannot delete the labels I created in Label editor section.
2. I cannot annotate entity in Annotate Data section.

I have spent few hours reading the source code and trying to debug the issue. However, I didn't find the solution. I printed out the some logs. The query indicates a insertion into table 'server_sequenceannotation', however, the error shows django.db.utils.OperationalError: no such table: main.server_label__old which doesnot exist in the sqlite3 tables. The 'using' parameter in the traceback error is 'default'. I suspect this might be the reason but I am not very sure.
Please help with any suggestion. Very much appreciate it!

Source code / logs

Traceback for case1:

Traceback (most recent call last):
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 303, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: main.server_document__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 483, in dispatch
response = self.handle_exception(exc)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 443, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 480, in dispatch
response = handler(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/generics.py", line 293, in delete
return self.destroy(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/mixins.py", line 93, in destroy
self.perform_destroy(instance)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/mixins.py", line 97, in perform_destroy
instance.delete()
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/base.py", line 891, in delete
return collector.delete()
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/deletion.py", line 283, in delete
count = qs._raw_delete(using=self.using)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/query.py", line 676, in _raw_delete
return sql.DeleteQuery(self.model).delete_qs(self, using)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/sql/subqueries.py", line 75, in delete_qs
cursor = self.get_compiler(using).execute_sql(CURSOR)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1066, in execute_sql
cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 303, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: main.server_document__old
[27/Dec/2018 04:41:14] "DELETE /api/projects/1/labels/1 HTTP/1.1" 500 21810

Traceback for case2:

Internal Server Error: /api/projects/1/docs/1/annotations/
Traceback (most recent call last):
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 303, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: main.server_label__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 483, in dispatch
response = self.handle_exception(exc)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 443, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/views.py", line 480, in dispatch
response = handler(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/generics.py", line 244, in post
return self.create(request, *args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/mixins.py", line 21, in create
self.perform_create(serializer)
File "/home/chitang/sourcecodes/doccano/app/server/api.py", line 134, in perform_create
serializer.save(document=doc, user=self.request.user)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "/home/chitang/sourcecodes/doccano/app/server/serializers.py", line 59, in create
annotation = SequenceAnnotation.objects.create(**validated_data)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/query.py", line 417, in create
obj.save(force_insert=True, using=self.db)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/base.py", line 729, in save
force_update=force_update, update_fields=update_fields)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/base.py", line 759, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/base.py", line 842, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/base.py", line 880, in _do_insert
using=using, raw=raw)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/query.py", line 1125, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1283, in execute_sql
cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/chitang/anaconda3/envs/py3/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 303, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: main.server_label__old
[27/Dec/2018 04:44:44] "POST /api/projects/1/docs/1/annotations/ HTTP/1.1" 500 23012

Feature Request

It would be nice to not have the db tracked in git

Describe the problem

The fact that the application database is tracked in git is troublesome if you ever want to interact with the repo. For example if a new feature gets implemented and I want to sync to the latest commit on master, I'd need to jump through hoops in order to pull down those changes without merge conflicts in the db.

This could also slow down development in forks (you'd need to reset the db every time you want to PR back into this repo).

System information

• Any OS

Describe the problem

The following problem was reported by a user:

I've accidentally selected almost entire text and annotated it. Now I cannot delete this annotation (pictures below). What could I do? Even if I knew how to redo it (the Ctrl+z does not work) I would not do it since I begun to annotate later texts. What should I do?

Solution

A quick-fix solution is to modify annotation.css as follows:

.content .tag {
  font-size: 16pt;
  height: 1.5em;
  padding-left: .5em;
  padding-right: .5em;
  white-space: unset;  /* Add */
  height: auto;        /* Add */
}

But I think we will check the problem carefully.

Is it possible to have different possible labels for each sentence and this list of labels be uploaded along with the sentences when doing sentence classification?

Describe the problem

It would be nice if it possible to import some examples metadata while importing the dataset. For now it is possible only to import the text, but in my expirience I rarely need only text. I often need some external id, source, etc to be passed into the annotation tool along the text. And it would be nice to have it within my exported dataset.

Currently the database schema for labels does not track when labels are added:

sqlite> .schema server_sequenceannotation
CREATE TABLE "server_sequenceannotation" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "prob" real NOT NULL, "manual" bool NOT NULL, "start_offset" integer NOT NULL, "end_offset" integer NOT NULL, "document_id" integer NOT NULL REFERENCES "server_document" ("id") DEFERRABLE INITIALLY DEFERRED, "label_id" integer NOT NULL REFERENCES "server_label" ("id") DEFERRABLE INITIALLY DEFERRED, "user_id" integer NOT NULL REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE INDEX "server_sequenceannotation_document_id_e1057912" ON "server_sequenceannotation" ("document_id");
CREATE INDEX "server_sequenceannotation_label_id_05d8e2eb" ON "server_sequenceannotation" ("label_id");
CREATE INDEX "server_sequenceannotation_user_id_713fbc2e" ON "server_sequenceannotation" ("user_id");
CREATE UNIQUE INDEX server_sequenceannotation_document_id_user_id_label_id_start_offset_end_offset_b8cc1717_uniq ON "server_sequenceannotation" ("document_id", "user_id", "label_id", "start_offset", "end_offset");

In order to track the speed at which labelling is completed, and compare performance of different labellers on the platform, it would be very useful for the date + time a label was created to be stored in the database (and ideally surfaced in the 'statistics' page of a project).

Hi,

I wish to modify some js files and then regenerate the bundle js files to get the new functionality. Can you please let me know how to go about it?

Thanks
Akhil

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• Python version:

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug or a feature request.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Describe the problem

We have 3 project types: document classification, sequence labeling and seq2seq. All of this types are represented as a single Project class with lots of if-elses on project type. It will be more better to use inheritance and proxy model to represent these polymorphic relations. It will be easier to extend, write and reason about.

This issue was extracted from the comment in this PR #55