Hi James

Is there a way to ignore data from unknown users? The current Mydata version uploads any data regardless of the user's status. We have a use case where ignoring data from unknown users is highly desirable.

Thanks

Broken since the MyData MyTardis API functionality moved into its own app.

I’m getting the following error when pressing 'Sync Now’:

• Error appears in .MyData_debug_log.txt
• Data files upload and verify fine via the web portal.
• No filters or patterns have been set.

folders.pyc - 229 - ShowMessageDialog - MainThread - WARNING - Refusing to show message dialog for message "{ "error_message": "(1062, "Duplicate entry 'b929f735-e2d3-11e6-954a-3c07542f560e' for key ‘uuid’\”)",
...
File "/opt/mytardis/develop-cas/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute\n return self.cursor.execute(sql, params)\n\n
File "/opt/mytardis/develop-cas/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute\n return self.cursor.execute(query, args)\n\n
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute\n self.errorhandler(self, exc, value)\n\n
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n raise errorclass, errorvalue\n\nIntegrityError: (1062, "Duplicate entry 'b929f735-e2d3-11e6-954a-3c07542f560e' for key 'uuid'")\n

Any ideas?

A test in the app 'mydata' fails when run on a MySQL database;

• tardis/apps/mydata/tests/test_api.py", line 70, in setUp
  super(UploaderAppResourceTest, self).setUp()
• IntegrityError: (1062, "Duplicate entry '4' for key 'user_id'")

See full text below:

Creating test database for alias 'default'...

S.......E...........................................................................................................S.S.S.S.S.S.S.....................S.............................S.S.S.S.S...S..................-------------------------------------------------------> ---------------
Traceback (most recent call last):
File "/opt/mytardis/develop/tardis/apps/mydata/tests/test_api.py", line 70, in setUp
super(UploaderAppResourceTest, self).setUp()

======================================================================
ERROR: test_get_uploader_by_id (tardis.apps.mydata.tests.test_api.UploaderAppResourceTest)
File "/opt/mytardis/develop/tardis/apps/mydata/tests/test_api.py", line 48, in setUp
self.user_profile = UserProfile(user=self.user).save()
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
force_update=force_update, update_fields=update_fields)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
using=using, raw=raw)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '4' for key 'user_id'")

FAILED (SKIP=15, errors=1)
----------------------------------------------------------------------
Ran 195 tests in 22.704s
Destroying test database for alias 'default'...

The API extensions provided by this app makes some assumptions about email address uniqueness amongst active users:

The following code will raise an exception (and trigger an Internal Server Error) if there is more than one active user with the same email address:

User.objects.get(email__iexact=user_folder_name, is_active=True)

The most likely scenario where this could happen is if a user logs in with a new authentication method (e.g. OAuth), having previously logged in with an old authentication method (e.g. LDAP), but they don't immediately migrate their account, so the old account remains active.

get could be changed to filter, and if multiple matches are found with User.objects.filter(email__iexact=user_folder_name, is_active=True), then it could choose the most recent one (largest User ID), or the largest User ID associated with at least one ObjectACL if one exists.

Hi,

I will use a scenario to explain the issue.

Given:

• DATA_DIRECTORY is a path to datasets
• MyData is configured to have a folder structure of "Username/Dataset". This folder structure resides under $DATA_DIRECTORY
• The users that are listed under $DATA_DIRECTORY are James, Martha, Peter and Richard.
• James, Martha and Richard are MyTardis users but Peter is not.

Result:

• MyData will upload datasets of James and Martha. But it will not upload Richard's datasets. This is because MyData stops uploading as soon as it finds a non-MyTardis user, which is Peter in this scenario.

Suggestion:
MyData should ignore non-MyTardis users and continue uploading datasets of MyTardis users. This is important especially if $DATA_DIRECTORY is a network drive that is accessed by both MyTardis and non-MyTardis users.

Cheers,

If MyData tries to upload a file which was created during a daylight savings time change, it can fail to create the DataFile record, raising an AmbiguousTimeError.

The exception is raised in the obj_create method of the DataFileAppResource class in api.py.

One workaround / solution could be to add an hour in this case, e.g.

replace this:

        try:
            retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)
        except IntegrityError as err:

with this:

        from dateutil.parser import parse
        from datetime import timedelta
        from pytz.exceptions import AmbiguousTimeError

        try:
            retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)
        except AmbiguousTimeError:
            # Manipulate bundle.data ...
            created_time = bundle.data.get('created_time')
            if created_time:
                parsed = parse(created_time)
                one_hour_later = parsed + timedelta(hours=1)
                bundle.data['created_time'] = one_hour_later.isoformat()
            modification_time = bundle.data.get('modification_time')
            if modification_time:
                parsed = parse(modification_time)
                one_hour_later = parsed + timedelta(hours=1)
                bundle.data['modification_time'] = one_hour_later.isoformat()
            retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)    
        except IntegrityError as err:

Hi,

Sometimes, we have SSH connection problems due to firewall, misconfiguration and the like. On such occasions, exceptions such as Exception: Exceeded max connections in SshControlMasterPool are thrown, and newly added files won't be uploaded to MyTardis server. Hence, it would be great to have a fault tolerance strategy that ensures data is uploaded to MyTardis server irrespective of such issues.

One of the strategies we could use is resending the data via HTTP POST. It is reasonable to assume that port 80 is opened, and HTTP POST will be successful more often than not.

On similar note, it would also be helpful to notify the facility manager and the MyTardis admin about the failure.

Iman

Folder Structure: User Group / Dataset

Dataset and datafile names are the same, but stopping and re-uploading continues upload of datafiles in dataset folder appended with "-1".

$ ls
201222_M03037_0009_000000000-J42WY  201222_M03037_0009_000000000-J42WY-1