mytardis / mytardis-app-mydata Goto Github PK
View Code? Open in Web Editor NEWServer-side functionality and data models for MyData
Server-side functionality and data models for MyData
Hi,
Sometimes, we have SSH connection problems due to firewall, misconfiguration and the like. On such occasions, exceptions such as Exception: Exceeded max connections in SshControlMasterPool are thrown, and newly added files won't be uploaded to MyTardis server. Hence, it would be great to have a fault tolerance strategy that ensures data is uploaded to MyTardis server irrespective of such issues.
One of the strategies we could use is resending the data via HTTP POST. It is reasonable to assume that port 80 is opened, and HTTP POST will be successful more often than not.
On similar note, it would also be helpful to notify the facility manager and the MyTardis admin about the failure.
Iman
Folder Structure: User Group / Dataset
Dataset and datafile names are the same, but stopping and re-uploading continues upload of datafiles in dataset folder appended with "-1".
$ ls
201222_M03037_0009_000000000-J42WY 201222_M03037_0009_000000000-J42WY-1
A test in the app 'mydata' fails when run on a MySQL database;
See full text below:
Creating test database for alias 'default'...
S.......E...........................................................................................................S.S.S.S.S.S.S.....................S.............................S.S.S.S.S...S..................-------------------------------------------------------> ---------------
Traceback (most recent call last):
File "/opt/mytardis/develop/tardis/apps/mydata/tests/test_api.py", line 70, in setUp
super(UploaderAppResourceTest, self).setUp()======================================================================
ERROR: test_get_uploader_by_id (tardis.apps.mydata.tests.test_api.UploaderAppResourceTest)
File "/opt/mytardis/develop/tardis/apps/mydata/tests/test_api.py", line 48, in setUp
self.user_profile = UserProfile(user=self.user).save()
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
force_update=force_update, update_fields=update_fields)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
using=using, raw=raw)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/opt/mytardis/develop/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '4' for key 'user_id'")FAILED (SKIP=15, errors=1)
----------------------------------------------------------------------
Ran 195 tests in 22.704s
Destroying test database for alias 'default'...
Hi James
Is there a way to ignore data from unknown users? The current Mydata version uploads any data regardless of the user's status. We have a use case where ignoring data from unknown users is highly desirable.
Thanks
I’m getting the following error when pressing 'Sync Now’:
folders.pyc - 229 - ShowMessageDialog - MainThread - WARNING - Refusing to show message dialog for message "{ "error_message": "(1062, "Duplicate entry 'b929f735-e2d3-11e6-954a-3c07542f560e' for key ‘uuid’\”)",
...
File "/opt/mytardis/develop-cas/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute\n return self.cursor.execute(sql, params)\n\n
File "/opt/mytardis/develop-cas/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute\n return self.cursor.execute(query, args)\n\n
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute\n self.errorhandler(self, exc, value)\n\n
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n raise errorclass, errorvalue\n\nIntegrityError: (1062, "Duplicate entry 'b929f735-e2d3-11e6-954a-3c07542f560e' for key 'uuid'")\n
Any ideas?
If MyData tries to upload a file which was created during a daylight savings time change, it can fail to create the DataFile record, raising an AmbiguousTimeError.
The exception is raised in the obj_create
method of the DataFileAppResource
class in api.py
.
One workaround / solution could be to add an hour in this case, e.g.
replace this:
try:
retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)
except IntegrityError as err:
with this:
from dateutil.parser import parse
from datetime import timedelta
from pytz.exceptions import AmbiguousTimeError
try:
retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)
except AmbiguousTimeError:
# Manipulate bundle.data ...
created_time = bundle.data.get('created_time')
if created_time:
parsed = parse(created_time)
one_hour_later = parsed + timedelta(hours=1)
bundle.data['created_time'] = one_hour_later.isoformat()
modification_time = bundle.data.get('modification_time')
if modification_time:
parsed = parse(modification_time)
one_hour_later = parsed + timedelta(hours=1)
bundle.data['modification_time'] = one_hour_later.isoformat()
retval = super(DataFileAppResource, self).obj_create(bundle, **kwargs)
except IntegrityError as err:
Hi,
I will use a scenario to explain the issue.
Given:
Result:
Suggestion:
MyData should ignore non-MyTardis users and continue uploading datasets of MyTardis users. This is important especially if $DATA_DIRECTORY is a network drive that is accessed by both MyTardis and non-MyTardis users.
Cheers,
Broken since the MyData MyTardis API functionality moved into its own app.
The API extensions provided by this app makes some assumptions about email address uniqueness amongst active users:
The following code will raise an exception (and trigger an Internal Server Error) if there is more than one active user with the same email address:
User.objects.get(email__iexact=user_folder_name, is_active=True)
The most likely scenario where this could happen is if a user logs in with a new authentication method (e.g. OAuth), having previously logged in with an old authentication method (e.g. LDAP), but they don't immediately migrate their account, so the old account remains active.
get
could be changed to filter
, and if multiple matches are found with User.objects.filter(email__iexact=user_folder_name, is_active=True)
, then it could choose the most recent one (largest User ID), or the largest User ID associated with at least one ObjectACL if one exists.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.