csf-ngs / hinkskalle Goto Github PK
View Code? Open in Web Editor NEWOCI + Singularity Container Registry
License: MIT License
OCI + Singularity Container Registry
License: MIT License
Please review the newly written specification for the singularity library protocol at https://github.com/singularityhub/library-api based on your knowledge of the API, and consider volunteering to be a maintainer as described in singularityhub/library-api#1.
Thanks!
setting for SINGULARITY_FLAVOR=(apptainer|singularity)
to switch various display/instructions
Push works fine for me with smaller images (~100MB) But I get NumericValueOutOfRange exception for files of a few gigabytes.
How can I expand the capacity?
After fixing the "Hinkskalle.uitl" python error I now get:
api_1 | error: exec: "./script/start.sh": stat ./script/start.sh: no such file or directory
I use the plain docker-compose.yml from share/deploy
and commented out all LDAP related stuff from the environment files. I set the system up by:
docker-compose up -d hinkdb
docker-compose run --rm api flask db upgrade
docker-compose run --rm api flask localdb add-user -u admin -p admin -e [email protected] -f Admin -l Hase --admin
docker-compose up
try to guess media type on blob push (user-agent singularity?)
If the display is too narrow, e.g. when the browser only fills half of the screen, the (green) sidebar is not accessible.
pip3 install '.[dev]'
requires postgres client libraries
start postgres container once for db init, run initial migrations
make it so postgres uses jsonb while sqlite still works?
enable groups, handle permissions
local/ldap group membership
create groups for users?
When we upload a large image (around 14 GB), hinkskalle returns a HTTP 500 and we get the following output in the log:
hinkdb_1 | 2023-01-24 12:50:34.094 UTC [47] ERROR: integer out of range
hinkdb_1 | 2023-01-24 12:50:34.094 UTC [47] STATEMENT: INSERT INTO image_upload_url (id, "expiresAt", path, size, md5sum, sha256sum, state, type, "partNumber", "totalParts", "createdAt", "createdBy", parent_id, image_id) VALUES ('35b54ce4-2a61-41c9-83f9-25713125b8f8', '2023-01-24T12:55:34.094330'::timestamp, '/mnt/images/_tmp/tmp61ddtvmq', 14068015104, NULL, NULL, 'initialized', 'multipart', NULL, 210, '2023-01-24T12:50:34.094355'::timestamp, 'koehlerm', NULL, 35)
api_1 | [2023-01-24 12:50:34,097] ERROR in app:1449 | Exception on /v2/imagefile/35/_multipart [POST]
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
api_1 | self.dialect.do_execute(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
api_1 | cursor.execute(statement, parameters)
api_1 | psycopg2.errors.NumericValueOutOfRange: integer out of range
api_1 |
api_1 |
api_1 | The above exception was the direct cause of the following exception:
api_1 |
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request
api_1 | rv = self.dispatch_request()
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request
api_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask_rebar/rebar.py", line 141, in wrapped
api_1 | rv = f(*args, **kwargs)
api_1 | File "/srv/hinkskalle/backend/Hinkskalle/routes/imagefiles.py", line 231, in push_image_v2_multi_init
api_1 | db.session.commit()
api_1 | File "<string>", line 2, in commit
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit
api_1 | self._transaction.commit(_to_root=self.future)
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit
api_1 | self._prepare_impl()
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
api_1 | self.session.flush()
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3429, in flush
api_1 | self._flush(objects)
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3569, in _flush
api_1 | transaction.rollback(_capture_exception=True)
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
api_1 | compat.raise_(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
api_1 | raise exception
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3529, in _flush
api_1 | flush_context.execute()
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
api_1 | rec.execute(self)
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
api_1 | util.preloaded.orm_persistence.save_obj(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
api_1 | _emit_insert_statements(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements
api_1 | result = connection._execute_20(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
api_1 | return meth(self, args_10style, kwargs_10style, execution_options)
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
api_1 | return connection._execute_clauseelement(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
api_1 | ret = self._execute_context(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
api_1 | self._handle_dbapi_exception(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
api_1 | util.raise_(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
api_1 | raise exception
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
api_1 | self.dialect.do_execute(
api_1 | File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
api_1 | cursor.execute(statement, parameters)
api_1 | sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range
api_1 |
api_1 | [SQL: INSERT INTO image_upload_url (id, "expiresAt", path, size, md5sum, sha256sum, state, type, "partNumber", "totalParts", "createdAt", "createdBy", parent_id, image_id) VALUES (%(id)s, %(expiresAt)s, %(path)s, %(size)s, %(md5sum)s, %(sha256sum)s, %(state)s, %(type)s, %(partNumber)s, %(totalParts)s, %(createdAt)s, %(createdBy)s, %(parent_id)s, %(image_id)s)]
api_1 | [parameters: {'id': '35b54ce4-2a61-41c9-83f9-25713125b8f8', 'expiresAt': datetime.datetime(2023, 1, 24, 12, 55, 34, 94330), 'path': '/mnt/images/_tmp/tmp61ddtvmq', 'size': 14068015104, 'md5sum': None, 'sha256sum': None, 'state': 'initialized', 'type': 'multipart', 'partNumber': None, 'totalParts': 210, 'createdAt': datetime.datetime(2023, 1, 24, 12, 50, 34, 94355), 'createdBy': 'koehlerm', 'parent_id': None, 'image_id': 35}]
api_1 | (Background on this error at: https://sqlalche.me/e/14/9h9h)
Running Hinkskalle 4.6.0 upgrade from 4.4.x.
Some details:
In models/Image.py the size
column is created as
size = db.Column(db.BigInteger())
but looking at the table in the postgres db we get:
hinkskalle=# \d image_upload_url
Table "public.image_upload_url"
Column | Type | Collation | Nullable | Default
------------+-----------------------------+-----------+----------+---------
id | character varying | | not null |
expiresAt | timestamp without time zone | | |
path | character varying | | not null |
size | integer | | |
md5sum | character varying | | |
sha256sum | character varying | | |
state | character varying | | |
createdAt | timestamp without time zone | | |
createdBy | character varying | | |
image_id | integer | | not null |
type | upload_types | | |
parent_id | character varying | | |
partNumber | integer | | |
totalParts | integer | | |
This results in a 4 byte integer column.
The reason seems that during the update to 4.6. the db migrations are not executed. For now, I did it manually:
docker exec -it hinkskalle_hinkdb_1 psql -U XXXXX -d hinkskalle
alter table image_upload_url alter column size TYPE BIGINT;
alter table image alter column size TYPE BIGINT;
Either we're missing something during upgrade or the migration does not work properly.
@h3kker I think, a short "upgrade" section in the docs would be nice.
In the current containers, downloaded by docker compose, the follwoing error is thrown:
# flask db upgrade
Usage: flask db upgrade [OPTIONS] [REVISION]
Try 'flask db upgrade --help' for help.
Error: While importing 'Hinkskalle', an ImportError was raised:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/cli.py", line 234, in locate_app
__import__(module_name)
File "/usr/local/lib/python3.8/dist-packages/Hinkskalle/__init__.py", line 17, in <module>
from Hinkskalle.util.swagger import register_authenticators
ModuleNotFoundError: No module named 'Hinkskalle.util'
disable login for some accounts (token/API only)
updates should be checked, though!
when mount is lost: pull clears uploaded, location -> not reversible
upload unused images to azure cold storage
or something.
[ ] download date
[ ] maintenance routine to move unused
[ ] set Image.location to storage url, e.g. https://myaccount.blob.core.windows.net/mycontainer/myblob
[ ] would need to wrap send_file for pulls. can singularity/oras/... do redirects?
[ ] maintenance/command to move back to local storage
https://pypi.org/project/cloudstorage/ seems dubious (documentation?)
In order to assist (or not confuse) users. one should activate either the Singularity
or the Apptainer
tab on the home screen depending on the setting of HINKSKALLE_SINGULARITY_COMMAND
.
Since I got hinkskalle running and the first experiments were statisfying I started integrating it in our infrastructure. Thereby, the LDAP sync allows us to login with our normal usersnames (something like lastname+x), hinkskalle uses lowercase(givenname.lastname)
, which is not intuitive in our setup. The reason for this is the following line:
in my installation, I changed it to:
user.username = _get_attr(attrs.get('uid'))
Since everybody has a bit different ldap installation, this should be configurable.
replace drone
possibly also symlink maintenance in same job #9
Hard to reproduce, but this keeps popping up at vbcf after some push (running v4.3.1):
api_1 | [2022-09-23 09:01:51,044] ERROR in app:1455 | Exception on /v1/latest [GET]
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask_rebar/compat.py", line 44, in dump
api_1 | result = schema.dump(data)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1 | result = self._serialize(processed_obj, many=many)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1 | value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 338, in serialize
api_1 | return self._serialize(value, attr, obj, **kwargs)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 634, in _serialize
api_1 | return schema.dump(nested_obj, many=many)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1 | result = self._serialize(processed_obj, many=many)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 514, in _serialize
api_1 | return [
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 515, in <listcomp>
api_1 | self._serialize(d, many=False)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1 | value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 338, in serialize
api_1 | return self._serialize(value, attr, obj, **kwargs)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 634, in _serialize
api_1 | return schema.dump(nested_obj, many=many)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1 | result = self._serialize(processed_obj, many=many)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1 | value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 330, in serialize
api_1 | value = self.get_value(obj, attr, accessor=accessor)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 260, in get_value
api_1 | return accessor_func(obj, check_key, default)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 479, in get_attribute
api_1 | return get_value(obj, attr, default)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/utils.py", line 241, in get_value
api_1 | return _get_value_for_key(obj, key, default)
api_1 | File "/usr/local/lib/python3.8/dist-packages/marshmallow/utils.py", line 255, in _get_value_for_key
api_1 | return getattr(obj, key, default)
api_1 | File "/srv/hinkskalle/backend/Hinkskalle/models/Container.py", line 174, in imageTags
api_1 | raise Exception(f"Tag {[tag.name](http://tag.name/)} has multiple architectures")
api_1 | Exception: Tag 0.21.0-asm294v2 has multiple architectures
api_1 | During handling of the above exception, another exception occurred:
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request
api_1 | rv = self.dispatch_request()
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request
api_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask_rebar/rebar.py", line 164, in wrapped
api_1 | marshaled = marshal(data=data, schema=schema)
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask_rebar/utils/request_utils.py", line 91, in marshal
api_1 | return compat.dump(schema=schema, data=data)
api_1 | File "/usr/local/lib/python3.8/dist-packages/flask_rebar/compat.py", line 48, in dump
api_1 | raise marshmallow.ValidationError(str(e))
api_1 | marshmallow.exceptions.ValidationError: Tag 0.21.0-asm294v2 has multiple architectures
After running hinkskalle now for two months, we found a nice to have enhancement. It would be great if we can set a default quota for users and groups on their first login or when the group is created.
should store tokens salted+hashed
after update to marshmallow 3.x dump_only fields are not accepted any more (instead of ignored)
swagger-to-typescript and swagger-to-python should not include those fields in serialize*
link handling should be done on image:
on tag route. also run on oci push for singularity media type?
trigger link maintenance route
link maintenance on all images -> rq job
could be folded into #5
mark tags as readonly -> cannot remove image, cannot change tag to different image, ...
origin validation fails with trailing /:
api_1 | webauthn.helpers.exceptions.InvalidRegistrationResponse: Unexpected client data origin "https://kubel.testha.se", expected "https://kubel.testha.se/"
sync groups and membership
grant read or read/write
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.