csf-ngs / hinkskalle Goto Github PK

OCI + Singularity Container Registry

License: MIT License

Dockerfile 0.27% Python 61.46% Smarty 0.05% Mako 0.04% Shell 0.41% JavaScript 0.08% Vue 16.29% TypeScript 20.34% Jinja 1.06%

docker singularity registry server oci oras

hinkskalle's People

Contributors

Stargazers

Watchers

Forkers

pedroalvesbatista

hinkskalle's Issues

Please review new library API specification

Please review the newly written specification for the singularity library protocol at https://github.com/singularityhub/library-api based on your knowledge of the API, and consider volunteering to be a maintainer as described in singularityhub/library-api#1.

Thanks!

singularity flavour switch

setting for SINGULARITY_FLAVOR=(apptainer|singularity) to switch various display/instructions

Max Image size

Push works fine for me with smaller images (~100MB) But I get NumericValueOutOfRange exception for files of a few gigabytes.
How can I expand the capacity?

error: exec: "./script/start.sh": stat ./script/start.sh: no such file or directory

After fixing the "Hinkskalle.uitl" python error I now get:

api_1           | error: exec: "./script/start.sh": stat ./script/start.sh: no such file or directory

I use the plain docker-compose.yml from share/deploy and commented out all LDAP related stuff from the environment files. I set the system up by:

docker-compose up -d hinkdb
docker-compose run --rm api flask db upgrade
docker-compose run --rm api flask localdb add-user   -u admin   -p admin   -e [email protected]   -f Admin -l Hase   --admin
docker-compose up

run signature check on manifest push? #50

try to guess media type on blob push (user-agent singularity?)

oci conformance tests fail at the moment

Sidebar not accessible on narrow screens

If the display is too narrow, e.g. when the browser only fills half of the screen, the (green) sidebar is not accessible.

update dev docs

pip3 install '.[dev]' requires postgres client libraries

start postgres container once for db init, run initial migrations

json field in postgres #39

make it so postgres uses jsonb while sqlite still works?

groups #17

enable groups, handle permissions

local/ldap group membership

create groups for users?

run conformance tests in ci pipeline

Integer out of Range when uploading large Images

When we upload a large image (around 14 GB), hinkskalle returns a HTTP 500 and we get the following output in the log:

hinkdb_1        | 2023-01-24 12:50:34.094 UTC [47] ERROR:  integer out of range
hinkdb_1        | 2023-01-24 12:50:34.094 UTC [47] STATEMENT:  INSERT INTO image_upload_url (id, "expiresAt", path, size, md5sum, sha256sum, state, type, "partNumber", "totalParts", "createdAt", "createdBy", parent_id, image_id) VALUES ('35b54ce4-2a61-41c9-83f9-25713125b8f8', '2023-01-24T12:55:34.094330'::timestamp, '/mnt/images/_tmp/tmp61ddtvmq', 14068015104, NULL, NULL, 'initialized', 'multipart', NULL, 210, '2023-01-24T12:50:34.094355'::timestamp, 'koehlerm', NULL, 35)
api_1           | [2023-01-24 12:50:34,097] ERROR in app:1449 | Exception on /v2/imagefile/35/_multipart [POST]
api_1           | Traceback (most recent call last):
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
api_1           |     self.dialect.do_execute(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
api_1           |     cursor.execute(statement, parameters)
api_1           | psycopg2.errors.NumericValueOutOfRange: integer out of range
api_1           | 
api_1           | 
api_1           | The above exception was the direct cause of the following exception:
api_1           | 
api_1           | Traceback (most recent call last):
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request
api_1           |     rv = self.dispatch_request()
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request
api_1           |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask_rebar/rebar.py", line 141, in wrapped
api_1           |     rv = f(*args, **kwargs)
api_1           |   File "/srv/hinkskalle/backend/Hinkskalle/routes/imagefiles.py", line 231, in push_image_v2_multi_init
api_1           |     db.session.commit()
api_1           |   File "<string>", line 2, in commit
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit
api_1           |     self._transaction.commit(_to_root=self.future)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit
api_1           |     self._prepare_impl()
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
api_1           |     self.session.flush()
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3429, in flush
api_1           |     self._flush(objects)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3569, in _flush
api_1           |     transaction.rollback(_capture_exception=True)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
api_1           |     compat.raise_(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
api_1           |     raise exception
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3529, in _flush
api_1           |     flush_context.execute()
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
api_1           |     rec.execute(self)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
api_1           |     util.preloaded.orm_persistence.save_obj(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
api_1           |     _emit_insert_statements(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements
api_1           |     result = connection._execute_20(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
api_1           |     return meth(self, args_10style, kwargs_10style, execution_options)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
api_1           |     return connection._execute_clauseelement(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
api_1           |     ret = self._execute_context(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
api_1           |     self._handle_dbapi_exception(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
api_1           |     util.raise_(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_
api_1           |     raise exception
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
api_1           |     self.dialect.do_execute(
api_1           |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
api_1           |     cursor.execute(statement, parameters)
api_1           | sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range
api_1           | 
api_1           | [SQL: INSERT INTO image_upload_url (id, "expiresAt", path, size, md5sum, sha256sum, state, type, "partNumber", "totalParts", "createdAt", "createdBy", parent_id, image_id) VALUES (%(id)s, %(expiresAt)s, %(path)s, %(size)s, %(md5sum)s, %(sha256sum)s, %(state)s, %(type)s, %(partNumber)s, %(totalParts)s, %(createdAt)s, %(createdBy)s, %(parent_id)s, %(image_id)s)]
api_1           | [parameters: {'id': '35b54ce4-2a61-41c9-83f9-25713125b8f8', 'expiresAt': datetime.datetime(2023, 1, 24, 12, 55, 34, 94330), 'path': '/mnt/images/_tmp/tmp61ddtvmq', 'size': 14068015104, 'md5sum': None, 'sha256sum': None, 'state': 'initialized', 'type': 'multipart', 'partNumber': None, 'totalParts': 210, 'createdAt': datetime.datetime(2023, 1, 24, 12, 50, 34, 94355), 'createdBy': 'koehlerm', 'parent_id': None, 'image_id': 35}]
api_1           | (Background on this error at: https://sqlalche.me/e/14/9h9h)

Running Hinkskalle 4.6.0 upgrade from 4.4.x.

Some details:

In models/Image.py the size column is created as

size = db.Column(db.BigInteger())

but looking at the table in the postgres db we get:

hinkskalle=# \d image_upload_url
                      Table "public.image_upload_url"
   Column   |            Type             | Collation | Nullable | Default 
------------+-----------------------------+-----------+----------+---------
 id         | character varying           |           | not null | 
 expiresAt  | timestamp without time zone |           |          | 
 path       | character varying           |           | not null | 
 size       | integer                     |           |          | 
 md5sum     | character varying           |           |          | 
 sha256sum  | character varying           |           |          | 
 state      | character varying           |           |          | 
 createdAt  | timestamp without time zone |           |          | 
 createdBy  | character varying           |           |          | 
 image_id   | integer                     |           | not null | 
 type       | upload_types                |           |          | 
 parent_id  | character varying           |           |          | 
 partNumber | integer                     |           |          | 
 totalParts | integer                     |           |          |

This results in a 4 byte integer column.

The reason seems that during the update to 4.6. the db migrations are not executed. For now, I did it manually:

docker exec -it hinkskalle_hinkdb_1 psql -U XXXXX -d hinkskalle
alter table image_upload_url alter column size TYPE BIGINT;
alter table image alter column size TYPE BIGINT;

Either we're missing something during upgrade or the migration does not work properly.
@h3kker I think, a short "upgrade" section in the docs would be nice.

ModuleNotFoundError: No module named 'Hinkskalle.util'

In the current containers, downloaded by docker compose, the follwoing error is thrown:

# flask db upgrade
Usage: flask db upgrade [OPTIONS] [REVISION]
Try 'flask db upgrade --help' for help.

Error: While importing 'Hinkskalle', an ImportError was raised:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/cli.py", line 234, in locate_app
    __import__(module_name)
  File "/usr/local/lib/python3.8/dist-packages/Hinkskalle/__init__.py", line 17, in <module>
    from Hinkskalle.util.swagger import register_authenticators
ModuleNotFoundError: No module named 'Hinkskalle.util'

build_frontend container can't find vue-cli-service

service accounts #44

disable login for some accounts (token/API only)

don't apply stricter username rules to dump of older entries

updates should be checked, though!

do not reset uploaded/location if img file not found #66

when mount is lost: pull clears uploaded, location -> not reversible

readonly manifests? #51

archive old images #62

upload unused images to azure cold storage

not download within 1 year
created >6 months ago+never downloaded

or something.

[ ] download date
[ ] maintenance routine to move unused
[ ] set Image.location to storage url, e.g. https://myaccount.blob.core.windows.net/mycontainer/myblob
[ ] would need to wrap send_file for pulls. can singularity/oras/... do redirects?
[ ] maintenance/command to move back to local storage

https://pypi.org/project/cloudstorage/ seems dubious (documentation?)

Idea: Make the default tab on the Home-Screen depending on the set singularity/apptainer command

In order to assist (or not confuse) users. one should activate either the Singularity or the Apptainer tab on the home screen depending on the setting of HINKSKALLE_SINGULARITY_COMMAND.

Make LDAP Sync Better to configure

Since I got hinkskalle running and the first experiments were statisfying I started integrating it in our infrastructure. Thereby, the LDAP sync allows us to login with our normal usersnames (something like lastname+x), hinkskalle uses lowercase(givenname.lastname), which is not intuitive in our setup. The reason for this is the following line:

hinkskalle/backend/Hinkskalle/util/auth/ldap.py

Line 99 in e25a5dc

user.username = slugify(_get_attr(attrs.get('cn')), separator='.')

in my installation, I changed it to:

user.username = _get_attr(attrs.get('uid'))

Since everybody has a bit different ldap installation, this should be configurable.

use circleci for CI

replace drone

image housekeeping job #64

check file exists
check correct size
check hash?
verify signature for singularity

possibly also symlink maintenance in same job #9

Tag has multiple architectures exception

Hard to reproduce, but this keeps popping up at vbcf after some push (running v4.3.1):

api_1           | [2022-09-23 09:01:51,044] ERROR in app:1455 | Exception on /v1/latest [GET]
api_1           | Traceback (most recent call last):
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask_rebar/compat.py", line 44, in dump
api_1           |     result = schema.dump(data)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1           |     result = self._serialize(processed_obj, many=many)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1           |     value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 338, in serialize
api_1           |     return self._serialize(value, attr, obj, **kwargs)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 634, in _serialize
api_1           |     return schema.dump(nested_obj, many=many)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1           |     result = self._serialize(processed_obj, many=many)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 514, in _serialize
api_1           |     return [
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 515, in <listcomp>
api_1           |     self._serialize(d, many=False)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1           |     value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 338, in serialize
api_1           |     return self._serialize(value, attr, obj, **kwargs)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 634, in _serialize
api_1           |     return schema.dump(nested_obj, many=many)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 552, in dump
api_1           |     result = self._serialize(processed_obj, many=many)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 520, in _serialize
api_1           |     value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 330, in serialize
api_1           |     value = self.get_value(obj, attr, accessor=accessor)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/fields.py", line 260, in get_value
api_1           |     return accessor_func(obj, check_key, default)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/schema.py", line 479, in get_attribute
api_1           |     return get_value(obj, attr, default)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/utils.py", line 241, in get_value
api_1           |     return _get_value_for_key(obj, key, default)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/marshmallow/utils.py", line 255, in _get_value_for_key
api_1           |     return getattr(obj, key, default)
api_1           |   File "/srv/hinkskalle/backend/Hinkskalle/models/Container.py", line 174, in imageTags
api_1           |     raise Exception(f"Tag {[tag.name](http://tag.name/)} has multiple architectures")
api_1           | Exception: Tag 0.21.0-asm294v2 has multiple architectures
api_1           | During handling of the above exception, another exception occurred:
api_1           | Traceback (most recent call last):
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request
api_1           |     rv = self.dispatch_request()
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request
api_1           |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask_rebar/rebar.py", line 164, in wrapped
api_1           |     marshaled = marshal(data=data, schema=schema)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask_rebar/utils/request_utils.py", line 91, in marshal
api_1           |     return compat.dump(schema=schema, data=data)
api_1           |   File "/usr/local/lib/python3.8/dist-packages/flask_rebar/compat.py", line 48, in dump
api_1           |     raise marshmallow.ValidationError(str(e))
api_1           | marshmallow.exceptions.ValidationError: Tag 0.21.0-asm294v2 has multiple architectures

gather current links
remove without tag
remove changed tag
add links for current tags

on tag route. also run on oci push for singularity media type?

trigger link maintenance route

link maintenance on all images -> rq job

could be folded into #5

readonly/fixed tags #68

mark tags as readonly -> cannot remove image, cannot change tag to different image, ...

webauthn backend url validate

origin validation fails with trailing /:

api_1           | webauthn.helpers.exceptions.InvalidRegistrationResponse: Unexpected client data origin "https://kubel.testha.se", expected "https://kubel.testha.se/"

ldap group sync

sync groups and membership

readonly tokens #47

grant read or read/write