thedatahub / datahub Goto Github PK
View Code? Open in Web Editor NEWDatahub - A standards compliant metadata aggregator platform
License: GNU General Public License v3.0
Datahub - A standards compliant metadata aggregator platform
License: GNU General Public License v3.0
The POST request test uses the old JSON body format to push a record to the Datahub. Like this:
list($crawler, $response, $data) = $this->apiRequest('POST', '/v1/data', [
'format' => $dataConverterId,
'data' => $testData,
]);
Later on, we decided to abandon this approach and go for a new way of POST'ing data to the API:
TODO
apiRequest()
method in OAuthTestCase
currently only allows for posting JSON data and needs to be expanded / adapted.PHP Unit tests are currently failing:
There was 1 failure:
1) DataHub\OAIBundle\Tests\ProviderControllerTest::testListRecords
Failed asserting that actual size 0 matches expected size 1.
/home/travis/build/thedatahub/Datahub/src/DataHub/OAIBundle/Tests/ProviderControllerTest.php:410
FAILURES!
Tests: 5, Assertions: 59, Failures: 1.
Bootstrap 3.3.7 is currently used to drive the (primitive) interface. However, Bootstrap 4.0 is the new stable version. Since the interface is still fairly uncomplicated, now's the chance to upgrade to Bootstrap 4.0.
This change is important to ensure the longevity and the maintenance of the project. While Bootstrap 3.3.7 certainly has it merits, it's better to go with the latest major stable release while there's still a window of opportunity to do so.
If we stay with 3.3.7 and develop a more advanced interface - pending certain requests - it will be harder to move to 4.x and later major versions.
The latter change is due anyhow since the original templates where conceived within a very limited scope.
Updating/deleting records results in an MongoException complaining about an invalid name.
request.CRITICAL: Uncaught PHP Exception MongoException: "MongoDB::__construct(): invalid name " at /vagrant/vendor/doctrine/mongodb/lib/Doctrine/MongoDB/Database.php line 579 {"exception":"[object] (MongoException(code: 2): MongoDB::__construct(): invalid name at /vagrant/vendor/doctrine/mongodb/lib/Doctrine/MongoDB/Database.php:579)"} []
PHP 7.2 support for Travis fails with authentication errors for MongoDB. To be researched what goes wrong and how it can be fixed.
We need to add functional tests for:
This is required to avoid inadvertently breaking existing functionality during development.
The "Users" section under /user now has this left hand "Navigation" sidebar with just one item: "Users" which links to /user/users. The navigation sidebar is confusing, not really needed and should be deprecated / removed.
This sidebar is only needed because of the convoluted /user/users routing. When the routing gets fixed in #41, the navigation sidebar can be deprecated, making the interface less complicated and more straightforward to use.
It should be possible to store records in different data types (EAD, LIDO) in a single Datahub installation.
Version 1.x doesn't support multiple data types. A Datahub instance only supports one data type. If you want to aggregate and disseminate data from multiple sources / formats, then you need to install multiple instances. i.e. an instance for LIDO and an instance for EAD.
Motivation
Discussion / Impact
The API's (OAI/REST) need to be able to differentiate between formats (EAD, LIDO,...). More accurately, the API's need to differentiate between data types of the records; or even differentiate between different types of collections of records independent of their formatting. Whereas a collection references a specific context (i.e. records describing all digitised photos created by an organisation, the archival holdings of James Joyce, sheet music written by a composer, archived medical records from a hospital, etc.)
Content negotiation allows the same record being delivered in different formats & data models. Since Dublin Core is a general purpose format / model, the same metadata record could be delivered as DC and LIDO. But it does not make sense to model the same record as LIDO and EAD since a record is either an archival description or an object record. Not both. Context matters. Formats such as MODS linger between the specificity of MARC (library data) and the general purpose flexibility of DC. It would be possible to describe museal objects in both MODS and LIDO since both models lend themselves to that end. But equally, we could store two discrete collections of records in the hub: MODS formatted library data & LIDO museal data.
The core principle of the Datahub is to avoid any internal data modelling (transformations, mappings, editing,...) of the records itself though, since those add complexity and assumptions that decrease flexibility. Data modelling is always deferred as an external responsibility. This implies that records might have a DC, MODS or LIDO representation, or not. This makes formal content negotiation where every record always has multiple representations, less feasible to implement.
Collections, on the other hand, could potentially be modelled through dynamic URL paths within both the REST and OAI endpoints. The hub would feature one REST API featuring collections, and multiple OAI endpoints (one per collection). Management would happen through the administrative interface.
This would also impact the OAuth implementation. Currently, protecting access to the data is all or nothing. But access is also a context driven aspect. One could differentiate between "digitized photos either accessible to the public or kept private because of copyright".
The instructions state that a default admin user was created, but the default password is not provided.
Add support for PHP Code Standards Sniffer. The idea is to perform linting of the app & src code against the PSR standards to pick up on potential optimalization.
This is an enhancement.
Linting code improves readability and maintainability, and also helps pick up potentially hidden bugs which are tedious to fix.
Currently, all the tests are spread out over the various internal bundles (a tests folder per bundle). It would be better to move the test classes to the more general tests
folder which lives in the root of the Symfony project.
This change is important because:
tests
folder but following the namespacing and directory structure of the bundles.scripts
folderThe config_*.yml files are currently nested into each other.
This setup is brittle, causes confusing and is error prone. Creating a separate security_test.yml and including it in config_test.yml, doesn't correctly override the main firewall in security.yml higher up the include chain, for instance.
This is important to allow clean functional and unit testing. And it's important because isolating concerns creates a less room for errors and bugs.
There's a difference between the bundle (folder) name and the class name. This is very confusing:
OAuthBundle > DataHubOauthBundle
SharedBundle > DataHubSharedBundle
UserBundle > DatahubUserBundle
OAIBundle > DataHubOAIBundle
ResourceBundle > DataHubResourceBundle
ResourceAPIBundle > DataHubResourceAPIBundle
Do we need the 'DataHub' prefix for each bundle? Omitting that would make things more clear and maintainable.
The OAI-PMH protocol allows harvesting of "sets" of records. These are subsets of records in a repository. If the Datahub allows assigning records into sets, partial harvesting would also become possible.
When data is aggregated from multiple data providers (different institutions, etc.), it's not possible to just harvest the data from that particular data provider without harvesting everything else. The onus of selecting the relevant records is completely on the client while harvesting everything means expending extra resources on the side of the Datahub. Subsets would solve this use case.
This is a new feature which needs to be designed / worked out.
Questions:
Issuing a PUT request causes a Fatal Error with an invalid argument to ::getRecords().
request.CRITICAL: Uncaught PHP Exception Symfony\Component\Debug\Exception\ContextErrorException: "Catchable Fatal Error: Argument 1 passed to DataHub\ResourceBundle\Data\Converter\CatmanduLidoXMLDataConverter::getRecords() must be of the type array, none given, called in /vagrant/src/DataHub/ResourceAPIBundle/Controller/DataController.php on line 282 and defined" at /vagrant/src/DataHub/ResourceBundle/Data/Converter/CatmanduLidoXMLDataConverter.php line 35 {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\ContextErrorException(code: 0): Catchable Fatal Error: Argument 1 passed to DataHub\\ResourceBundle\\Data\\Converter\\CatmanduLidoXMLDataConverter::getRecords() must be of the type array, none given, called in /vagrant/src/DataHub/ResourceAPIBundle/Controller/DataController.php on line 282 and defined at /vagrant/src/DataHub/ResourceBundle/Data/Converter/CatmanduLidoXMLDataConverter.php:35)"} []
See job #22.2 .
When creating a new user - and editing this same user - several important fields are missing:
=> When you create a new user - or edit afterwards - it is not possible to set the password.
=> When you create a new user - or edit afterwards - it is not possible to set the status to 'enabled'.
When you run ./scripts/run_tests
the unit tests will all pass, but we know that the API does throw PHP and other types of errors.
Cause:
The "LIDOXML" array key in $testDataList should be lower case in DataControllerTest:
$testDataList = [
'LIDOXML' => [
file_get_contents(__DIR__.'/../Resources/LidoXML/LIDO-Example_FMobj00154983-LaPrimavera.xml'),
file_get_contents(__DIR__.'/../Resources/LidoXML/LIDO-Example_FMobj20344012-Fontana_del_Moro.xml'),
],
];
MARCXML is an XML schema based on the common MARC21 standards.[4] MARCXML was developed by the Library of Congress and adopted by it and others as a means of facilitating the sharing of, and networked access to, bibliographic information.[4] Being easy to parse by various systems allows it to be used as an aggregation format, as it is in software packages such as MetaLib, though that package merges it into a wider DTD specification.
See: https://en.wikipedia.org/wiki/MARC_standards
See: http://www.loc.gov/standards/marcxml/
Supporting MARC(XML) will open up the use case for Libraries to leverage the Datahub as an aggregator.
Replace current "Hello, World" message with a more suitable message. Short term idea: just display the total number of records stored in the database.
Problem
After installation, this error will pop up:
ContextErrorException in OAuthStorage.php line 80:
Catchable Fatal Error: Argument 5 passed to FOS\OAuthServerBundle\Storage\OAuthStorage::__construct() must implement interface Symfony\Component\Security\Core\User\UserProviderInterface, instance of FOS\UserBundle\Doctrine\UserManager given, called in /vagrant/app/cache/dev/appDevDebugProjectContainer.php on line 2171 and defined
Cause
fos_user.user_manager became deprecated in favor of fos_user.user_provider.
This was fixed in FriendsOfSymfony/FOSOAuthServerBundle#26
Solution
config.yml:
service:
user_provider: fos_user.user_provider.username
The scripts
folder is not part of the Symfony framework. It's a folder containing
These scripts (partly) encapsulate functionality that should reside outside the core package because they violate the single responsibility principle.
While there is definitely a use case for these scripts, including them as is - undocumented, untested - creates confusing for users who have no direct need for these scripts.
The scripts related to unit testing with Docker are assuming a preference for Docker as 'go-to' tool for setting a up a testing environment. On the contrary, from the point of view of the Datahub, it is of no concern which technology lower in the stack is used to run the unit tests. As such, Docker support should not be part of a generic package.
The users controller uses this routing:
/user/users
/user/users/new
/user/users//show
This is confusing and complicated. The routing should be made simpler:
/user
/user/new
/user/
/user//edit
/user//delete
Refactoring the routing paths, would simplify the way users are added. Make it more understandable. Also, there's this second-level "navigation" sidebar in the Users section with just one menu item "users", to get to the second level (user/users) which could be deprecated, making the interface less cluttered.
EAD or Encoded Archival Description is an XML format used to serialize archival records.
The EAD standard's XML schema specifies the elements to be used to describe a manuscript collection as well as the arrangement of those elements (for example, which elements are required, or which are permitted inside which other elements). The EAD tag set has 146 elements and is used both to describe a collection as a whole, and also to encode a detailed multi-level inventory of the collection. Many EAD elements have been, or can be, mapped to content standards (such as DACS and ISAD(G)) and other structural standards (such as MARC or Dublin Core), increasing the flexibility and interoperability of the data.[9]
See: https://en.wikipedia.org/wiki/Encoded_Archival_Description
Supporting for this format will allow archives to use the Datahub as an aggregator.
Package managers and tools such as Composer and packagist.org allow for extremely easy distribution and deployment of PHP packages. This includes applications too.
Before publishing the Datahub as a package on packagist.org, it's important to package the application according to best practices.
The League of Extraordinary Packages is a band of PHP developers who aim to deliver high quality PHP libraries according to best practices. They set the benchmark against which to build a quality software package.
We should aim to re-package this repo according to their Skeleton project:
The app is split in a few core bundles who each manages their own services.yml
with a custom dependencyInjection class. Since we don't plan to release those bundles as separate, stand alone package, it's overkill to spread all the service definitions across bundles.
Instead, we should centralise them in app/config/services.yml
.
This is important because it improves maintainability. Additionally, this would allow us to use Auto Wiring for the application logic.
app/config/services.yml
We should be aware of issues with namespacing and renaming bundles and services. See #51.
The JSON API currently returns an error when requesting GET api/v1/data while there are no records present.
{
"code": 500,
"message": "You have requested a non-existent service \"serializer\". Did you mean one of these: \"fos_rest.serializer\", \"fos_rest.serializer.exception_normalizer.jms\", \"hateoas.serializer.exclusion_manager\", \"jms_serializer\", \"jms_serializer.accessor_strategy\", \"jms_serializer.array_collection_handler\", \"jms_serializer.constraint_violation_handler\", \"jms_serializer.datetime_handler\", \"jms_serializer.deserialization_context_factory\", \"jms_serializer.doctrine_proxy_subscriber\", \"jms_serializer.expression_evaluator\", \"jms_serializer.form_error_handler\", \"jms_serializer.handler_registry\", \"jms_serializer.json_deserialization_visitor\", \"jms_serializer.json_serialization_visitor\", \"jms_serializer.metadata_driver\", \"jms_serializer.naming_strategy\", \"jms_serializer.php_collection_handler\", \"jms_serializer.serialization_context_factory\", \"jms_serializer.stopwatch_subscriber\", \"jms_serializer.templating.helper.serializer\", \"jms_serializer.xml_deserialization_visitor\", \"jms_serializer.xml_serialization_visitor\", \"jms_serializer.yaml_serialization_visitor\", \"fos_rest.serializer.jms_handler_registry\", \"jms_serializer.metadata.file_locator\", \"jms_serializer.metadata_factory\"?"
}
Expected: if no records are present, an error should be thrown with an appropriate HTTP status message.
Symfony 4.0 was released on November 30th. Right now, the Datahub is build on top of Symfony 2.8.x. We should research if switching to 4.0 is a good proposition.
OAI-PMH models the concept of deleted records. These are records that are no longer available in the repository, but are still listed in the endpoint. When a client harvests a record which is denoted as "deleted", it can take an appropriate action based on this status.
Currently, the Datahub does not support this. Using DELETE on a record, will perform a hard removal of the resource without any trace.
This issue involves:
Tangential, this means that:
When editing or updating an entity as an administrator (user, client,...) you don't get a message if the edit wether or not the edit was successful. This leaves the administrator in the dark if things went as expected.
When a form is submitted, it should always show a proper flash message reporting whether an action went as expected.
Usability issue.
When running tests, artefacts such as extra tables, etc. will be created. Since these are temporary, they should be purged when testing has finished.
When running tests, a datahub-* (datahub-test) database is automatically created. This database should be dropped after testing.
Currently, the parameters.yml file for the FlemishArtCollection instance is included in the project. This should be removed from the repository leaving only the parameters.yml.dist file.
The parameters.yml file should be generated during composer install
.
(v1.0.0) The installer still requests the path to catmandu, but as far as I can see the application is not used in the datahub itself.
Currently, the administrator is loaded as a fixture via a separate console command. This is unsafe since a password needs to be reset manually after initial installation; an action which can be easily forgotten.
Generating the administrator user + password during installation runtime circumvents this.
Dublin Core is an ubiquitous serialization format to exchange basic descriptive and identification information about web resources. Supporting this format allows content creators beyond GLAM institutions to leverage The Datahub as an aggregator (i.e. aggregate blogposts, news articles, forum posts, etc., etc.)
See: http://dublincore.org/schemas/xmls/
See: https://en.wikipedia.org/wiki/Dublin_Core
OAI-PMH assumes Dublin Core support out of the box.
The Datahub only accepts XML formatted documents as an input. Not JSON formatted documents.
We should document this explicitly.
This is not a bug, but a design choice since most of the traditional cultural formats (MODS, EAD, LIDO,...) are XML based. JSON itself is originally schema-less. Efforts such as JSON Schema are trying to change this.
Documenting this is important since future extensions (new formats) which are JSON based (outside the cultural domain) won't be supported by the Datahub.
The default Symfony templates in app/resources/views
weren't removed.
These templates need to be removed:
views/base.html.twig
views/default/index.html.twig
Neither template is used in the application. These are placeholders which come with Symfony2 and don't serve any functional purpose.
Removing them makes things cleaner and more maintainable (less cruft).
Right now, navigation is hard wired throughout different templates. The topnav.html.twig
template contains presentation and business logic to render the menu. Depending on the context (logged in/ logged out) different menu items are shown. Then there are the sidebar menu items on "users", "clients" and "tokens".
The result is hard to understand code which is difficult to maintain.
We should move KnpMenuBundle for building and managing menu's.
Follow the implementation guide to start.
This issue currently depends on #50 which deprecates much of the SharedBundle folder and turns it into a dedicated CoreBundle instead.
Currently, most templates live in the src/Datahub/<bundle> folders
. However, these bundles are a core part of the application and the templates are thus available sitewide. Moreover, there's no intention to publish the bundles as separate, independent projects.
Per the documentation:
templates/
The application's views directory can contain application-wide base templates (i.e. your application's layouts and templates of the application bundle) as well as templates that override third party bundle templates (see How to Override Templates from Third-Party Bundles).
vendor/path/to/CoolBundle/Resources/views/
Each third party bundle houses its templates in its Resources/views/ directory (and subdirectories). When you plan to share your bundle, you should put the templates in the bundle instead of the templates/ directory.
We should move / restructure / consolidate the templates of the bundles into the templates/
folder.
Right now, the templates are spread out across different bundles. Moreover, they reference each other creating all kinds of weird nesting issues. This is particularly noticeable in the UsersBundle that references to templates in the SharedBundle while overriding templates in the FOSUser bundle.
Cleaning this up would greatly enhance maintainability and durability.
Currently, authentication at the level of mongodb databases isn't supported. You need to disable authentication on the entire mongodb instance in order to connect and perform actions on the databases.
Should to be added: support for database connection authentication. This should also include supporting authentication for tests since running PHPUnit tests will create a new datahub-test database.
Alternatively, we need to rethink how tests are performed since creating a separate database isn't always possible or even allowed.
The unit tests were written against an older API contract. The POST /v1/data call in DataControllerTest.php does not reflect the newer structure.
The body of the HTTP request is a JSON object with 2 parameters: "format" and "data". The new API call should just receive a serialized record as is (ie. a LIDO record) and have the format specified as a HTTP "Content-Type" request header. The datahub instance itself has a global parameter which defines the type of format which will be accepted.
Unit tests should also test both valid and invalid input.
This is a meta issue. Refer to this one through specific issues aimed at changing / fixing concrete unit tests.
Just like WordPress or Drupal, it should be as easy as dropping the code into a hosting and when you point to the homepage, you get an installer wizard via the browser.
Once installed, you land on the homepage of a fully functional Datahub.
This assumes that:
When you a user navigates to the dashboard, there should be an indication or notification of the active format (LIDO, Dublin Core, MODS, etc.). This allows the user to anticipate the format of the data stored in the hub and made available through the API.
Preferably the notification displays this information:
These should be stored in the PHP classes that define the format types.
The dashboard should also feature a short overal description of the data it contained. i.e. "Archival records stored in the City Archives" or "Library records from the University of Foo Bar" or "Miscellaneous blog posts aggregated from several sources" (Dublin core), etc.
This should be part of the general configuration in parameters.yml
Documentation for the Rest API at /api/docs
is generated through NelmioApiDocBundle. Right now, the REST API is extremely straightforward (1 controller, 5 actions).
We should remove this bundle and replace it with a self maintained static documentation page.
The complexity of wiring up this bundle, generating the HTML documentation and then wiring up the generated documentation in the UI, currently weighs in far more then the benefit it creates: a documented API.
Removal would reduce the number of dependencies which needs to be maintained and makes the package more robust.
The SharedBundle bundle contains a lot of "general" code which is called in the other bundles such as traits, templates, functions. It also contains the DefaultController which has the Dashboard action. And finally, it has the app:setup command and a few fixtures.
The Bundle itself doesn't serve a real purposes. It's just something other bundles depend upon. As such, it violates the Single Responsibility Principle. Deprecating the module would vastly improve maintainability and testability of code.
Breaks down in these tasks:
From Wikipedia:
The Metadata Object Description Schema (MODS) is an XML-based bibliographic description schema developed by the United States Library of Congress' Network Development and Standards Office. MODS was designed as a compromise between the complexity of the MARC format used by libraries and the extreme simplicity of Dublin Core metadata.
See: http://www.loc.gov/standards/mods/
Todo:
The Datahub uses FOS User Bundle which comes with /profile route (and subroutes) which allows the current logged in user to change their own mail address and password. Currently, the interface doesn't link to those paths and neither the forms / views on those paths are themed.
It would be a good, userfriendly enhancement to include them in the Datahub.
In README, some required packages and caveats are missing:
The installation of Bootstrap seems to be unnecessary. If so, it can be deleted from the composer.json file.
Problem:
A bulk ingest of records in the Datahub will fail if it takes longer then 1 hour to complete. It will fail with this error:
Response code: 401
Response headers:
Response body:
{"error":"invalid_grant","error_description":"The access token provided has expired."}
2016/12/13 13:13:16 [80941] - ERROR main::catch {...} bin/dh-factory.pl (80) time=3743487 : Error while adding item S-54: 401 Unauthorized
Cause
The Access token life time is too short. By default, this is set to 3600 seconds by FOSOAuthServerBundle.
Solution
Try and make changes in the config.yml per the documentation
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.