arxiv / arxiv-submission-core Goto Github PK

View Code? Open in Web Editor NEW

13.0 14.0 12.0 24.85 MB

arXiv submission core

Home Page: https://arxiv.github.io/arxiv-submission-core/

License: MIT License

Python 99.42% Shell 0.28% Dockerfile 0.10% HTML 0.12% Mako 0.04% TSQL 0.03%

arxiv-submission

arxiv-submission-core's Introduction

arXiv Submission

This repository houses development related to the arXiv-NG submission system. See https://arxiv.github.io/arxiv-submission-core/ for the latest documentation.

Documentation

Freshen/build

Update the API doc source refs with:

sphinx-apidoc -o docs/source/arxiv.submission -e -f -M --implicit-namespaces core/arxiv *test*/*

Build HTML docs with:

cd docs
make html SPHINXBUILD=$(pipenv --venv)/bin/sphinx-build

Contributions

https://github.com/cul-it/arxiv-submission-core/blob/master/CONTRIBUTING.md

What's in the repo

The events core package provides integrations with the submission database and notification streams, and exposes a Python API for event-based operations on submission (meta)data. Any web services that modify submission data must do so via this package.
The submission agent is a Kinesis consumer that orchestrates backend processes based on rules triggered by submission events.

In progress/stale

These components are considerably behind, or only partially complete. Future development milestones will focus on these services, possibly in separate repositories.

The API service provides the client-facing interface for submission-related requests. Status: In progress
The Webhooks service provides an API for creating and managing submission-related webhooks. Status: Schema only
A toy Gateway service provides a minimal NGINX server configured to utilize the authentication service (below). It provides (proxy) access to client-facing services, including the API service. This is close (but not identical) to what is run in production.

Related components/dependencies

The authentication service handles sub-requests from the gateway to authorize client requests, and mints encrypted JWTs for use by other services.
The client registry provides OAuth2 workflows. Currently supports the client_credentials and authorization_code grant types.
The file management service is responsible for handling client/user uploads, and perform sanitization and other QA checks.
The submission UI provides a form-driven UI for submission. The UI is built on top of the submission core package (this repo).
The compiler service is responsible for compiling LaTeX sources to PDF, DVI, and other formats.

Python dependencies

This project uses pipenv to manage Python dependencies. To install dependencies do:

$ pipenv install

To install dev/testing dependencies, use:

$ pipenv install --dev

Local deployment with Docker Compose (for testing only)

A Compose file (docker-compose.yml) is included in the root of this repository, and can be used to run the services in this project for local testing and development purposes.

See the Docker documentation for information about using Docker Compose.

Starting the service cluster

The Compose file included here deploys all services on a custom network, and exposes the gateway service at port 8000 on your local machine.

To start up (from the root of the repo, containing docker-compose.yml):

$ docker-compose build
$ docker-compose up

This will generate a lot of output, and bind your terminal. Some things to watch for:

MariaDB starting up successfully:

submission-maria            | 2018-04-11 13:49:16 0 [Note] Reading of all Master_info entries succeded
submission-maria            | 2018-04-11 13:49:16 0 [Note] Added new Master_info '' to hash table
submission-maria            | 2018-04-11 13:49:16 0 [Note] mysqld: ready for connections.
submission-maria            | Version: '10.3.5-MariaDB-10.3.5+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

The metadata API service waiting for the database to be ready:

submission-metadata         | application 11/Apr/2018:09:49:13 +0000 - __main__ - None - [arxiv:null] - INFO: "...waiting 4 seconds..."

After the DB is available, you should see something like this as the metadata service initializes the database and adds some data.

submission-metadata         | application 11/Apr/2018:09:49:17 +0000 - __main__ - None - [arxiv:null] - INFO: "Checking for database"
submission-metadata         | application 11/Apr/2018:09:49:17 +0000 - __main__ - None - [arxiv:null] - INFO: "Database not yet initialized; creating tables"
submission-metadata         | application 11/Apr/2018:09:49:18 +0000 - __main__ - None - [arxiv:null] - INFO: "Populate with base data..."
submission-metadata         | application 11/Apr/2018:09:49:18 +0000 - __main__ - None - [arxiv:null] - INFO: "Added 10 licenses"
submission-metadata         | application 11/Apr/2018:09:49:18 +0000 - __main__ - None - [arxiv:null] - INFO: "Added 3 policy classes"
submission-metadata         | application 11/Apr/2018:09:49:18 +0000 - __main__ - None - [arxiv:null] - INFO: "Added 150 categories"
submission-metadata         | application 11/Apr/2018:09:49:18 +0000 - __main__ - None - [arxiv:null] - INFO: "Added 500 users"

Note that the users generated here are fake. Licenses, categories, etc are all "realistic".

After generating data, the web service starts:

submission-metadata         | spawned uWSGI master process (pid: 13)
submission-metadata         | spawned uWSGI worker 1 (pid: 21, cores: 100)
submission-metadata         | spawned uWSGI worker 2 (pid: 22, cores: 100)
submission-metadata         | spawned uWSGI worker 3 (pid: 23, cores: 100)
submission-metadata         | spawned uWSGI worker 4 (pid: 24, cores: 100)
submission-metadata         | spawned uWSGI worker 5 (pid: 25, cores: 100)
submission-metadata         | spawned uWSGI worker 6 (pid: 26, cores: 100)
submission-metadata         | spawned uWSGI worker 7 (pid: 27, cores: 100)
submission-metadata         | spawned uWSGI worker 8 (pid: 28, cores: 100)

Creating a client

The arxiv/registry image provides a helper script to create a new API client.

Be sure to change [NETWORK_NAME] to the actual name of the Docker network that the service cluster is using. It should be [something]arxiv-submission-local. E.g, it might be arxivsubmissioncore_arxiv-submission-local. To be sure, run docker network ls | grep arxiv-submission-local.

docker run -it -e REGISTRY_DATABASE_URI=mysql+mysqldb://foouser:foopass@registry-maria:3306/registry?charset=utf8 --network=[NETWORK NAME] arxiv/registry:0.1 -- python create_client.py

Follow the prompts to create your client. For best results, use the default scopes (just press enter):

Brief client name: client
Info URL for the client: http://client
What is it: a client
Space-delimited authorized scopes [public:read submission:create submission:update submission:read upload:create upload:update upload:read upload:read_logs]:
Redirect URI: http://localhost:1234/callback
Created client client with ID 2 and secret o86ScxuqcOffbWKxvyGke6e4wFTIukkjiJEc4ofBj7cDmNLz

Note the client ID and secret at the end -- you'll need those.

Client credentials authorization

In production, the submission API will require an authorization code (three-legged authorization) grant. For local testing, however, it may be more convenient to use client credentials.

Use your client ID and secret (above) to get an access token.

Include the header: Content-Type: application/x-www-form-urlencoded

Include the following parameters:

grant_type -- This should be client_credentials
client_id -- This should be your client ID
client_secret -- This should be your client secret

$ curl -XPOST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=2&client_secret=o86ScxuqcOffbWKxvyGke6e4wFTIukkjiJEc4ofBj7cDmNLz" http://127.0.0.1:8000/api/token
{"access_token": "4tkstLJauH65EwpewmpJ0IugdqFLMctHiIjl5IvWxK", "expires_in": 864000, "token_type": "Bearer"}

You should get back a JSON document with the following properties:

access_token -- That's your access token.
expires_in -- Number of seconds until the token expires.
token_type -- This should be "Bearer".

Create a submission

At this point, you should be able to interact with the submission API.

Be sure to include the headers:

Content-Type: application/json -- That's because the submission metadata API speaks JSON.
Authorization: Bearer [your access token] -- If you don't include a valid token, you'll get a 401 Unauthorized response. Or a 403, if you've really been unruly.

You can use the pre-baked submission in metadata/examples/complete_submission.json to get up and running.

$ curl -i -X POST -H "Content-Type: application/json" \
    -H "Authorization: Bearer 4tkstLJauH65EwpewmpJ0IugdqFLMctHiIjl5IvWxK" \
    --data-binary "@metadata/examples/complete_submission.json" http://localhost:8000/submission/

But note that you can create a submission with far less than that!

$ curl -i -X POST -H "Content-Type: application/json" \
    -H "Authorization: Bearer 4tkstLJauH65EwpewmpJ0IugdqFLMctHiIjl5IvWxK" \
    --data "{}" http://localhost:8000/submission/

HTTP/1.1 201 CREATED
Server: nginx/1.13.12
Date: Mon, 24 Sep 2018 19:47:33 GMT
Content-Type: application/json
Content-Length: 1027
Location: http://localhost:8000/submission/7/
Connection: keep-alive

{"active":true,"arxiv_id":null,"client":{"client_id":"2"},"compilations":[],"created":"2018-09-24T19:47:33.251494","creator":{"affiliation":"","agent_type":"User","email":"","endorsements":["*.*"],"forename":"","identifier":null,"name":"  ","native_id":null,"suffix":"","surname":"","user_id":null},"delegations":{},"finalized":false,"license":null,"metadata":{"abstract":null,"acm_class":null,"authors":[],"authors_display":"","comments":"","doi":null,"journal_ref":null,"msc_class":null,"report_num":null,"title":null},"owner":{"affiliation":"","agent_type":"User","email":"","endorsements":["*.*"],"forename":"","identifier":null,"name":"  ","native_id":null,"suffix":"","surname":"","user_id":null},"primary_classification":null,"proxy":null,"announced":false,"secondary_classification":[],"source_content":null,"status":"working","submission_id":7,"submitter_accepts_policy":null,"submitter_confirmed_preview":false,"submitter_contact_verified":false,"submitter_is_author":null,"updated":"2018-09-24T19:47:33.251494"}

You can update a submission by POSTing fields that you want to update.

$ curl -i -X POST -H "Content-Type: application/json" \
    -H "Authorization: Bearer 4tkstLJauH65EwpewmpJ0IugdqFLMctHiIjl5IvWxK"  \
    --data '{"metadata":{"title":"The theory of life and everything","doi":"10.00123/foo45678"}}' \
    http://localhost:8000/submission/7/

HTTP/1.1 200 OK
Server: nginx/1.13.12
Date: Mon, 24 Sep 2018 20:23:57 GMT
Content-Type: application/json
Content-Length: 1060
Location: http://localhost:8000/submission/7/
Connection: keep-alive

{"active":true,"arxiv_id":null,"client":null,"compilations":[],"created":"2018-09-24T20:22:33.498688","creator":{"affiliation":"","agent_type":"User","email":"","endorsements":["*.*"],"forename":"","identifier":null,"name":"  ","native_id":null,"suffix":"","surname":"","user_id":null},"delegations":{},"finalized":false,"license":null,"metadata":{"abstract":null,"acm_class":null,"authors":[],"authors_display":"","comments":"","doi":"10.00123/foo45678","journal_ref":null,"msc_class":null,"report_num":null,"title":"The theory of life and everything"},"owner":{"affiliation":"","agent_type":"User","email":"","endorsements":["*.*"],"forename":"","identifier":null,"name":"  ","native_id":null,"suffix":"","surname":"","user_id":null},"primary_classification":null,"proxy":null,"announced":false,"secondary_classification":[],"source_content":null,"status":"working","submission_id":7,"submitter_accepts_policy":null,"submitter_confirmed_preview":false,"submitter_contact_verified":false,"submitter_is_author":null,"updated":"2018-09-24T20:23:57.754003"}

You can finalize the submission by updating the finalize field to true. But if fields are missing, you'll get an error.

$ curl -i -X POST -H "Content-Type: application/json" \
    -H "Authorization: Bearer 4tkstLJauH65EwpewmpJ0IugdqFLMctHiIjl5IvWxK" \
    --data '{"finalized": true}' \
    http://localhost:8000/submission/7/

HTTP/1.1 400 BAD REQUEST
Server: nginx/1.13.12
Date: Mon, 24 Sep 2018 20:25:32 GMT
Content-Type: application/json
Content-Length: 62
Connection: keep-alive

{"reason":"Invalid Stack:\n\tMissing primary_classification"}

arxiv-submission-core's People

Contributors

Stargazers

Watchers

Forkers

neolithera based-god-fucked-my-bitch-fuckzig rjt1990 tubbz-alt woutersj

arxiv-submission-core's Issues

Add documentation for fm:// notation in source path

Presumably this has to do with the file management service? A brief comment would be helpful.

Originally posted by @mhl10 in #64

Submissions that contain German umlauts do not compile

Describe the bug
Hello, my name is "Tim Daubenschütz". And I tried to submit a paper with this name, but every time I tried, I got this error:

TeX capacity exceeded, sorry [input stack size=5000]

from your compiler.

It's being talked about here too: https://tex.stackexchange.com/a/64924
To Reproduce
Steps to reproduce the behavior:

create paper.tex and add an "ü" somewhere
Try to submit to arxiv

Expected behavior
I expect that any unicode character can be used to submit a paper. Maybe there's already babys named "🌵 ", lol.

Metadata clean up for submission

In the classic system we have various metadata fixes happening after the user submits:

PG auto changes
Student metadata fixes
System metadata flags
Admin script bin/report/check_mailing.pl

The goal is to build those into the submission, preferably upfront so the user is aware and approves the changes.

Includes WL-01 and WL-46.

Note some currently automated flags and changes may not be necessary with unicode support.

Admins should be able to grant exceptions to oversize limits ahead of time, independent of hold mechanism

Currently the oversize workflow is clunky. An exception for oversize is a release form Hold.
Could this be a dbflag instead?
That way we could grant exceptions prior to submission. And if the user needs to update after the initial submit it doesn't require another admin release.
Maybe this is for later on if it is dependent on admin ui changes to make it work.

Broken documentation link in the readme

The current README.md file starts off by linking to https://cul-it.github.io/arxiv-submission-core/ as the site of the latest documentation; this link appears to be broken.

Presumably this should link to https://arxiv.github.io/arxiv-submission-core/ instead?

On brave, when clicking "View" or "View Article" even though submission compilation was successful, Brave displays "corrupt PDF"

Describe the bug
On brave, when clicking "View" or "View Article" even though submission compilation was successful, Brave displays "corrupt PDF"

To Reproduce
Steps to reproduce the behavior:

Download brave browser
Submit a working paper to arxiv
Click on "View" (Process) or "View Article" (Preview)
PDF will not render. Try same with fresh Installation of Firefox. It works.

Expected behavior
All browsers should work equally.

Desktop (please complete the following information):
Brave: Version 0.63.55 Chromium: 74.0.3729.131 (Official Build) (64-bit)

Typo in events.services.classic.models

https://github.com/cul-it/arxiv-submission-core/blob/546ca142aa08c8459cc78cfe98241d46e1f92e5b/core/events/services/classic/models.py#L65

Should be wdr

Additional classification rules

Per @mhl10 :

There are some additional rules that wouldn't be obvious from looking at the classic code (understatement).

submitters may not choose more than 4 secondary categories in the classic submission workflow. This rule isn't enforced outside that workflow; i.e. admins (and moderators) may elect to add more than four.
crossing to physics.gen-ph is generally not allowed.
There are others I'm sure I'm missing. We should probably have a separate ticket for reviewing some of these rules.

Wordsmiting classification selection step

Steinn mentioned having users propose classification rather than select classification. The point is to ensure users understand that arXiv classification is set by moderators, and that it just happens in most cases that submitters and moderators align.

Once we have a mock up on the classification page @eawoods and @jimentwood should have a chat with Steinn and admins to wordsmith that.

Note to self: If we do make this perspective shift, then admins should roll it into help/moderation and in communication with mods (blog post by Steinn?).

Potential dependency conflicts between arxiv-submission-core and bleach

Hi, as shown in the following full dependency graph of arxiv-submission-core, arxiv-submission-core requires bleach (the latest version)， while the installed version of arxiv-base(0.15.9) requires bleach==3.1.0.

According to Pip's “first found wins” installation strategy, bleach 3.1.0 is the actually installed version.

Although the first found package version bleach 3.1.0 just satisfies the later dependency constraint （bleach==3.1.0), it will lead to a build failure once developers release a newer version of bleach.

Dependency tree--------

arxiv-submission-core<version range:>
| +-arxiv-auth<version range:>
| | +-arxiv-base<version range:>
| | | +-backports-datetime-fromisoformat<version range:==1.0.0>
| | | +-bleach<version range:==3.1.0>
| | | +-boto3<version range:>
| | | +-flask<version range:>
| | | +-jsonschema<version range:>
| | | +-pytz<version range:>
| | | +-typing-extensions<version range:>
| | | +-uwsgi<version range:>
| | +-mysqlclient<version range:>
| | +-pycountry<version range:>
| | +-pyjwt<version range:>
| | +-python-dateutil<version range:>
| | +-redis<version range:==2.10.6>
| | +-redis-py-cluster<version range:==1.3.6>
| | | +-redis<version range:==2.10.6>
| | +-sqlalchemy<version range:>
| +-arxiv-base<version range:>
| | +-backports-datetime-fromisoformat<version range:==1.0.0>
| | +-bleach<version range:==3.1.0>
| | +-boto3<version range:>
| | +-flask<version range:>
| | +-jsonschema<version range:>
| | +-pytz<version range:>
| | +-typing-extensions<version range:>
| | +-uwsgi<version range:>
| +-bleach<version range:>
| +-celery<version range:==4.1.0>
| +-dataclasses<version range:>
| +-flask<version range:>
| +-kombu<version range:==4.1.0>
| +-mysqlclient<version range:>
| +-python-dateutil<version range:>
| +-redis<version range:==2.10.6>
| +-sqlalchemy<version range:>
| +-unidecode<version range:>

Thanks for your attention.
Best,
Neolith

Submitting LaTeX-produced publisher PDF files of open-access papers

Use case:

I am an author of a paper published under a Creative Commons license in an Open Access journal, and I would like to deposit the Version of Record of the paper into arXiv, as allowed by the license. All I have available for this is the PDF file itself, and I do not have access to the source used by the publisher to produce the VoR.

For some publishers (e.g. APS) this seems to work fine, but for some others (including IOP) the publisher-produced PDF triggers the arXiv warnings against submitting PDF files produced from LaTeX, and requests the LaTeX source for the document.

Now, I fully understand and support the decision to require TeX source wherever it is available, but in this specific case (which, I would argue, should become more common), the submitter does not have the source available. Of course, one can argue that posting to arXiv is not necessary as such, strictly speaking, since the paper is already freely available elsewhere, but it is definitely odd that, having removed the hurdle of copyright restrictions, one cannot submit to arXiv because of the LaTeX detector.

As such, it would be good if the processing system can detect these cases, or allow the user to declare them (possibly going through manual review?), and allow the authors to submit the PDF on its own.

(Apologies if I have posted this in the wrong place -- if that's the case, please direct me to the correct venue!)

Add additional documentation for StreamPublisher

It would be good to have a little documentation on this class.