Should respond to GET and HEAD requests, exercise the search index connection, and respond as quickly as possible.

UI nitpicking:

• The "1" button is slightly smaller vertically than all other buttons
• Balance vertical margins between sort/page controls, paginator and results
• Put "Advanced/Authors" and "Simple/Authors" into a paragraph with similar bottom margin so that it harmonizes with the beautiful level thing that Erick set up :)
• Also, "Simple | Authors" and "Simple | Advanced" bounce (due to underlines turning on/off) on hover, but "Advanced | Authors" on the simple search page does not. We want it to be like the latter.

If there is a problem contacting the search index, the variable index_error will be injected into the form rendering context. We should show an informative message to the user.

We're currently indexing one document at a time; this is impractical/inefficient for indexing the entire corpus.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

History:

At one time arXiv mirrors either ran a home-brew search engine or full blown Lucene server (LANL). When the home-brew search started to fall over due to the immense size of the collection we switched to using the main site to service search requests for mirrors.

Current Implementation:

When a user submits a search on a mirror the search is amended with a 'client_host' argument containing the client host and then passed (redirected) to the find request on the main site.

The main site executes the search and then formats the results page such that all links on the search results page point the client host. The user has no idea the search was handle remotely and the links keep the user on the mirror site.

Implementation for new search:

This might be possible with two modifications: 1) replace the current search page on the mirror such that it reflects the new search page / advanced search page 2) the main search api updates all links to direct to the optional host specified in the request. We currently check that client_host is mirror before doing this modification. 

For conference proceedings users can create html pages with links to individual articles. If the html submission is before the individual articles then we have a special markup for users to enter the report numbers. Then we have an html parser that converts the report number markup to search based links.

In general I would like to green field how to handle proceedings for NG, but that doesn't need to happen now.

For now some options could be:

• modify the html parser to work with the new search
• modify the new search to support the existing links
• modify the parser to just search the db for report number and then establish a direct link to the arXivID (if that fails then generate a link)
• other options???

Current documentation:
https://arxiv.org/help/submit_index

Example html page
https://arxiv.org/html/1108.3558v1

https://beta.arxiv.org/search/?query=what+is+your+favorite+color&searchtype=all

L'evy looks odd compared to Lévy, is this behavior expected?
re: https://arxiv.org/abs/1711.04210v1

Submitters often include URIs in their comments. It would be great to parse them and present them as external links.

I stumbled upon this issue from our friends at Inspire, which discusses a jinja2 helper for external links:
inspirehep/inspire-next#585

It might be nice to do something similar (i.e. consistent about how we link to external resources).

https://github.com/cul-it/arxiv-search/blob/0fcafb01655503b119928e0b8b4b9ac121b22b55/requirements.txt#L8

Single years tested (2016, 2015, 2014, 2013, 2012) produce a "No Results" page.
Date range 2001-01-01 to 2018-01-01 produces a "No results" page.

Ran into this by mistake.

If you do a strict search "reimann hypothesis" that contains a typo the search is smart enough to return related results. But there is no warning that there were no exact matches.

I don't know that this actually needs to be changed so this is more of a question...

Can we imagine use cases of searching for a specific term or phrase and wanting to see only exact matches?

Should there be a warning that the exact search was not found and so we are displaying related results?

https://beta.arxiv.org/search/advanced?terms-0-operator=AND&terms-0-term=%22reimann+hypothesis%22&terms-0-field=title&terms-1-operator=AND&terms-1-term=proof&terms-1-field=title&classification-physics_archives=all&date-year=&date-from_date=&date-to_date=&size=25

I recognize that this is a brand new and quickly implemented function! Props on getting it working in part. Some observations on names and the breakup into sur/fore in particular, to start a list of things to work out in the author search functionality.

Examples:

Chaimae El Aisati is broken into surname= Chaimae El forename=Aisati. Searching for any fragment of these doesn't work (C / Aisati, Chaimae El / A) but searching for forename Chaimae El and surname Aisati does. Other examples: P. Vasquez Regueiro, M. Vieites Diaz.

Double initials: R/Appleby does not return results, but R. B./Appleby does.

Organizations: some break on the last word (example: for the IceCube / Collaboration), some don't have a forename at all (example: / LHCb collaboration)

International: for Asian names that have forename/surname flipped, how do we know which is which? Can/should we search both sur- and forename fields for a string given as a name or name stem? We can then filter by relevance (and later facet by whole name, possibly). Also relevant for other cultures, or for the treatment of organization names in the author field.

Using a partial surname doesn't work, nor does using a wildcard.

I recognize that implementing secondaries may not yet be in the plan (we don't display them in the results yet either), but generally I think a subject search would turn up both primaries and secondaries, and we could facet on the position (or use as a relevance indicator).

Example: https://arxiv.org/abs/physics/0001066 "The Genetic Code as a Periodic Table: Algebraic Aspects" is classified as physics.bio-ph and secondarily as q-bio, but only findable by searching physics.

The module is one enormous file. It could use some cleanup, and possibly decomposition into submodules.

Also, typing dates into boxes does not select that option automatically. A search performed after typing in data without selecting the range returns all dates (all results) without informing the user.

On the results page, there is a method to access the two forms that weren't used to make the query, and there's a link to reformulate the query - but no method to use the same form for a completely different search.

Author search for Smith; result #3 in our test cases has over 200 names listed.

Without the super long author lists, our new stuff looks GOOD on mobile!!

Wider margins required on the results page to use best design practices for scanning/reading of text.

Possibly a checkbox to exclude withdrawn articles.
Or alternately visually demote or indicate withdrawn articles in the search results.
Or possibly both a filter and a visual identity for withdrawn articles.

From beta feedback, many test cases.

The search field gets very short for mobile widths.

To reproduce:

1. Authors form: enter "cat*" in surname field (get list of 4 results)
2. Sort by: select Submitted date - descending, click Go
3. Get sent to Advanced Search page (instead of reordering results)

Possibly: pages with < 25 results don't reorder? Testing this hypothesis now.

Looking at the ES documentation on nested objects. I guess I'm still not clear about why we would use separate fields for secondary_archives, secondary_groups, and secondary_categories (here rather than a single nested property, e.g. secondary_classification.

What we have now:

        "secondary_categories": {
          "properties": {
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            }
          }
        },
        "secondary_archives": {
          "properties": {
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            }
          }
        },
        "secondary_groups": {
          "properties": {
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            }
          }
        },

A proposed alternative:

        "secondary_classification": {
            "type": "nested",
            "properties": {
                "group": {
                    "type": "nested",
                    "properties": {
                        "id": {"type": "keyword"},
                        "name": {"type": "keyword"}
                    }
                },
                "archive": {
                    "type": "nested",
                    "properties": {
                        "id": {"type": "keyword"},
                        "name": {"type": "keyword"}
                    }
                },
                "category": {
                    "type": "nested",
                    "properties": {
                        "id": {"type": "keyword"},
                        "name": {"type": "keyword"}
                    }
                }
            }
        }

It seems like in the former case, we would need to apply some knowledge about the arXiv taxonomy (external to the search document) in order to generate a sensical faceted search (e.g. which category is allowable with which archive). In the latter case, however, it seems like we wouldn't need to know anything about the taxonomy: we could generate aggregations based entirely on what is present in the search index.

@mhl10 @JaimieMurdock @eawoods thoughts?

When a validation error has occurred (example: forename without surname), error conditions persist after user attempts to correct error.

Also, my browser (OS X El Capitan, Chrome 63.0) puts up an odd little icon to the right of forename once an error has occurred. No tooltips to say what this is, might be LastPass related? Or my individual browser addon settings? See screenshots. Error condition also propagates to additional field.

If we use a monospace font (Lucida Grande), button widths for single characters like + and - will be more consistent. I already have a class that specifies LG for tiny type, due to slightly better readability -- make this more globally applicable (and apply to buttons).

Anything typed into the mini box on the upper right goes to the last query page from which one started a search, without performing search or displaying results. It does preserve and send along the mini box query, for example:
http://127.0.0.1:5000/search/authors?query=first&searchtype=Title

Particularly look at Jref and author, possibly some others

Broad searches will already jump way over max_results_window (defaults to 10,000).

We can support pagination beyond that point, but the two mechanisms to support that in ES are focused on actually paging through progressively (not jumping to an arbitrary page). Thus we may need to make some adjustments to UI components for pagination.

For reference:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html

On the search results page, build error happens when to change the sort order or number of pages when the controls at the top are changed.

Page size (number of results) does work when changed on the Advanced Search page.

Error encountered:

builtins.OSError
OSError: Problem communicating with ES: TransportError(400, 'search_phase_execution_exception', 'No mapping found for [submitted_date] in order to sort on')

And also change "submission date (ascending) to "oldest -> most recent"
Or some such language

On the advanced search, if a user selects one of the Physics sub archives can we automatically check the Physics box?

When one selects the button to create a new field in the advanced search interface, it's annoying that the tabindex has flowed past the field. Thus, in order to fill out the newly created fields, one must tab all the way back through all the controls on the page to fill out the search term that was newly created (and not filling it out will produce an error on search).

e.g. https://arxiv.org/docmeta/cs/9901013

Bug is in arXiv::Identifier.

We're not making this field available in any way at the moment, so not critical for release.

When you type a single word into the simple form, if there are no results the title changes but the position of every element on the page stays the same -- making it look like nothing happened. This is a design issue, functionality is working as expected.

I'm wondering if on the search results view if when you change the '# results per page' or 'sort results by' if it could automatically refresh the view rather than having to hit "Go".

E.g. http://127.0.0.1:5000/authors?authors-0-forename=&authors-0-surname=romeu+&size=25

Already confirmed: single-term search for "data" and "broad" successful on both simple search and advanced search pages.

On the advanced search form, I entered "data" and selected "Abstract", clicked the "+", and added a field. Entered "broad" and selected "Title". Resulting search brought up the following errors:

builtins.AttributeError
AttributeError: 'str' object has no attribute 'field'

Traceback (most recent call last)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1997, in call
return self.wsgi_app(environ, start_response)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/eaw82/anaconda3/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/routes/ui.py", line 34, in advanced_search
response, code, headers = advanced.search(request.args)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/controllers/advanced/init.py", line 59, in search
response_data.update(index.search(q))
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 398, in search
return current_session().search(query)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 325, in search
search = self._prepare(query)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 173, in _prepare
& self._classifications_to_q(query)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 141, in _fielded_terms_to_q
return cls._grouped_terms_to_q(terms)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 98, in _grouped_terms_to_q
term_b = SearchSession._grouped_terms_to_q(term_b)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 94, in _grouped_terms_to_q
term_a = SearchSession._field_term_to_q(term_a)
File "/Users/eaw82/Documents/arXiv/arxiv-search/search/services/index.py", line 75, in _field_term_to_q
if term.field in ['title', 'abstract']:
AttributeError: 'str' object has no attribute 'field'

Click on last page link for everything in the last 12 months, fails to load

https://beta.arxiv.org/search/advanced?advanced=1&terms-0-term=&terms-0-operator=AND&terms-0-field=title&classification-physics_archives=all&date-past_12=y&date-year=2017&date-from_date=2018-01-01&date-to_date=2017-02-03&size=100&order=&start=162500

I frequently search by other metadata fields than Title, Author, Abstract.
Can we include all the searchable fields in the advanced search?

For the time being, the query is displayed just above the sort and number of results controls; find a better way to display this.

It seems the old search has more current version information to display. Note the search-beta result for arXiv:1710.10131 in the attached screenshot shows v1 info, while the screenshot of the old search result
shows v2 info.

Or maybe there is something else going on here.

https://github.com/cul-it/arxiv-search/blob/0fcafb01655503b119928e0b8b4b9ac121b22b55/search/services/metadata.py#L14

If number of results is set to the default 25, the "Go" button to adjust number of results or sort order does not appear.