thestanforddaily / archives-web Goto Github PK

Go to https://archives.stanforddaily.com/search?page=1&pagelen=20&q=Bakkegard%20said%20Student%20Justice%20Project%20did%20not%20influence%20decision&year_end=1904&year_start=1892

I get a blank screen and this message:

Instead, we want it to not have a blank screen, but just give an error message on the UI.

We should eventually also add the (physical) paper published after 2014 to the archives site. For now, we can link newer dates to our main site such as https://www.stanforddaily.com/2019/06/16/.

We should also use the Internet Archive Wayback Machine periodically to store our articles and link to that from our archives site.

e.g. clicking on author name brings us to all articles published by that author

For example, if I select "ASSU President", "ASSU Parlimentarian" [sic], or "STAFF FOOTBALL WRITERS", I get no results.

We should probably be getting at least one result here.

We should add highlighting in search results for fields such as title and author.

Need to combine
https://stanforddailyarchive.s3.amazonaws.com/data.2013-nov/data/stanford/1991/09/19_01/Stanford_Daily_19910919_0001-METS.xml

and

https://tiles.archives.stanforddaily.com/data.2013-nov/data/stanford/1991/09/19_02/Stanford_Daily_19910919_0001-METS.xml

currently, we only show the latter (19_02) but miss all the (19_01) pages -- see https://archives.stanforddaily.com/1991/09/19?page=1&section=MODSMD_ARTICLE2#issue

This is important because we're actually missing articles on the new site (which were there previously on the old site).

"About this collection" "Search the Collection" and "Subscribe" are currently in the same row. Fix this by orienting each header and its respective components vertically one after the other.

Bonus: Make this feature responsive by orienting vertically if the page size is small and switch to horizontal when the page size is big.

• Be able to search everything with an empty query
• Add slider which only lets you select allowed years

• Fix data repeating
• Redirect old site to new site - use now.sh for this
• Search result heights with an expand button - #36
• Be able to search everything, add slider - #37
• Get dns control of stanforddailyarchive.com

Get this done by Nov 17

The dependency react-bootstrap was updated from 1.0.0-beta.12 to 1.0.0-beta.13.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

react-bootstrap is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

create a form on frontend for users to create and submit corrections to articles. Then, someone at TSD needs to approve the changes, and run the pipeline (to be implemented, see: #55) to propagate the changes.

For example, try going to https://archives.stanforddaily.com/1985/08/09?page=11&section=DIVL319#article (clear cache) -- the request takes 15 seconds (1.6 MB)

checkout a subdirectory of repo

Need to add new IAM policy - archives-cloudsearch-deploy

[secrets](https://github.com/TheStanfordDaily/stanforddaily/blob/master/.github/workflows/deploy.yml
.github/workflows/deploy.yml)

[matrix jobs](https://github.com/TheStanfordDaily/stanforddaily/blob/master/.github/workflows/deploy.yml
.github/workflows/deploy.yml)

would be cool to see how many times a page has been viewed. Not sure how to do this though.

searchworks.stanford.edu has a good UI for filtering searches, which would be good to base our UI on

• article_text
• article_title
• article_subtitle
• author_name

or a union of these

create a guide/key/help bar that user can expand which shows some hints for searching e.g. you can add quotations to do an exact text match, just like when google searching.

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

integrate cloudsearch suggester into the frontend. suggester suggests against article_text (high fuzzy search) or article_title (high fuzzy search) or author (exact match), depending on what index field the user is searching by.

in search results

Fetch first page’s first article title and display it?

Use seadragon

The box ids for each article are stored in the METS file
The ALTO file stores the actual text for each box, and coordinates for each box.

1. Get the article id from the server.
2. Read the article contents from the ALTO XML files.
3. Do a search for highlighted word (if needed) and highlight it.

https://openseadragon.github.io/examples/ui-overlays/
https://openseadragon.github.io/examples/tilesource-sequence/

For search results, it should only cover the search results themselves.

Otherwise it makes it not very smooth for the user to quickly paginate or change results.

Current behavior:

https://facebook.github.io/create-react-app/docs/adding-a-sass-stylesheet

Getting errors when running the script

The last successful file parsed was text/1894/4/26/DIVL151.txt Adv. 17 Page 4

(node:15545) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'nodeValue' of undefined
    at Paper._callee$ (/home/ec2-user/environment/node_modules/@thestanforddaily/archives-web/lib/classes/Paper.js:85:63)
    at tryCatch (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:271:22)
    at Generator.prototype.(anonymous function) [as next] (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/home/ec2-user/environment/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/home/ec2-user/environment/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:15545) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15545) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Right now, a single search result shows around 2 paragraphs and basically fills up the entire screen. We'd like each search result to be smaller and show only a snippet of text, so that the user can view many more search results on the screen at the same time.

right now, we're limited to 10,000 search results for any search. to unlock maximum potential, use cloudsearch cursor

https://docs.aws.amazon.com/cloudsearch/latest/developerguide/paginating-results.html

Make this similar to the "share to facebook" etc. buttons that are on the main website.

☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io

The dependency query-string was updated from 6.11.1 to 6.12.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

query-string is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Go to this page, and you see an article titled "Work on the Track" (https://archives.stanforddaily.com/1896/02/05?page=1&section=MODSMD_ARTICLE2#article).

Now, search "work on the track" and this article doesn't even show up... (https://archives.stanforddaily.com/search?page=1&pagelen=20&q=work%20on%20the%20track&year_end=1897&year_start=1892) something wrong with the search process?

User can view possible author titles and select one or multiple (union of selected if multiple) to filter search results by

For example, https://archives.stanforddaily.com/2002/05/31 could have a link to https://web.archive.org/web/20020602164045/http://www.stanforddaily.com/.

And https://archives.stanforddaily.com/2002/05/31#page=1&section%5B%5D=MODSMD_ARTICLE4 could have a link to https://web.archive.org/web/20020629021038/http://www.stanforddaily.com/tempo?page=content&id=8524&repository=0001_article.

These online articles could also be the source for archives-text.

For earlier papers and articles, we probably need to find the links manually.

For later papers and articles, maybe there will be some patterns in the URL?

The dependency interactjs was updated from 1.5.1 to 1.5.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

interactjs is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

• Parse METS/ALTO files to see if images can be delineated
• Create a dynamic tiler that can create images at arbitrary sizes / locations (like https://tiles.archives.stanforddaily.com/data.2013-sep/data/stanford/1951/02/06_01/Stanford_Daily-IMG/Stanford_Daily_19510206_0001_0001.jp2/0,8192,6816,224/852,/0/default.jpg)

Right now, search is in the right corner, which is a little difficult to see initially imo, especially on screens w/ wider aspect ratios. maybe put some padding to make it no so far right?

In METS file, first traverse "Physical Structure" to get all pages (e.g. ALTO00001).
e.g. (https://s3.amazonaws.com/stanforddailyarchive/data.2013-nov/data/stanford/1999/12/01_01/Stanford_Daily_19991201_0001-METS.xml)

<structMap LABEL="Physical Structure" TYPE="PHYSICAL">
		<div ID="DIVP1" LABEL="The Stanford Daily" TYPE="Newspaper" DMDID="MODSMD_PRINT MODSMD_ELEC">
			<div ID="DIVP2" ORDER="1" ORDERLABEL="1A" TYPE="COVER_PAGE">
				<fptr>
					<par>
						<area FILEID="IMG00001"/>
						<area FILEID="ALTO00001" BETYPE="IDREF" BEGIN="P1"/>
					</par>
				</fptr>
			</div>
...

Then traverse the whole file to find all elements with attribute e.g. [FILEID="ALTO00001"]. Then find their parents that has TYPE="ARTICLE" and add this as part of the overlays for that parent.

e.g.

					<div ID="DIVL10" TYPE="CONTENT">
						<div ID="DIVL11" TYPE="ARTICLE" DMDID="MODSMD_ARTICLE1" LABEL="Earthquakes rock Stanford in '06, '89">
							<div ID="DIVL12" TYPE="HEADING">
								<div ID="DIVL13" TYPE="TITLE">
									<fptr>
										<area BETYPE="IDREF" FILEID="ALTO00001" BEGIN="P1_TB00006"/>
									</fptr>
								</div>
								<div ID="DIVL14" TYPE="AUTHOR">
									<fptr>
										<area BETYPE="IDREF" FILEID="ALTO00001" BEGIN="P1_TB00007"/>
									</fptr>
								</div>
							</div>

It will add P1_TB00007 and P1_TB00006 to overlay for MODSMD_ARTICLE1.

Then find corresponding positions and size in the ALTO file. (e.g. https://s3.amazonaws.com/stanforddailyarchive/data.2013-nov/data/stanford/1999/12/01_01/Stanford_Daily-ALTO/Stanford_Daily_19991201_0001_ALTO0001.xml)

@epicfaace do you think there's any easier way? Also, do we need to highlight anything other than TYPE="ARTICLE"? (e.g. TYPE="TITLE_SECTION" and TYPE="ADVERTISEMENT")

Currently, on https://archives.stanforddaily.com/, the first page of results is not shown.

1. Inspect network requests from devtools, and search for something.
2. Note that the following request is sent:

https://ehabp6fuc5.execute-api.us-east-1.amazonaws.com/prod?q=test&size=20&start=20&highlight.article_text=%7Bformat:%27html%27,max_phrases:5%7D&fq=publish_date:[%271892-01-01T12:00:00Z%27,%272015-01-02T12:00:00Z%27]

However, I believe we should actually be sending the following query:

https://ehabp6fuc5.execute-api.us-east-1.amazonaws.com/prod?q=test&size=20&start=0&highlight.article_text=%7Bformat:%27html%27,max_phrases:5%7D&fq=publish_date:[%271892-01-01T12:00:00Z%27,%272015-01-02T12:00:00Z%27]

When clicking to go to the next page of search results, don't replace the entire screen with a blank screen to show the loading screen. Instead, keep the existing results, but show the loading icon as an overlay.

currently, when you make a long search query w/ cloudsearch, it seems to not find results even if you took an exact excerpt from an article. To fix this, consider "or"ing together phrases or words from long search queries

https://stackoverflow.com/questions/31668928/amazon-cloudsearch-matching-long-strings-against-domain-documents

in the past, alumni were able to "sponsor" the addition of articles. Perhaps bring this feature back. Currently, archives are missing all the content pre 2014 that wasn't sponsored, as well as all content post 2014

Look into image to text services (amazon Textract, for example).

☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io

The dependency react-scripts was updated from 3.4.0 to 3.4.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

react-scripts is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

npm install --save --save-exact [email protected]

yarn add --exact [email protected]

Your Greenkeeper Bot 🌴