thestanforddaily / archives-web Goto Github PK
View Code? Open in Web Editor NEWHelper functions and web app for METS/ALTO archive viewing.
Home Page: https://archives.stanforddaily.com
Helper functions and web app for METS/ALTO archive viewing.
Home Page: https://archives.stanforddaily.com
I get a blank screen and this message:
Instead, we want it to not have a blank screen, but just give an error message on the UI.
We should eventually also add the (physical) paper published after 2014 to the archives site. For now, we can link newer dates to our main site such as https://www.stanforddaily.com/2019/06/16/.
We should also use the Internet Archive Wayback Machine periodically to store our articles and link to that from our archives site.
e.g. clicking on author name brings us to all articles published by that author
For example, if I select "ASSU President", "ASSU Parlimentarian" [sic], or "STAFF FOOTBALL WRITERS", I get no results.
We should probably be getting at least one result here.
We should add highlighting in search results for fields such as title and author.
Need to combine
https://stanforddailyarchive.s3.amazonaws.com/data.2013-nov/data/stanford/1991/09/19_01/Stanford_Daily_19910919_0001-METS.xml
and
currently, we only show the latter (19_02) but miss all the (19_01) pages -- see https://archives.stanforddaily.com/1991/09/19?page=1§ion=MODSMD_ARTICLE2#issue
This is important because we're actually missing articles on the new site (which were there previously on the old site).
"About this collection" "Search the Collection" and "Subscribe" are currently in the same row. Fix this by orienting each header and its respective components vertically one after the other.
Bonus: Make this feature responsive by orienting vertically if the page size is small and switch to horizontal when the page size is big.
1.0.0-beta.12
to 1.0.0-beta.13
.This version is covered by your current version range and after updating it in your project the build failed.
react-bootstrap is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
create a form on frontend for users to create and submit corrections to articles. Then, someone at TSD needs to approve the changes, and run the pipeline (to be implemented, see: #55) to propagate the changes.
For example, try going to https://archives.stanforddaily.com/1985/08/09?page=11§ion=DIVL319#article (clear cache) -- the request takes 15 seconds (1.6 MB)
checkout a subdirectory of repo
Need to add new IAM policy - archives-cloudsearch-deploy
[secrets](https://github.com/TheStanfordDaily/stanforddaily/blob/master/.github/workflows/deploy.yml
.github/workflows/deploy.yml)
[matrix jobs](https://github.com/TheStanfordDaily/stanforddaily/blob/master/.github/workflows/deploy.yml
.github/workflows/deploy.yml)
would be cool to see how many times a page has been viewed. Not sure how to do this though.
searchworks.stanford.edu has a good UI for filtering searches, which would be good to base our UI on
or a union of these
create a guide/key/help bar that user can expand which shows some hints for searching e.g. you can add quotations to do an exact text match, just like when google searching.
🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial
branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/
.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.
integrate cloudsearch suggester into the frontend. suggester suggests against article_text (high fuzzy search) or article_title (high fuzzy search) or author (exact match), depending on what index field the user is searching by.
in search results
Fetch first page’s first article title and display it?
Use seadragon
The box ids for each article are stored in the METS file
The ALTO file stores the actual text for each box, and coordinates for each box.
https://openseadragon.github.io/examples/ui-overlays/
https://openseadragon.github.io/examples/tilesource-sequence/
Getting errors when running the script
The last successful file parsed was text/1894/4/26/DIVL151.txt Adv. 17 Page 4
(node:15545) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'nodeValue' of undefined
at Paper._callee$ (/home/ec2-user/environment/node_modules/@thestanforddaily/archives-web/lib/classes/Paper.js:85:63)
at tryCatch (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:45:40)
at Generator.invoke [as _invoke] (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:271:22)
at Generator.prototype.(anonymous function) [as next] (/home/ec2-user/environment/node_modules/regenerator-runtime/runtime.js:97:21)
at asyncGeneratorStep (/home/ec2-user/environment/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
at _next (/home/ec2-user/environment/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:15545) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15545) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
right now, we're limited to 10,000 search results for any search. to unlock maximum potential, use cloudsearch cursor
https://docs.aws.amazon.com/cloudsearch/latest/developerguide/paginating-results.html
Make this similar to the "share to facebook" etc. buttons that are on the main website.
☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io
6.11.1
to 6.12.0
.This version is covered by your current version range and after updating it in your project the build failed.
query-string is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
Go to this page, and you see an article titled "Work on the Track" (https://archives.stanforddaily.com/1896/02/05?page=1§ion=MODSMD_ARTICLE2#article).
Now, search "work on the track" and this article doesn't even show up... (https://archives.stanforddaily.com/search?page=1&pagelen=20&q=work%20on%20the%20track&year_end=1897&year_start=1892) something wrong with the search process?
User can view possible author titles and select one or multiple (union of selected if multiple) to filter search results by
For example, https://archives.stanforddaily.com/2002/05/31 could have a link to https://web.archive.org/web/20020602164045/http://www.stanforddaily.com/.
And https://archives.stanforddaily.com/2002/05/31#page=1§ion%5B%5D=MODSMD_ARTICLE4 could have a link to https://web.archive.org/web/20020629021038/http://www.stanforddaily.com/tempo?page=content&id=8524&repository=0001_article.
These online articles could also be the source for archives-text
.
For earlier papers and articles, we probably need to find the links manually.
For later papers and articles, maybe there will be some patterns in the URL?
1.5.1
to 1.5.2
.This version is covered by your current version range and after updating it in your project the build failed.
interactjs is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
Right now, search is in the right corner, which is a little difficult to see initially imo, especially on screens w/ wider aspect ratios. maybe put some padding to make it no so far right?
In METS file, first traverse "Physical Structure" to get all pages (e.g. ALTO00001
).
e.g. (https://s3.amazonaws.com/stanforddailyarchive/data.2013-nov/data/stanford/1999/12/01_01/Stanford_Daily_19991201_0001-METS.xml)
<structMap LABEL="Physical Structure" TYPE="PHYSICAL">
<div ID="DIVP1" LABEL="The Stanford Daily" TYPE="Newspaper" DMDID="MODSMD_PRINT MODSMD_ELEC">
<div ID="DIVP2" ORDER="1" ORDERLABEL="1A" TYPE="COVER_PAGE">
<fptr>
<par>
<area FILEID="IMG00001"/>
<area FILEID="ALTO00001" BETYPE="IDREF" BEGIN="P1"/>
</par>
</fptr>
</div>
...
Then traverse the whole file to find all elements with attribute e.g. [FILEID="ALTO00001"]
. Then find their parents that has TYPE="ARTICLE"
and add this as part of the overlays for that parent.
e.g.
<div ID="DIVL10" TYPE="CONTENT">
<div ID="DIVL11" TYPE="ARTICLE" DMDID="MODSMD_ARTICLE1" LABEL="Earthquakes rock Stanford in '06, '89">
<div ID="DIVL12" TYPE="HEADING">
<div ID="DIVL13" TYPE="TITLE">
<fptr>
<area BETYPE="IDREF" FILEID="ALTO00001" BEGIN="P1_TB00006"/>
</fptr>
</div>
<div ID="DIVL14" TYPE="AUTHOR">
<fptr>
<area BETYPE="IDREF" FILEID="ALTO00001" BEGIN="P1_TB00007"/>
</fptr>
</div>
</div>
It will add P1_TB00007
and P1_TB00006
to overlay for MODSMD_ARTICLE1
.
Then find corresponding positions and size in the ALTO file. (e.g. https://s3.amazonaws.com/stanforddailyarchive/data.2013-nov/data/stanford/1999/12/01_01/Stanford_Daily-ALTO/Stanford_Daily_19991201_0001_ALTO0001.xml)
archives-web/src/classes/Page.js
Lines 23 to 43 in 8a48f38
@epicfaace do you think there's any easier way? Also, do we need to highlight anything other than TYPE="ARTICLE"
? (e.g. TYPE="TITLE_SECTION"
and TYPE="ADVERTISEMENT"
)
Currently, on https://archives.stanforddaily.com/, the first page of results is not shown.
However, I believe we should actually be sending the following query:
currently, when you make a long search query w/ cloudsearch, it seems to not find results even if you took an exact excerpt from an article. To fix this, consider "or"ing together phrases or words from long search queries
in the past, alumni were able to "sponsor" the addition of articles. Perhaps bring this feature back. Currently, archives are missing all the content pre 2014 that wasn't sponsored, as well as all content post 2014
Look into image to text services (amazon Textract, for example).
☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io
3.4.0
to 3.4.1
.This version is covered by your current version range and after updating it in your project the build failed.
react-scripts is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
v3.4.1 is a maintenance release that includes minor bug fixes and documentation updates including upgrading Babel to fix a bug in the 7.8 release line. This release also brings support for TypeScript 3.8.
react-scripts
babel-preset-react-app
cra-template
, eslint-config-react-app
, react-scripts
cra-template-typescript
, cra-template
react-scripts
react-scripts
babel-preset-react-app
, create-react-app
, react-dev-utils
, react-error-overlay
, react-scripts
react-scripts
Inside any created project that has not been ejected, run:
npm install --save --save-exact [email protected]
or
yarn add --exact [email protected]
The new version differs by 13 commits.
d2f813f
Publish
7641a3c
Prepare 3.4.1 release
d5b527f
Update to Babel 7.9 (#8681)
6adb82a
Add React.StrictMode to default templates (#8558)
a452ddc
Bump dependencies (#8620)
3f699fd
Fix proxying API request docs (#8515)
4d26208
Use native ESLint behaviour when extending (#8276)
8ba0ccb
Whitelist main in template.json (#8539)
7d3b72c
Update template example in docs (#8561)
2030ee1
Fix optional chaining and nullish coalescing support (#8526)
038e6fa
Widen eslint-config-react-app peer dependency versions (#7790)
7e6d6cd
Closes webpack dev server and exits process on "end" stdin (#7203)
af926d5
Bump pnp-webpack-plugin (#8509)
See the full diff
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.