uplynxed / twitter-archive Goto Github PK

Currently only working in main user's bio for some reason.

This will save us from a lot of unnecessary if statement nesting.

For example:

Currently it just disappears due to layout changes - make it a sidebar slider using bootstrap.
Not having it severely limits functionality.

Loads infinitely / no 404 type message

The following error is getting thrown on interaction with working AND broken popovers, as well as mouseenter event on the document itself?

tooltip.js:444 Uncaught TypeError: Cannot read properties of undefined (reading 'trigger')
    at un._setListeners (tooltip.js:444:35)
    at new cn (tooltip.js:123:10)
    at new un (popover.js:42:1)
    at HTMLDocument.<anonymous> (tweets.js:1543:13)
    at HTMLDocument.dispatch (jquery.min.js:2:43184)
    at y.handle (jquery.min.js:2:41168)
    at HTMLDocument.c (rocket-loader.min.js:1:9405)
_setListeners @ tooltip.js:444
cn @ tooltip.js:123
un @ popover.js:42
(anonymous) @ tweets.js:1543
dispatch @ jquery.min.js:2
y.handle @ jquery.min.js:2
c @ rocket-loader.min.js:1

Blocked by #3
Avoid making #5 worse

Sidebar stays open after navigating.
This might obstruct visibility to the content on smaller viewports and lead to the user not realizing anything has happened.

Primarily so that any changes to the layout of the page doesn't require replacing index.html in active installations due to the title and meta tag customization being required to be done in it due to social media link embedding limitations.

Related to #5, keep template file size in mind.

Potentially move page layout templates to a separate file from the tweets layout templates?

• Make a new issue out of this when it comes to this point.

Read up on some JSRender/JSViews best practices and then break up templates into more modular parts.

• Show replies to replies (up to a certain amount with the option to render more).
• Show thread between linked reply and the original tweet.
• Show thread by original tweet poster before other replies.

Make them clickable but unscrapable

As with the picture content, video content should cycle through the given URL substitutions until we get one that (properly) works.
Currently twitter videos are playing for a second before they stop. We need to figure out how to keep it playing, or attempt to load a local archived version instead (if available).

• Multiple terms support (phrases enclosed in quotes)
• Modifier support (from:@username, mentions:@username)
• Replacing former usernames in queries with current ones (if defined in config)
• Including search results for former usernames when searching for a current one (if defined in config)

This can of course be customized, but it's a good idea to have a proper default.

The current page loads in JS and JSON from external files and remote locations, which does not work for local files.

Solution:
Compile all files into a single html page and serve it as a download option.

Issues:

• The loadConfig() and loadTweets() functions will need refactoring somehow to not load from external files if the contents of those files are already present in the html.

TODO:

• Refactor loadConfig() and loadTweets() to fix the issue mentioned above
• Write a JS function to compile all this and start a download (perhaps checking to see if it's not running compiled already).
  • If the above fails to work, do it via PHP. It only needs to run on a server anyhow.
• Make sure it is dynamic so it stays up to date with any changes.

Can be used for some boilerplate about text regarding the project as well as some customizable text by the person setting up the archive.

Also potentially a place to display license info, like for the Twemojis #24

When changing a filter on mobile, the offcanvas sidebar gets refreshed with the rest of the content area causing it to be closed without the state of the page being updated, thus breaking scrolling and any interaction. Only workaround is to do a full page reload.

Predetermined in the config.json file with the options "light", "dark" and "auto".

• Implement auto option to adhere to a user's browser settings

• Implement the option to allow or disallow theme toggling

• Implement theme toggling

• May require some kind of settings modal or offcanvas sidebar.

• Close issue upon Public Stable Version Milestone completion

Could expand and implement the custom icon font designed for valk.cam

Similar to this:

Twitter offers a way to archive your own account, so look into how the data is saved for that and figure out if we can parse it to be compatible with this project.

• Potentially and hopefully allow merging of self-archived data with scraped data for a more complete data set.
• Look into how and if it backs up media, and how and if we can use that to save unnecessary download traffic on #26

The wayback machine (WBM) rewrites all the media URLs even on a javascript and json entries level.
This means all media URLs are being rewritten to their web.archive.org archived counterparts.

However, when trying to retrieve a URL stored in an attribute in the html, the WBM rewrites that URL as well, removing the web.archive.org prefix from it at some point between the value in the html and the function in js.

This is an issue because we are storing media file replacements with the keys set to the (cleaned up) original media file URL.
Because of WBM messing with the URLs, all keys (and all property values) have the prefix, but the value we retrieved does not. We can not use this value to retrieve its associated media_replacements object. Rewriting the URL to include the prefix again might also not be an issue as the timestamp of when the URL was archived is part of that prefix, and there is probably no way to tell what it might be.

Potential fixes:

• Figure out how to get the full value from the html attribute without WBM interfering
• Figure out how to rewrite the URL again to include the web.archive.org prefix
• Rewrite the media_replacements object and related methods to avoid using a full URL as the index, perhaps just filename and filetype? Might have to account for potential conflicts with matching filenames/filetypes, but at least it's unlikely when it comes to media pulled from Twitter itself.

Problems with this approach:

• When selecting the next set of tweets to render onto the page:
  • No visual update occurs if the next set does not contain any matching tweets (major issue)
  • Inconsistency in number of tweets it appears to render per scroll (minor issue)
  • No visual update means a user might not feel prompted to scroll again, making it appear as if it is loading infinitely.
    • Implement a timeout based end to the "loading more tweets" message

Blocked by #4

Confirmed to crash in Opera GX by at least 1 person.

Can't replicate locally.

Implement a feature to allow for more familiar pathing.

Issues:

• Requires refactoring a lot of functions
• Requires .htaccess
  • Will therefore only be available on server hosted instances

Related to #3
See related issue for the problems it creates in the "front-end".

TODO

• Refactor filtering to filter in the data, much like the way it loads the media page, favorites, etc. (find and bind to an event?)
• Save and load filters to localstorage
• Decide on and potentially implement a toggle to save filters?

Problems happen when the cards fill up a lot of vertical space:

• Position Sticky means they go over the navbar when you scroll to the footer
  • Other options so far seem to push it over the footer instead
• Seems to be no good way to make cards fill up the height to a consistent max while also distributing them properly.

I think a programmatic solution is in order, unless Firefox finally releases its :has() CSS functionality.
Related commit: ac6791c

Allow sorting by:

• Date (Newest)
• Date (Oldest)
• Likes (Most)
• Likes (Least)
• Retweets (Most)
• Retweets (Least)
• Replies (Most)
• Replies (Least)
• Bookmarks (Most)
• Bookmarks (Least)

Blocked by #3 as we need to filter the list before sorting.

Potentially making use of the newer substituteMedia() function to make sure we grab the highest quality media.

1. Write function to get all of the media of at least the main user account in a list
2. Write function to go through the list and download the media

• Make sure the media is successfully downloaded
• If not, store the list of unsuccessful downloads and try again later until the list is done.

Current testing seems to suggest this is happening in retweeted quote tweets.
The specific example is related to tweet #INTERRO/status/1373556489391067137, which has a v2 format.

Utilize the format as seen in config.aliases:

"aliases": [
	{
		"current": "INTERRO",
		"former": ["heyinterrobang", "InterroTalks"]
	}
]

With the current implementation of letting the browser resolve the best working image source in order of (original > local > backup) means that every time we load any image media it has to try loading the failed sources again. This gets bad especially with user avatars.

Given that we will have already resolved most media down to a working source at least once, we should store that source and use that directly for subsequent instances of the media.

We could even store this info in localStorage to save us from having to do it all again on a reload.

Solution may be similar to favorite tweets export/import feature

• Save state to LS
• Load state from LS
• Determine if exporting has value, and if so:
  • Export state to JSON
  • Import state from JSON

See also #4

Figure out a way to remove any ads that may have snuck into the json archive

• Support changing per loop
• Support search queries reflecting in the title