smattr / needtoknow Goto Github PK

Some servers use an etags or modified header in a way that, if the client takes advantage of it, can save bandwidth and server load from duplicate fetches. feedparser supports this, so we should probably be using it: https://pythonhosted.org/feedparser/http-etag.html

CC BY-NC is an unusual license for software, I don't know what your reasons for choosing it are but you might want to look into changing it. Creative Commons advises that their licenses should not be used for software: https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software

When running interactively, we currently get the following warning:

.../needtoknow/needtoknow.py:82: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if not isinstance(conf, collections.Mapping):

One particular site that we're monitoring with the htmldiff feeder is producing diffs that are garbage. The content is a valid diff but the data being diffed looks like something binary being interpreted as text. Without any further investigation, my initial guess is that the content is being delivered compressed and we're misinterpreting the HTTP header. The site involved and the symptoms are viewable from current logs.

There is no documentation of the config format, could you upload a example file?

We are embedding images in emails and then linking to them using Content-ID (CID) URLs. We use the index of the image in the email to construct the CID. Unfortunately, RFC 2111 has this to say about CIDs:

The Content-ID of a MIME body part is required to be globally unique.

I did not realize that "globally" here means across your entire message store (or even literally globally if you choose to take it that far). This is apparently an intentional feature to allow cross references in messages. It is a little bit crazy (displaying any email is O(n) in the number of emails you have) but apparently iOS's mail app chooses to implement it at its word.

As a result, on iOS you occasionally see the wrong image displayed in a needtoknow email. To get the right email you need to force quit mail to flush its cache. We should just fix this by using a hash or even a UUID for CIDs.

Relatedly, this also probably explains why loading inline images sometimes stalls indefinitely on iOS. Some more discussion:

• https://css-tricks.com/forums/topic/iphone-mail-displaying-wrong-images/
• https://discussions.apple.com/thread/7456614
• https://discussions.apple.com/thread/5146502

imaplib doesn't handle the quoting or encoding of mailbox paths by itself, so we have to do it ourselves. This means at present needtoknow is not compatible with names including spaces or special characters. imap uses a nonstandard utf7 encoding for mailbox names, but there are a few encoders made by other people, e.g: https://github.com/MarechJ/py3_imap_utf7

I already implemented quoting in my fork, and I might do the encoding if I have a bit of spare time.

On a macOS machine, Power Nap results in stalling needtoknow. I haven't investigated what's going on and I don't know exactly what Power Nap does, but I suspect it's putting network sockets into some dormant state where reads block. Because we're not setting timeouts, perhaps this results in us stalling.

1. Will this project work with SMS Gateways? (ie, sending a text email to a phone number)
2. Will this work with geojson (with a bit of modification) of is it too different from rss?