IsFeed K/V not transferred to redirection about news-crawl HOT 9 CLOSED

Thanks. The number of crawled pages went significantly up after deploying the changed config (feed.sniffContent: true) and refetching all redirected seed feeds. Opened #14 and DigitalPebble/storm-crawler#386 to force a refetch of redirected seeds with low frequency (weekly or monthly).

from news-crawl.

Could simply rewrite the seed files for now or add a better auto detection

from news-crawl.

Improved detection in SC [master 1dd6b24] Feed bolt : detection based on content

from news-crawl.

Should also refetch the original seed in regular intervals, in case it redirects to dynamically changing URLs.

from news-crawl.

@sebastian-nagel could you give a few examples of redirs for which the auto detection doesn't work? Thanks

from news-crawl.

One example (the newspaper I daily get on paper): http://www.taz.de/rss.xml redirects to http://www.taz.de/!p4608;rss/ and the server sends Content-Type: text/xml;charset=utf-8 which makes the auto detection fail.
The auto detection is nice, esp. because it allows to identify additional seeds. But it may be not sufficient for certain cases, e.g., if a seed is redirected to a dynamic ephemeral URL: we then need to fetch and follow the redirected seed in regular intervals.
But I'll prepare a complete list of problematic seeds to see whether such cases exist.

from news-crawl.

The autodetection is not just based on the mimetype but also on the content and actually works with the URL above.

from news-crawl.

Ok, got it: the auto-detection wasn't enabled.

from news-crawl.

Closing as not a bug. It makes no sense to pass on the feedKV blindly to the redirection : it might be a feed too, or not but there is no guarantee of it.

from news-crawl.