witjem / feedpls Goto Github PK

Context
I would like to have a web page where you can generate and check the feed configuration.

Proposal
Do web page like editor.swagger.io
TODO

Context
Now feedpls use go1.17, at the moment it's not the latest.

Proposal
Update to latest Golang version.

Context
Some website use yoast for SEO. This plugin on the HTML page has all needed information for feed.

<script type="application/ld+json" class="yoast-schema-graph">yoast json</script>

Proposal
Maybe do feed config like this?

- id: feed-id
  title: feed title
  description: feed desc
  url: http://example.com/news
  matchers:
    engine: xpath-yoast
    itemUrl:
      expr: //article//a/@href

Context
Some website shows time in one timezone.
For example: https://www.kyivpost.com/post/11824 shows in Europe/Kyiv timezone, but system parse time in UTC.

Proposal
Add tz field to publish section to feeds.yaml config.
Example:

published:
  expr: "//div[@class='post-info']/text()[2]"
  layout: "January 2, 2006, 3:04 pm"
  tz: "Europe/Kyiv"

Reminder
Should updates docs and examples too.

Integrate project with https://goreleaser.com

Context

Examples in the project can be broken for several reasons:

• Project codebase changed, but the examples have not been updated.
• Website of examples was changed, and therefore feeds.yml is outdated.

Propose

Added tests to GitHub pipeline which will be triggered by codebase changing or cron one a day.
Tests should run service from example and call endpoints from examples.

Context
Some website shows time with locale format, for example: Лютий 19, 2023, 12:19 in uk_UA locale

Proposal
Add locale field to publish section to feeds.yaml config.
Example:

published:
  expr: "//div[@class='post-info']/text()[2]"
  layout: "January 2, 2006, 3:04 pm"
  tz: "Europe/Kyiv"
  locale: "uk_UA"

For parse time can use https://github.com/goodsign/monday

Reminder
Should updates docs and examples too.

Context
RSS and Atom items has author field. It will be good if service will be support it.

Proposal
Add optional field author to feeds.yml config

matchers:
  engine: xpath
  author: # Optional props.
    name:
      expr: //meta[@name='author']/@content
    email:
      expr: //meta[@name='author']/@content

Should ignore author on the RSS if author name not found.

Reminder
Updates docs and examples too.

Context
pkg/errors has been archived, that's why better remove it from the project.

Proposal
Migrate pkg/errors to standard go errors

Find all place where use standart logs and replace it to go-pkgs/lgr

Context
Some news website add some prefix - DNN News to title, like: Apple presented a new iPhone - DNN News

Proposal
Add replace postprocesor which will be replace content by regex.

- id: some-blog-open-source
  title: The Some Blog | Open Source
  description: Open Source Archives
  url: https://someblog.blog/category/open-source/
  matchers:
    itemUrl:
      selector: article a
      attr: href
    title:
      selector: title
    description:
      selector: meta[name='description']
      attr: content
    published:
      selector: meta[name='parsely-pub-date']
      attr: content
      layout: 2006-01-02T15:04:05Z07:00
 
  postprocessors:  // new field on yaml
    - replace:
        field: title // can be: title, description
        from: some_regex
        to: ""

You can always offer your option

Context
Service use goquery for select feed data from web page. Will be good have alternative XPath parsing mechanism.

Proposal
Add XPath engine to select data for feeds.
Libs:

• https://github.com/antchfx/xpath
• https://github.com/antchfx/htmlquery
• https://github.com/antchfx/jsonquery
• https://github.com/antchfx/xmlquery

feeds.yml:

- id: github-blog-open-source
  title: The GitHub Blog | Open Source
  description: Open Source Archives
  url: https://github.blog/category/open-source/
  matchers:
   
    # new props
    engine: xpath # engine name

    itemUrl:
      selector: article a
      attr: href
    title:
      selector: title
    description:
      selector: meta[name='description']
      attr: content
    published:
      selector: meta[name='parsely-pub-date']
      attr: content
      layout: 2006-01-02T15:04:05Z07:00