Feature Request: Adding "add_jmes" and "replace_jmes" method to ItemLoader about itemloaders HOT 6 CLOSED

I had done something similar in parsel. scrapy/parsel#27

from itemloaders.

@voith that is really nice! You went the step further and got json out of html using xpath then using jmespath to get the stuff you want. However, my request is more for pure Json data that contains no html what so ever. I am scraping a lot of sites using the React framework and all of the data I need from these sites is in the Json. Plus, apis that consume Json is also another need to use jmespath. Just like xpath/css selectors are favor over regular expressions, jmespath are also favored over turning the json data into a python dict and access directly in loops and using key indexes. Jmespath provides a query string that can be used the same way as xpath/css selectors.

from itemloaders.

@IAlwaysBeCoding I understand your requirement. The above PR can also work only with Jmespath too.
My requirement was such that I needed chain selectors. I've scraped sites which give json with some embedded html in the response. So it would be nice to have nested selectors that would solve everyones need.
But the problem is that building such nested selectors is not that easy. My PR too is not a full proof solution. Its just a POC to take the idea ahead(I suspect that there will be several bugs in my implementation). There was quite a discussion held in scrapy/parsel#25 because of the complexity.

from itemloaders.

I think this feature should go to the ItemLoader repository: https://github.com/scrapy/itemloaders/

But I want it very much! The current workaround is probably defining a separate Item type for that (json) data item and then setting SelectJmes as input_processor.
That way, the item unfortunately is specific to the page.

from itemloaders.

I have this working in a project, and the path forward requires 4 changes in 3 repositories, 2 of them in Scrapy:

• scrapy/scrapy#4961
• scrapy/parsel#181
• Add JMESPath support to itemloaders
• Add JMESPath support to Scrapy (response.add_jmespath, otherwise you would need to use response.selector.add_jmespath)

from itemloaders.

I’m moving this to itemloaders since we have scrapy/scrapy#5894 for the remaining Scrapy work.

from itemloaders.