Giter VIP home page Giter VIP logo

dmm-crawler's Introduction

🇺🇸 🇯🇵 Build Status Gem Version Build Status Code Climate Gem

DMM Crawler

⚠️ Cation ⚠️

FANZA does not accept crawling pages so I don't recommend to use this gem.

I do not take any responsibility or liability for any damage or loss caused by mine gem.

What is DMM Crawler

Show DMM and DMM.R18's crawled data. Now, All rankings for doujin is crawlable.

Installation

On your gemfile.

gem 'dmm-crawler'

Usage

DMM.R18 Doujin Ranking

require 'dmm-crawler'

include DMMCrawler

client = Client.new do |agent|
  agent.ignore_bad_chunking = false
end

client.rankings(term: '24', submedia: 'cg')
# =>
# {
#   title: "title",
#   title_link: "title url",
#   image_url: "Link to title"s main image",
#   submedia: "cg",
#   author: "author",
#   informations: [{key: 'key', value: 'value'}],
#   rank: '1'
#   tags: ["tag1", "tag2"]
# }

For example, Above command will show the doujin cg 24's ranking.

Available arguments

  • Arguments for the term is available to use all, comic, cg, game, voice.
  • Arguments for the submedia is available to use 24, weekly, monthly, total.

From an art's URL

require 'dmm-crawler'

include DMMCrawler

client = Client.new do |agent|
  agent.ignore_bad_chunking = false
end

URL = 'https://www.dmm.co.jp/dc/doujin/-/detail/=/cid=d_087090'

client.get_attributes(URL)
# =>
# [
# ...
# ]

It returns art information by Array.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

dmm-crawler's People

Contributors

sachin21 avatar

Stargazers

 avatar  avatar

Watchers

 avatar

dmm-crawler's Issues

Change the author name element

Summary

Now, We use the element of p.circleProductTitle__main to fetch author name. but this is sometimes, return an empty string.

ToDo

  • Fix a bug.

v0.0.7

Hotfix release

  • Push to rubygems.org.
  • Add tag
  • Write release note.

v0.1.2

Tasks

  • Write change logs.
  • Add a tag.
  • Push to rubygems.org.

Do not lock gem versions

Summary

Do not lock gem version in dmm-crawler.gemspec.

Problem Summary

dmm-crawler have a security issue of RuboCop. So I don't want to lock gem versions in dmm-crawler.gemspec.

screen shot 2017-11-18 at 10 02 38 pm

v0.0.5

Tasks

  • Add a tag for version.
  • Release chang logs.
  • Push to rubygems.org.

Do not fix submedia

Summary

I want to also crawl besides doujin genre.

ToDo

  • Remove code to fix.

v0.2.1

Tasks

  • Write change log.
  • Add a tag.
  • Push to rubygem.org.

Convert double quotes to single qoutes

Problem Summary

description_raw": "<p class=\"rank-desc\"></p>"

Now, content is escaped, So when I use this string, I should remove backslush.

Tasks

  • Convert it.

v0.1.3

Tasks

  • Write change logs.
  • Add a tag.
  • Push to rubygems.org.

v0.1.0

Release Note

  • Add an attribute raw data of description that include HTML tag.

v0.0.2

Tasks

  • Add the tag to be bumping commit.
  • Write CHANGELOG.md

v0.1.4

Summary

  • Move the series to last.

Tasks

  • Write change logs.
  • Add a tag.
  • Push to rubygems.org.

v0.0.4

Tasks

  • Write release note.
  • Add the tag.
  • Bump version.

v0.2.3

Change log

  • Do not lock gem versions in 'dmm-crawler.gemspec'.

ToDo

  • CHANGELOG.md.

v0.1.1

Tasks

  • Write change logs.
  • Add a tag.
  • Push to rubygems.org.

v0.2.2

Change log

  • Do not manage 'Gemfile.lock'.

ToDo

  • CHANGELOG.md.

v0.0.6

Tasks

  • Push to rubygems.org.
  • Add a tag.
  • Write release note.

Do not fixing types of submedia

Reports

Submedias are many exists on dmm.co.jp. So I thought that fixing is not needed.

Tasks

  • Remove fixing types of submedia.

Add crawling information

Features

Add information for the following.

  • 配信開始日
  • 利用期限
  • ファイル容量
  • 音声本数
  • ページ数
  • ゲームジャンル
  • 動画本数
  • シリーズ
  • 題材

Tasks

  • Fetch more information

v0.0.3

Change logs

  • Rename the method from books to arts.
  • Correct examples for README.md.

v0.2.0

Tasks

  • Write change log.
  • Add a tag.
  • Push to rubygem.org.

v0.1.5

Tasks

  • Write change logs.
  • Add a tag.
  • Push to rubygems.org.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.