xofred / deviantart-gallery-downloader Goto Github PK

The generated credentials are brand new. I can log in thru the browser using them just fine (I'm unable to test them in other software such as 4pr0n/ripme due to changes in DA's website (see 4pr0n/ripme#425); I'm not sure if perhaps this issue is caused by the same changes?

[efreak:~/source/deviantart-gallery-downloader] master 17s 1 ± ruby fetch.rb -n http://starkiteckt.deviantart.com/gallery/
Using netrc's credential
Connecting to deviantART
Logging in
undefined method `username=' for nil:NilClass
will retry after 1 second
undefined method `username=' for nil:NilClass
will retry after 1 second
undefined method `username=' for nil:NilClass
will retry after 1 second
undefined method `username=' for nil:NilClass
login failed after 3 retries
You might not be able to fetch the age restricted resources
Connecting to gallery
/home/efreak/source/deviantart-gallery-downloader/lib/deviantart_gallery_downloader.rb:24:in `fetch': undefined method `[]' for nil:NilClass (NoMethodError)
        from fetch.rb:3:in `<main>'

The multi-folder downloading version has been working great, but obviously with the latest changes that you just fixed, it has not been working.

Any chance we could get those changes merged together?

If not, I can probably figure it out, but I'm not super familiar with ruby, so if you were willing to merge 'em, it'd probably be easier, but if you are not willing, I will work on it, and submit a pull if you'd like once I figure it out.

Hi,

I tried to use the tool to salvage my late DeviantArt account, but was unable to connect to it. After doing some debugging, I discovered that DeviantArt sent out different content to the Mechanize compared to desktop or mobile, but adding the User-Agent header to something desktop-y worked. I was, however, unable to log in, but that might have been a different issue altogether. I didn’t delve more into that, as I did not have any mature/private content I needed to scrape.

Here’s the diff of the changes I did to make the core download work, at least. Note: As I am not a Ruby developer, it’s makeshift and does not follow the Ruby style guide.

ruby with_login.rb EMAIL PASSWORD http://s-c.deviantart.com/gallery/

108 items seen. 84 items end result.

I already have installed bundler, netrc and mechanize but I don't know hot to install this as it isn't a .gem file. Cheers.

I've attempted to run the master and the dev branch of this and have had not been getting the full number of deviations.
for example ruby fetch.rb -n https://phillip-the-2.deviantart.com/gallery/?catpath=/

This only downloads the first 12 deviation, while there's 235 currently in the gallery.

I have set the display to 24 thumb nails and paging to Click through pages. Is there something that needs to be included in the "?catpath=" part of the URL to continue paging?

• More secure login method, username and PW are in command not good for multi-user systems
• ~/.netrc capability for auth, youtube-dl has this
• Default behavior to mirror albums when sent to main /gallery and organize rips accordingly locally
• Inclusion in gem repository

Great work love the script. Only one that works or isn't crashy. Thank you.

Today i got following error

Updating netrc's entry
Logging in
403 => Net::HTTPForbidden for https://www.deviantart.com/users/login -- unhandled response
Login failed after 3 retries
You might not be able to fetch the age restricted resources
Connecting to gallery
gallery has no images, abort

This is more of a feature request, and feel free to say no, but:

Would it be possible to make the downloader compatible with /gallery/?catpath=/?
Under normal circumstances, if you provide it with a gallery URL, it downloads all images in that page. However, users can also move images into folders, which causes them to no longer show up in that list. If you go to the URL /gallery/?catpath=/ then it will however show all images uploaded by that person.

If this is impractical or impossible, I understand. Just asking because this is the best deviantart archiver I've found, and manually copying in each deviation folder takes a significant portion of time.

As with #15, I'm not able to use the 'log in' feature anymore. Instead of fetching the login page, dA returns the 'we suspect you're a bot' page (which includes a captcha). furaffinity-dl uses a cookies.txt file to bypass FA's login page (which also has a captcha), so something like that might work here. I'm looking right now at another gem by the makers of Mechanize for handling cookies (http-cookie).