Because Chegg only gave few days to access their solutions for free when I rent their books so I decided to create a web crawler to save all the needed solutions offline.
I developed a simmillar bot for scraping answers and used 2capcha to bypass recapcha , there was a callback function call "handlecapcha" which i used to callback and bypass capcha. This doesnt seem to work now. any idea Why?
I am new to his. How to start the program? I can't find any exe file.
I'm guessing this doesn't have support for Mac?
I'm also curious how you get around their bot detection. Whenever I try to scrape answers I get a message that I'm using automation tools and that prevents me from scraping the web site.
Images aren't downloaded, but the link to the Chegg cdn is called. Must have an internet connection to view the pictures, and if Chegg restricts access to its cdn many solutions become useless. Is it possible to have the program download the actual image files in a solution?
The program successfully loads the page & logs in, it then proceeds to sequentially load each solution page. At the beginning of each chapter the program creates a folder, the first is "Chapter 1", and after it has scraped the information from the last question in the chapter it creates a file in the folder "Chapter 1" called "-nosolution.html". The file contains a single line of text which is "no solution".
When the program has the answer pages loaded, a popup appears. I'm not sure if this has any effect, whether I close it or leave it open I still have the same issue.
I have installed the program using the latest .msi installer, and also have Visual C++ Runtime 2013 installed on my machine (both 32 and 64 bit).
Actually how to start this program. Could not reach the 3rd step as you show (i.e. OPEN THE PROGRAM and copy the url into the program)
When logging in, the verification code interface displays abnormally and cannot work
Basically, the title. It asks for device verification. window too small to enter anything
Hi,
I'm trying to download some solutions, but I have no succes on it. Using Release 1.1, I get no errors but it looks like the crawler only opens the first page and does nothing. The application looks be static (not frozen) and the output folder is never populated.
This is before I click "Execute":
Then, I got this and it is static:
This is the error log:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at WindowsFormsApp1.MainForm.<Crawl>d__6.MoveNext() in C:\Projects\chegg-solutions-saver\src\CheggSolutionsCrawler\MainForm.cs:line 103
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at WindowsFormsApp1.MainForm.<LoopThroughEachSolution>d__4.MoveNext() in C:\Projects\chegg-solutions-saver\src\CheggSolutionsCrawler\MainForm.cs:line 70
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at WindowsFormsApp1.MainForm.<InitializeChromium>d__1.MoveNext() in C:\Projects\chegg-solutions-saver\src\CheggSolutionsCrawler\MainForm.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c.<ThrowAsync>b__6_0(Object state)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4121.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
WindowsFormsApp1
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/temp/CheggCrawler/WindowsFormsApp1.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4121.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4110.0 built by: NET48REL1LAST_B
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3761.0 built by: NET48REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3761.0 built by: NET48REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4121.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3761.0 built by: NET48REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.8.3761.0 built by: NET48REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
CefSharp.WinForms
Assembly Version: 71.0.0.0
Win32 Version: 71.0.0.0
CodeBase: file:///C:/temp/CheggCrawler/x86/CefSharp.WinForms.DLL
----------------------------------------
CefSharp.Core
Assembly Version: 71.0.0.0
Win32 Version:
CodeBase: file:///C:/temp/CheggCrawler/x86/CefSharp.Core.DLL
----------------------------------------
CefSharp
Assembly Version: 71.0.0.0
Win32 Version: 71.0.0.0
CodeBase: file:///C:/temp/CheggCrawler/x86/CefSharp.DLL
----------------------------------------
System.ServiceModel
Assembly Version: 4.0.0.0
Win32 Version: 4.8.4121.0 built by: NET48REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.ServiceModel/v4.0_4.0.0.0__b77a5c561934e089/System.ServiceModel.dll
----------------------------------------
HtmlAgilityPack
Assembly Version: 1.8.14.0
Win32 Version: 1.8.14
CodeBase: file:///C:/temp/CheggCrawler/HtmlAgilityPack.DLL
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
So, I have followed the instructions, logout before execute, but nothing works.
Using .net Core version, I got the following problem:
Thank you very much!
Is it possible to have the crawler save the problem as well as the solution? Figuring out which problem goes with each solution is sometimes challenging
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.