google-scraper's People
google-scraper's Issues
[Question] Why batching keywords instead of processing them individually?
Issue
The current implementation processes keyword in batches of 20:
google-scraper/app/services/scraper/create_crawl_job_service.rb
Lines 20 to 26 in e2f9bcc
This leads to having to loop through keywords:
google-scraper/app/services/scraper/crawl_google_search_service.rb
Lines 16 to 27 in e2f9bcc
The downside of loops is that any unhandled error can stop the processing of the remaining keywords ๐ฅ
Expected
- Keywords should be first stored in the database with a flag to track the scraping status.
- A distinct background job to scrap the Google page is then scheduled.
The benefits are:
- Fast CSV upload request without losing any data.
- Observability via the Sidekiq dashboard is possible.
- Retries can be managed at the keyword level.
- The scraping of each keyword is isolated, i.e., one could error while others could be successful.
[Chore] Add database and Redis setup instructions to the README
Issue
While the README has detailed information on how you have approached the technical challenge of scraping, following the instructions were not sufficient to run the application. I need to have both Postgres and Redis running and running the Rails command to set up the Postgres database and run the migration.
Expected
The README file must contain all the required information for developers to set up the application in their local environments. So the following pieces of information are required: Ruby / Node versions and database(s) setup.
Note
Using Docker compose to set up the database services would make the project setup easier.
Start of the code review process ๐
Hello Tuan ๐ , thank you for your effort on the code submission. I am Olivier at Nimble, and I am happy to be the reviewer for our code review session.
During the review process, I would like to know more about your decisions, so I will create issues in some areas where there could be more improvements regarding your submission. Since solving every possible problem would take too long, I will prioritize the most important ones.
At the same time, please keep in mind that this is a bi-directional process, and I would love to hear back from you as well. Therefore, do not hesitate to ask questions or share your opinions about the implementation (if any) during the process.
We expect the code review process to be completed within 2-3 days. As a result, please make sure you are responsive during this process. If you need more time, please let us know as soon as possible so we can plan accordingly.
If we are aligned on any issue, and you would like to correct them, please address the issue using a proper git flow (creating a new branch, opening a Pull Request (PR) per issue, and merging the code when you are ready), and I will follow up on those fixes. Just so you know, you don't have to close any of my created issues after merging your PRs; I will help verify and close them for you once they pass. ๐
In the end, I do hope that you find the process enjoyable. Good luck and happy coding. ๐ค
[Bug] CSV upload fails
[Chore] Increase test coverage
Issue
Automated tests do not cover the core business logic (CSV upload, scraping).
Expected
While 100% test coverage is not required for this code challenge, all critical paths of the application should be unit-tested and, ideally, UI-tested.
Note
The main challenge is to test the scraper reliably, i.e., it should not make network requests.
[Feature] Render the stored HTML in the UI
Issue
While the Google search result content is stored in the database, users can only view the HTML as code. The page is not viewable.
google-scraper/app/views/crawl_results/show.html.slim
Lines 9 to 14 in e2f9bcc
As a result, a user cannot verify if the scraping results are correct.
Expected
The HTML content should be viewable on the application, e.g., there could be a view of the content on a new page or in a modal.
Note
There is a bit of a challenge to render HTML content from another page. Hence it is an expected feature of the application.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.