yorkittran / google-scraper Goto Github PK

Issue

The current implementation processes keyword in batches of 20:

This leads to having to loop through keywords:

The downside of loops is that any unhandled error can stop the processing of the remaining keywords 💥

Expected

• Keywords should be first stored in the database with a flag to track the scraping status.
• A distinct background job to scrap the Google page is then scheduled.

The benefits are:

• Fast CSV upload request without losing any data.
• Observability via the Sidekiq dashboard is possible.
• Retries can be managed at the keyword level.
• The scraping of each keyword is isolated, i.e., one could error while others could be successful.

Issue

While the README has detailed information on how you have approached the technical challenge of scraping, following the instructions were not sufficient to run the application. I need to have both Postgres and Redis running and running the Rails command to set up the Postgres database and run the migration.

Expected

The README file must contain all the required information for developers to set up the application in their local environments. So the following pieces of information are required: Ruby / Node versions and database(s) setup.

Note
Using Docker compose to set up the database services would make the project setup easier.

Hello Tuan 👋 , thank you for your effort on the code submission. I am Olivier at Nimble, and I am happy to be the reviewer for our code review session.

During the review process, I would like to know more about your decisions, so I will create issues in some areas where there could be more improvements regarding your submission. Since solving every possible problem would take too long, I will prioritize the most important ones.

At the same time, please keep in mind that this is a bi-directional process, and I would love to hear back from you as well. Therefore, do not hesitate to ask questions or share your opinions about the implementation (if any) during the process.

We expect the code review process to be completed within 2-3 days. As a result, please make sure you are responsive during this process. If you need more time, please let us know as soon as possible so we can plan accordingly.

If we are aligned on any issue, and you would like to correct them, please address the issue using a proper git flow (creating a new branch, opening a Pull Request (PR) per issue, and merging the code when you are ready), and I will follow up on those fixes. Just so you know, you don't have to close any of my created issues after merging your PRs; I will help verify and close them for you once they pass. 😇

In the end, I do hope that you find the process enjoyable. Good luck and happy coding. 🤘

Issue

Upon uploading a CSV file, the UI shows a success message:

However, the web server log shows an authorization issue:

As a result, I have not been able to verify the scraping results yet 😅

Issue

Automated tests do not cover the core business logic (CSV upload, scraping).

Expected

While 100% test coverage is not required for this code challenge, all critical paths of the application should be unit-tested and, ideally, UI-tested.

Note
The main challenge is to test the scraper reliably, i.e., it should not make network requests.

Issue

While the Google search result content is stored in the database, users can only view the HTML as code. The page is not viewable.

As a result, a user cannot verify if the scraping results are correct.

Expected

The HTML content should be viewable on the application, e.g., there could be a view of the content on a new page or in a modal.

Note
There is a bit of a challenge to render HTML content from another page. Hence it is an expected feature of the application.