malparty / google-search-ruby Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 1.01 MB

A Google scrapper focus on AdWords data. Build with Rails.

Shell 0.25% Ruby 83.26% Dockerfile 1.34% JavaScript 2.10% SCSS 0.46% TypeScript 0.08% HTML 12.46% Procfile 0.04%

google-search-ruby's People

Contributors

Watchers

google-search-ruby's Issues

[Frontend] As a User, I can reset my password via the Web UI

Build UI for #28

Why?

In order to unlock the user in case he forget his password or if he wishes to update it.

Who Benefits?

Users who forget their password
Users showing interests in security concerns
Support Team (as this reduce number of manual requests)

[Backend] As a User, I can see a list of my previous keyword results

Why?

In order to compare different keywords & review past results.

Additional notes:

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI
API expose the feature to enable third app to consume it

No Gos:

No search feature [yet]
No filter feature
No multiple keywords [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

[FrontEnd] Bug - Layout is not responsive

What happened?

Header user-menu as well as footer do not display properly on mobile devices.

Footer: `Nimble` getting out from footer	Header: `user-menu` not hidden & hamburger menu not working

Who is impacted?

User using mobile devises to connect.

Solution

Review UI code in Layout views.

[API] As a user, my keywords are being parsed in background

Why?

So that I do not need to wait before getting the file-well uploaded response from the application.
So that I can close my web-browser while parsing continue.

Additional notes:

This is a direct improvement to #9
Work is only backend, there is no frontend related task.

Who benefits?

Logged users who just uploaded a CSV file.

[Chore] -nice-to-have- Improve presenter specs with Fabricate.build

Why?

To save time when running tests (as no DB queries will be involved)

As identified here.

Who will benefits?

Developers

[Backend] As a User, I can search across keywords and results

Why?

In order to provide insights/learning to the user, from the data gathered.
While individual keywords results might solve localized problems, our users needs to understand trends across multiple keywords.

Additional notes:

Problems to solve:

Enable the user quickly find a given keyword and see its results
Enable the user to perform advanced queries in order to learn insights from the data gathered

No Gos:

Search queries are not persisted [yet]

Who Benefits?

Logged Users.

[Frontend] As a user, I can save/share a filtered view from its URL

Frontend for #87

Why?

In order to save time once a user found it's perfect filters and wants to reapply them later on.

Who Benefits?

User who regularly uses the advanced search feature #10 / #15

[Frontend] As a User, I can see a list of my previous keywords

Build Front End for #26

Why?

In order for the user to navigate through his previously uploaded keywords.
In order to provide an entry point for #8 & #12

Additional notes:

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI

No Gos:

No search feature [yet]
No filter feature
No multiple keywords upload/parsing [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

Configure UI test automation

Why?

Save time with automation and enforce quality checks by:

Enable UI test execution via GitHub Actions on PR creation/updates.
Enforce UI tests before PR validation/merge.

Who Benefits?

Development team

[Chore] -nice-to-have- Progress-card UI improvements

Why?

The current progress notification card has some space for improvements:

Missing padding for progress-bar inner text
progress-bar inner content hidden when small percent
'keywords' not singularized when only 1x keyword

Who Benefits?

The user thanks to improve UX.

[Chore] Remove false positives in Review workflow

Why?

Many times while all auto tests are passing, Pull Requests are showed as "NOK" ❌ simply because

Danger cannot find code coverage data
Danger cannot find undercover reports
false positive: Danger indicates a package vulnerability "2.10 is at risk, please update > 2.2"

Who benefits?

Dev team & code reviewers ;-)

[Frontend] As a User, I can see a list of my previous keyword results

Build advanced UI for #8

Why?

In order to compare different keywords & review past results.

Additional notes:

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI
API expose the feature to enable third app to consume it

No Gos:

No search feature [yet]
No filter feature
No multiple keywords upload/parsing [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

[Chore] -nice-to-have- Review all Request specs to use json helper

Included with Rails-templates, we have a json_response method that simplifies the JSON.parse(response.body, symbolize_names: true).

It also symbolizes names making the overall test cleaner.

We can review the previous request tests with this improvement.

Example:

Before

expect(JSON.parse(response.body)['data']['attributes']['status']).to eq('pending')

After

expect(json_response[:data][:attributes][:status]).to eq('pending')

Missing ticket/issue for password reset

I am missing the user password ticket. I think you can create a ticket for reseting password(No need for API I think)

[Chore] Setup Turbo.hotwired.dev

Why?

To learn about it (yeah, we are in an IC)
Provides with our user a more dynamic experience without adding much complexity in the Frontend code

Who benefits?

Developer: less JavaScript & Frontend maintenance
Users: Faster and more dynamic app

Configure deployment automation

Why?

Ensure no manual action is made for deployment, thus reducing the risks for human errors.
Increase traceability of events while enforce PR & validations before any deployment.

Additional notes:

Define which hosting provider to use (Heroku, AWS, Google Cloud or Digital Ocean)
Configure auto-deploy for 2 environments:

Staging: develop branch
Production: main branch

Who Benefits?

Project team & stakeholders

[Frontend] As a User, I can upload a csv file in order to run multiple keyword searches

Build advanced UI for #9

Why?

Enable the user to save time by processing multiple keywords from 1 single user action (update a file).
Save even more time by processing keywords asynchronously.

Additional notes:

Problems to solve:

Enable user to run from 1 to 1.000 search queries from 1 single action (upload csf file)
Enable user to be aware if the selected csf file is not valid
Enable user to see the file name from a given keyword
Enable third parts to submit files via API too

No Gos:

No fancy drag & drop
No multiple files upload
No dynamic status progress update [yet]
No search/filters [yet]

Who Benefits?

Logged Users.

[API] As a User, I can see a list of my previous keywords

Build API for #26

Why?

In order for the user to navigate through his previously uploaded keywords.
In order to provide an entry point for #8 & #12

Additional notes:

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI

No Gos:

No search feature [yet]
No filter feature
No multiple keywords upload/parsing [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

[BackEnd] As a User, I can see a list of my previous keywords

Why?

In order for the user to navigate through his previously uploaded keywords.
In order to provide an entry point for #8 & #12

Additional notes:

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI

No Gos:

No search feature [yet]
No filter feature
No multiple keywords upload/parsing [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

[Frontend] As a User, I get notified on the current progress of my keywords parsing

Build advanced UI for #11

Why?

In order to see newly parsed results as soon as possible.

Problems to solve:

Improve user experience and awareness of his/her current parsing status (no page refresh needed!)
Enable user to see quickly his/her newly parsed results

Who Benefits?

Logged in Users

Do not commit anything directly to develop branch.

Why

We should not be committing anything directly to develop branch such as https://github.com/malparty/google-search-ruby/commits/develop. Just a initial commit with blank readme file maybe be enough, and then everything gets merged via PR, even if it is created via Rails template

Who Benefits

The reviewer mainly.

[Chore] As a developer, I can download the screenshot of failed system tests from Github Actions

Why?

In order to ease troubleshooting when system tests passed on local but not in the CI.

Who Benefits?

Developers

[Backend] As a User, I can reset my password via the Web UI

Why?

In order to unlock the user in case he forget his password or if he wishes to update it.

Who Benefits?

Users who forget their password
Users showing interests in security concerns
Support Team (as this reduce number of manual requests)

[Backend] As a User, I can see a list of my previous keyword results (actions)

Why?

So that a user can access to search results

Today the previous #8 tasks created the DB model but missed actions in controller (keywords#show and rawviews#show)

Who Benefits?

Authenticated User

[Backend] As a User, I can query a single keyword and get its Google search results parsed

Why?

In order to see results directly and save time/effort.

Additionnal notes:

Problems to solve:

Save time for user by parsing Google search HTML content automatically
Enabling advanced results search for later features

No Gos:

No data persistence [yet]
No Csv File upload/parsing [yet]
No fancy UI as such UI would not be sustained

Who Benefits?

Logged users.

[Frontend] As a User, I can search across keywords and results

Build advanced UI for #10

Why?

Additional notes:

Problems to solve:

Enable the user quickly find a given keyword and see its results
Enable the user to perform advanced queries in order to learn insights from the data gathered

No Gos:

Search queries are not persisted [yet]

Who Benefits?

Logged Users.

[Backend] As a User, I can upload a csv file in order to run multiple keyword searches

Why?

Enable the user to save time by processing multiple keywords from 1 single user action (update a file).
Save even more time by processing keywords asynchronously.

Additional notes:

Problems to solve:

Enable user to run from 1 to 1.000 search queries from 1 single action (upload csf file)
Enable user to be aware if the selected csf file is not valid
Enable user to see the file name from a given keyword
Enable third parts to submit files via API too

No Gos:

No fancy drag & drop
No multiple files upload
No dynamic status progress update [yet]
No search/filters [yet]

Who Benefits?

Logged Users.

[Chore] -nice-to-have- Add default 0 value for keyword counts columns

Why?

Reduce logic in presenter, makes code cleaner and good for DB performances too
As identified here

WHo benefits?

Developers: Cleaner code
User: subtle perf improvement

Backlog Management: Issue format

What Happened

The issues are not following our convention https://github.com/nimblehq/git-template/tree/main/.github/ISSUE_TEMPLATE.

Expectation

The issues should match the format, something similar to junan/elixir_google_scraper#6

[Backend] As a User, I can sign up and sign in/out with a username and password

Why?

In order to provide a secure data segregation within the application.

Additional notes:

Authentication shall be available from web UI as well as API (Oauth 2 RESTful JSON API ).
UI here will be minimal. Advanced UI being done in #16

Who Benefits?

Non logged Users

[Frontend] As a User, I can sign up and sign in/out with a username and password

Build advanced UI for #5

Why?

In order to provide a secure data segregation within the application.

Additional notes:

Authentication shall be available from web UI as well as API (Oauth 2 RESTful JSON API ).
UI here will be minimal. Advanced UI being done in #16

Who Benefits?

Non logged Users

Setup Rails template

Why

Starting the project on well proven foundations with common ground for tools, language and conventions. This not only helps collaboration and maintainability, but also increase the project quality.

Template in use: Nimble Rails template

Who Benefits?

Development team

Configure unit test automations

Why

Saving time (as opposed to manual tests) and enforcing quality by:

Enable unit test execution via GitHub Actions on every push.
Enforce tests before PR validation/merge.

Who Benefits?

Development Team

[Backend] As a User, I get notified on the current progress of my keywords parsing

Why?

In order to see newly parsed results as soon as possible.

Problems to solve:

Improve user experience and awareness of his/her current parsing status (no page refresh needed!)
Enable user to see quickly his/her newly parsed results

Who Benefits?

Logged in Users

[Bug] Bootstrap icons are not visible in staging/prod env

[API] As a User, I can sign up and sign in/out with a username and password

Build API for #5

Why?

In order to provide a secure data segregation within the application.

Additional notes:

Authentication shall be available from web UI as well as API (Oauth 2 RESTful JSON API ).
UI here will be minimal. Advanced UI being done in #16

Who Benefits?

Non logged Users

Create a issue/ticket for listing user Keywords

Currently I am missing a ticket for listing uploaded keywords Like the bellow image(ignore wrong action field 😄 )

Then when user will check the show it will show the details google response result.

This listing keywords page can be used in the dashboard page or homepage. What do you think?

[API] As a User, I can see a list of my previous keyword results

Build API for #8

Why?

In order to compare different keywords & review past results.

Additional notes:

Exposes 2x API (Keyword list and search results)

Problems to solve:

User can retrieve past data thanks to persistence
User can see many keywords at once
User can see results quickly even when many (10K +) keywords are already persisted (performance & UX to address)
User can use his phone or desktop to navigate via UI
API expose the feature to enable third app to consume it

No Gos:

No search feature [yet]
No filter feature
No multiple keywords [yet]
No CSV File upload [yet]

Who Benefits?

Logged Users.

[API] As a User, I can upload a csv file in order to run multiple keyword searches

Build API for #9

Why?

Enable the user to save time by processing multiple keywords from 1 single user action (update a file).
Save even more time by processing keywords asynchronously.

Additional notes:

Problems to solve:

Enable user to run from 1 to 1.000 search queries from 1 single action (upload csf file)
Enable user to be aware if the selected csf file is not valid
Enable user to see the file name from a given keyword
Enable third parts to submit files via API too

No Gos:

No fancy drag & drop
No multiple files upload
No dynamic status progress update [yet]
No search/filters [yet]

Who Benefits?

Logged Users.

[Backend] As a user, my keywords are being parsed in background

Why?

So that I do not need to wait before getting the file-well uploaded response from the application.
So that I can close my web-browser while parsing continue.

Additional notes:

This is a direct improvement to #9
Work is only backend, there is no frontend related task.

Who benefits?

Logged users who just uploaded a CSV file.

[FrontEnd] Review layout with offcanvas

Why?

Improve UI and prepare for coming features
Use native JavaScript

Who benefits?

Developers for ease of codding future FrontEnd tasks.
User for modern UI and improved UX

As a user, I can query a single keyword and get its Google search raw response

Why?

In order for the user to manually analyse results (AdWords, links, etc...).

Additional notes:

Problems to solve:

AdWords link are included in the result (google remove adwords for bot requests!!)
Each query executes in less than 1s
Results come from https://google.com/search

No Gos

No data persistence [yet]
No query parsing [yet]
No CSV File upload/parsing [yet]
No fancy UI as such UI would not be sustained

Who Benefits?

Logged users.

[API] As a User, I can search across keywords and results

Build API for #10

Why?

Additional notes:

Problems to solve:

Enable the user quickly find a given keyword and see its results
Enable the user to perform advanced queries in order to learn insights from the data gathered

No Gos:

Search queries are not persisted [yet]

Who Benefits?

Logged Users.

[Backend] As a user, I can save/share a filtered view from its URL

Backend for #88

Why?

In order to save time once a user found it's perfect filters and wants to reapply them later on.

Who Benefits?

User who regularly uses the advanced search feature #10 / #15