It is a Chrome extension that takes custom screenshots and extracts text from them.
๐ Front-End Stacks:
Tesseract OCR
โ ๏ธ Prerequisites
- Recommended to have the latest version of Google Chrome
To get a local copy up and running follow these simple steps.
-
Open Git Bash and change directory
cd path
-
Clone the repo
git clone https://github.com/gitgoap/AI-Snipping-Tool_IIT-D_Hackathon.git
-
Open Google Chrome
-
Click 3 dots at top right corner.
-
Go to Extensions> Manage Extensions
-
Click Load Unpack and select the repo where you saved initially while cloning.
- Make sure to raise an issue before raising a Pull Request.
- Mention the issue number (Eg: #4) while raising a Pull Request in the description.
We often need to type out text from videos, images, or thumbnails on websites. This can be a tedious and error-prone process, especially if the text is long or has tricky stuff like website links, technical words, code examples, or math equations. This issue comes up on YouTube, where useful info is shown in videos or thumbnail images.
![](https://private-user-images.githubusercontent.com/117789470/329738245-325fe373-5215-4602-b582-1e2d3e91d96c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNDQ2NDQsIm5iZiI6MTcyNDI0NDM0NCwicGF0aCI6Ii8xMTc3ODk0NzAvMzI5NzM4MjQ1LTMyNWZlMzczLTUyMTUtNDYwMi1iNTgyLTFlMmQzZTkxZDk2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyMVQxMjQ1NDRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wODE4NmZjYmU1NzIyYTA4YjE3ZDAxN2YwMDhlMDAyNjhkNjg2OGM2YTQzZWVmOWM5MGI1YzQyMmJhMjE2ZGM3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.xydwxDJui1oiG5qzPoIuCdnChsZxVeJUQhk9s43FSN8)
In the MrBeast Riddle video, he displays a YouTube link to visit as part of the riddle, but manually typing the link is inefficient and prone to errors.
![](https://private-user-images.githubusercontent.com/117789470/329738071-1d84b1e5-6e1e-49b4-aefc-3482b74f29d6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNDQ2NDQsIm5iZiI6MTcyNDI0NDM0NCwicGF0aCI6Ii8xMTc3ODk0NzAvMzI5NzM4MDcxLTFkODRiMWU1LTZlMWUtNDliNC1hZWZjLTM0ODJiNzRmMjlkNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyMVQxMjQ1NDRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02YzljMDA5NmNmZTE3ODM1MzA1NDQyMzYzYjdhMWI1MTY2MjEzOGYxYzJlYjdmYjQ3MTM1OWMxZTIyMDJlNTAzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.GLtGDsrI3EqwapGfpAKuP5lwmK15hp4ORqqjGeBBVXY)
- This will allow the extraction of images from a particular part of the image to extract hyperlinks, code, etc.
- The extension's background script
worker.js
listens for user actions, such as clicking the extension's icon or using the context menu. - When the user initiates a screenshot, the extension captures the visible tab using
chrome.tabs.captureVisibleTab
and gets the screenshot data as a PNG image. - The extension then injects several scripts (
helper.js
,response.js
,elements.js
, andcustom-elements.min.js
) into the current tab's context. These scripts are responsible for rendering the OCR result and handling the OCR process. - The ocr-result custom element, defined in
elements.js
, is used to perform the OCR operation. This custom element utilizes thetesseract.js
library, which is a pure JavaScript port of the Tesseract OCR engine. - The ocr-result element is configured with user preferences such as language, accuracy, and other settings fetched from the Chrome extension's local storage.
- The captured screenshot data (PNG image) is passed to the ocr-result element, which then runs the Tesseract OCR engine on the image to extract the text.
- The extracted text is rendered within the ocr-result element, allowing the user to view and interact with the OCR results.
Tesseract OCR: It is an open-source optical character recognition (OCR) engine developed by Google. It is designed to extract text from images and documents without a text layer, outputting the document in various formats such as plain text, HTML, PDF, and more. Tesseract supports recognition of over 100 languages "out of the box" and is highly customizable
-
YouTube Video Screenshot: Capture text in the current video frame only on the YouTube website.
-
Integration of Gemini API key for:
Webpage Summary: Summarize text from a screenshot capture of any webpage in the user-chosen language (MultiModal).