natural-language-youtube-search's Introduction

Natural Language YouTube Search

Use OpenAI's CLIP neural network to search inside YouTube videos. You can try it by running the notebook on Google Colab.

New

Integrated to Huggingface Spaces with Gradio. See demo:

How it works

Download the YouTube video
Extract every N-th frame
Encode all frames using CLIP
Encode a natural language search query using CLIP
Find the images that best match the search query

For more details see the notebook.

Examples

Here are some example searches from this YouTube video of a car driving around San Francisco.

"A fire truck"

"Road works"

"People crossing the street"

"The Embarcadero"

"Waiting at the red light"

"Green bike lane"

"A street with tram tracks"

"The Transamerica Pyramid"

Natural language search on Unsplah

You can also try my other project to search from 2M photos on Unsplash using natural language queries:

Natural Language Image Search

natural-language-youtube-search's Issues

Saving the timeline in a new csv, xlsx, or even a list.

Hello.
Thanks for your effort.
Is there an option to save the time result to any xlsx, CSV, or any list, pandas types?.
What are the available options to save the results?
Many thanks in advance!

Download video from youtube link in the HF Space

Proposing a new functionality in the HF Space,

Given a YT link
Download the video
Use text to search in the video

Hi.
Would it be possible to show in which second each of the three frames was found?
Don't know anything about programming but think it would be useful for the application.
Congratulations for the work.

Error in YouTube(video_url).streams.... function

I get an error when going through this Colab notebook in the cell where the "streams =" is executed. I verified that the youtube video exists.

HTTPError Traceback (most recent call last)

in ()
2
3 # Choose a video stream with resolution of 360p
----> 4 streams = YouTube(video_url).streams.filter(adaptive=True, subtype="mp4", resolution="360p", only_video=True)
5
6 # Check if there is a valid stream

12 frames

/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648 def http_error_default(self, req, fp, code, msg, hdrs):
--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)
650
651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 404: Not Found

Strange CLIP results

Hi, first of all many thanks for this wonderful script :)
I've trying some searchs, and I found some strange results. Probably is how CLIP words, but not sure.

If I search "CAR" (there are a lot of cars in the video), and if I look at the value of the frame with the best similarity, I get, e.g. 26.65
Then I search something stupid like "sdfsdflksdfj", and I check at the same value...I was expecting to get a near-to-zero value, but instead I get, e.g. 21.55.
Is this a bug? Or is the way CLIP works? Is there a way to detect how good the prediction is?
Many thanks!

Recommend Projects