URL and title mismatch about araa-search HOT 8 OPEN

You should probably just revert back to before these changes, then just do lots of testing before you push them again.
Right now, if I search something, I just get Internal Server Error.

from araa-search.

Please revert these changes from the main branch, and push them off to a seperate branch in the mean time until they are free of these bugs. You shouldn't just leave the project broken like this for an extended period of time.

from araa-search.

https://araa.extravi.dev/search?q=chatgpt&t=text
https://araa.extravi.dev/search?q=telegram&t=text

from araa-search.

It does that for some results, but not all. I was trying to update the proxy to scrape from MjjYud so I could get more results like video urls, etc. from Google Search and later add the image base64 next to the results.

from araa-search.

yeah i need to also idk how to fix this

from araa-search.

Here's a solution that stops the mismatched links issue:

    # loop through each div
    for div in result_divs:
        # retrieve links
        if not div.find("div", class_="cUnQKe"):
            link = div.find("a")
            if link is not None:
                link = link.get("href")
            else:
                continue

            # retrieve title
            title = div.find("h3")
            if title is not None:
                title = title.text.strip()
            else:
                continue

            hrefs.append(link)
            titles.append(title)

            # retrieve description
            desc = div.find("div", {"class": "VwiC3b"})
            if desc is not None:
                descriptions.append(desc.text.strip())
            else:
                descriptions.append("No description available.")

This only appends the link and title if neither are empty.

The issue is caused by there being no else case for if the title or link is empty.
So the title may be there, and so it gets appended, but the link isn't, so that doesn't get appended.

However, this doesn't fix some other issues, like when it doesn't return any results. For example, if I search 'macbeth', only the wikipedia snippet is shown, which shows that there are results, but the current scraper isn't picking them up.

from araa-search.

Also, this doesn't solve the issue where some links just say 'Images', rather than their actual title.

from araa-search.

i went back to the older version for now

aa96129

from araa-search.