OpenGraph data for Twitter no longer being fetched about opengraph HOT 10 CLOSED

@satoshi-takano haha no worries! I think we can safely close this now - I'll test out the new version tomorrow, but I can't imagine anything will go wrong!

from opengraph.

@satoshi-takano Finally got a chance to implement the WebView route and it works seamlessly.

Interestingly, YouTube links were also not returning metadata anymore and it turns out that's because YouTube requires a custom User Agent. I found facebookexternalhit/1.1 as a recommendation on Stack Overflow and running through the same process as you added for Twitter I was able to get the YouTube metadata.

Now, is that something you might want to integrate with the pod itself? Something you can optionally call fetch with?

from opengraph.

@Innocuousjoe Thanks for your additional report.
In my opinion, passing custom User Agent to fetch method as follows is better than integrating any specific custom User Agent in this library to solve those issues at the time.

OpenGraph.fetch(url: URL(string: "xxx")!, headers: ["User-Agent": "facebookexternalhit/1.1"]) { result in
  // Handle result
}

from opengraph.

@Innocuousjoe
Thanks for your report. Unfortunately, I found that a raw HTML of a tweet doesn't include og meta tags besides og:site_name.
You can look that behavior at https://en.rakko.tools/tools/9/ .

from opengraph.

Thanks for the response!

What's weird to me is if you inspect the html headers for a tweet, e.g. this one, there are definitely og properties defined besides site_name...are they being hidden somehow?

from opengraph.

@Innocuousjoe
After investigation of the issue, I found that a tweet renders other og meta tags using client side JavaScript somehow.
(So we could find all og tags only in our browser console)

I tried to execute their lines of JavaScript code for initialization using WKWebView, but it needs to add the webView to a window hierarchy. And I think it's inappropriate to include any platform specific views in OpenGraph library for it's portability.
Therefore I made the public initializer that accepts raw HTML string to parse OG tags as follows.

class ViewController: UIViewController, WKNavigationDelegate {
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        // It's needed to execute lines of JavaScript code for initialization
        let webView = WKWebView(frame: .zero)
        view.addSubview(webView)

        webView.navigationDelegate = self
        webView.load(URLRequest(url: URL(string: "https://twitter.com/JensenKarp/status/1374352501319757827")!))
    }

    func webView(_ webView: WKWebView, didFinish navigation: WKNavigation!) {
        // Some delay is needed to execute their initialization.
        DispatchQueue.main.asyncAfter(deadline: .now() + 1) {
            webView.evaluateJavaScript("document.documentElement.outerHTML.toString()") { (html: Any?, erroer: Error?) in
                let og = OpenGraph(htmlString: html as! String)
                print(og[.title] ?? "")
            }
        }
    }
}

I think this is a viable solution for now.

from opengraph.

Wow, that's kind of gross but it'll work! Thanks for digging into it!

from opengraph.

@satoshi-takano Is there some extra process for getting the new version on to Cocoapods? I'm still seeing 1.2.2 in my local repo 🤔

from opengraph.

@Innocuousjoe Oops. I forgot to push new version to CocoaPods. I just pushed it and you can now update to OpenGraph 1.3.1!

from opengraph.

Thanks!!

from opengraph.