[Android] Json null object reference about jraw HOT 16 CLOSED

Desktop Java yields the desired result:

RedditClient reddit = new RedditClient("MY-JRAW-REDDIT-ANDROID-APP");
LoggedInAccount loggedInAccount = reddit.login(Credentials.standard("thatJavaNerd", "super_secret_password"));

System.out.println(loggedInAccount.getFullName());
System.out.println(loggedInAccount.getCommentKarma());
System.out.println(loggedInAccount.getLinkKarma());

thatJavaNerd
416
248

I'll try it out on Android and see if it's any different.

from jraw.

I had been doing some research on this. Any malformed JSON will cause the Exception

It can be avoided here:
https://github.com/thatJavaNerd/JRAW/blob/master/src/main/java/net/dean/jraw/RedditClient.java#L241

You need to add a check for the malformed JSON. The problem is that the cookies that reddit asks the client to set don't get set. All that gets set is the modhash here:
https://github.com/thatJavaNerd/JRAW/blob/master/src/main/java/net/dean/jraw/RedditClient.java#L218

I was poking around in RestClient trying to find a way to make the cookies get set, particularly reddit_session. As a workaround, I've added this as line 219 in RedditClient.java
defaultHeaders.put("Cookie", cookieJar.getCookies().toString());

It just gets the cookie jar as a string and sends that as the Cookie header. Unfortunately, since reddit now uses cloudfront as their CDN, cloudfront also sets their own cookie, which the HTTP client in JRAW DOES set, so the client will wind up sending two Cookie headers.

I'll keep looking around in RestClient to see what I found, but analyzing other reddit clients on Android, sending the X-Modhash header isn't enough.

To sum things up:

• The log in POST request gets successfully sent by JRAW when logging in
• The POST response sends a few set-cookie headers that get ignored by the JRAW HTTP client
• JRAW then tries to pull the logged in account info from from the me.json endoing WITHOUT sending the reddit_session cookie (only the X-Modhash header gets sent)
  me.json returns an empty json object {}
• JRAW tries to parse the empty JSON object without doing a check, when trying to parse get info from the data object, the null pointer exception is thrown (since there is no data object)

from jraw.

Let's get a few things out of the way.

1. JRAW does not ignore cookies (or it should not be). See here.
2. JRAW also checks for malformed JSON. When ObjectMapper.readTree(String) is called in RestResponse's constructor, it throws a JsonProcessingException when it encounters JSON with invalid syntax.

Since dalvik is complaining about calling jsonNode.has(String) when jsonNode is null, that means one of three things:

1. JrawUtils.typeComparison(MediaType, MediaType) is not functioning as it should
2. The returned Content-Type was not application/json
3. The response body was empty

(see here for reference)

We need to know the exact response that was returned in order to diagnose this.

Here is an example curl request:

curl -i --data 'user=<omitted>&passwd=<omitted>&api_type=json' https://ssl.reddit.com/api/login
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Mon, 05 Jan 2015 22:28:31 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 203
Connection: keep-alive
Set-Cookie: __cfduid=dc22cba123de9bb0085d4d4b8d3dd2e721420496911; expires=Tue, 05-Jan-16 22:28:31 GMT; path=/; domain=.reddit.com; HttpOnly
x-ua-compatible: IE=edge
cache-control: no-cache
pragma: no-cache
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
set-cookie: secure_session=; Domain=reddit.com; Max-Age=-1420496911; Path=/; expires=Thu, 01-Jan-1970 00:00:01 GMT; HttpOnly
set-cookie: reddit_session=<omitted>; Domain=reddit.com; Path=/; HttpOnly
X-Moose: majestic
cache-control: no-cache
CF-RAY: 1a431c001da10850-IAD

{"json": {"errors": [], "data": {"need_https": false, "modhash": "<omitted>", "cookie": "<omitted>"}}}

As you can see, the Content-Type was application/json and the body was not empty, and JrawUtils.typeComparison(MediaType, MediaType) has been tested (albeit not very thoroughly) here and used throughout the project (see here) without failing other tests.

from jraw.

Analyzing the traffic between JRAW and the servers I've been seeing the following. Note that I've removed headers related to caching, etc.

JRAW Authentication POST request

POST /api/login HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: ssl.reddit.com
Connection: Keep-Alive
Accept-Encoding: gzip
passwd=PASSWORD&user=USERNAME&api_type=json

Reddit server POST response (note the two set-cookie headers)

HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Mon, 05 Jan 2015 23:52:58 GMT
Content-Type: application/json; charset=UTF-8
set-cookie: reddit_session=DATE_AND_SESSION_TOKEN_HERE; Domain=reddit.com; Path=/; secure; HttpOnly
set-cookie: secure_session=1; Domain=reddit.com; Path=/; HttpOnly

{"json": {"errors": [], "data": {"need_https": true, "hsts_redir": "https://reddit.com/modify_hsts_grant?dest=", "modhash": "MODHASH_HERE", "cookie": "DATE_AND_SESSION_TOKEN_HERE"}}}

JRAW me.json GET request (note the lack of reddit_session in the Cookie header)

GET /api/me.json HTTP/1.1
X-Modhash: MODHASH_HERE
Host: www.reddit.com
Connection: Keep-Alive
Accept-Encoding: gzip
Cookie: __cfduid=CLOUDFLARE_ID_HERE

me.json GET response, note the empty JSON object

HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Mon, 05 Jan 2015 23:53:01 GMT
Content-Type: application/json; charset=UTF-8

{}

When the lines of code I mentioned before try to parse that (in the me() method of the client), this exception is thrown:
Attempt to invoke virtual method 'boolean org.codehaus.jackson.JsonNode.has(java.lang.String)' on a null object reference

This happens when JRAW tries to parse the data object from the empty JSON response.

from jraw.

Reddit responds with {} when there is no logged in user. However, there still is content in the response body, meaning that this would not cause the bug described.

OkHttp takes care of all cookies and the setting of the Cookie header:

OkHttp may add headers that are absent from the original request, including Content-Length, Transfer-Encoding, User-Agent, Host, Connection, and Content-Type. It will add an Accept-Encoding header for transparent response compression unless the header is already present. If you’ve got cookies, OkHttp will add Cookie headers for them.

source

In all unit tests, OkHttp correctly handles cookies and as a result, every login request with valid credentials has succeeded.

How are you analyzing this traffic? JRAW's HttpLogger logs headers that occur before OkHttp rewrites the request for sending, and any other means of analyzing the traffic would have a high probability to being inaccurate, as it would have to mirror JRAW's sending of HTTP requests and handling of their responses verbatim.

from jraw.

As far as I could tell by reading the code and stepping through it with a debugger, it seems to be setting cookies correctly (it DOES set __cfduid)

I'm analyzing the traffic using an HTTP proxy. I can also analayze using tpacketcapture and then dump that into wireshark but that gets old quickly, so I'm using burp with a system-wide proxy. I installed the CA on my tablet so I can decrypt the TLS traffic. I also ran the unit tests and they DO pass, but it's when I tried to use JRAW in another project that I noticed the login failing.

from jraw.

Do you have the code for the other project?

from jraw.

String user = "USERNAME";
String pass = "PASSWORD";
Credentials credentials = Credentials.standard(user, pass);
RedditClient mRedditClient = new RedditClient("CLIENT_USER_AGENT");
mRedditClient.setRequestLoggingEnabled(true);
LoggedInAccount account = mRedditClient.login(credentials);

Here is the logcat output

W/System.err﹕ [AsyncTask #1] INFO JRAW - SPDY/3.1 200 OK
W/System.err﹕ [AsyncTask #1] INFO JRAW -     content-type: application/json; charset=UTF-8
W/System.err﹕ [AsyncTask #1] INFO JRAW -     response-body: {"json": {"errors": [], "data": {"need_https": true, "hsts_redir": "https://reddit.com/modify_hsts_grant?dest=", "modhash": "MODHASH_HERE", "cookie": "REDDIT_SESSION_COOKIE"}}}
W/System.err﹕ [AsyncTask #1] INFO JRAW - Slept for 0.025008 seconds
W/System.err﹕ [AsyncTask #1] INFO JRAW -
W/System.err﹕ [AsyncTask #1] INFO JRAW - GET https://www.reddit.com/api/me.json
W/System.err﹕ [AsyncTask #1] INFO JRAW -     headers: {X-Modhash=MODHASH_HERE,
W/System.err﹕ [AsyncTask #1] INFO JRAW -               User-Agent=CLIENT_USER_AGENT}
W/System.err﹕ [AsyncTask #1] INFO JRAW - SPDY/3.1 200 OK
W/System.err﹕ [AsyncTask #1] INFO JRAW -     content-type: application/json; charset=UTF-8
W/System.err﹕ [AsyncTask #1] INFO JRAW -     response-body: {}
E/LinkDownloadTask﹕ Attempt to invoke virtual method 'boolean org.codehaus.jackson.JsonNode.has(java.lang.String)' on a null object reference
W/System.err﹕ java.lang.NullPointerException: Attempt to invoke virtual method 'boolean org.codehaus.jackson.JsonNode.has(java.lang.String)' on a null object reference

from jraw.

That's strange. It seems that on Android, the CookieManager's CookieStore is an InMemoryCookieStore is used, while on the desktop, a CookieStoreImpl is used. Both classes are package protected. I'm thinking this is an Android specific issue.

from jraw.

Yeah it sure is looking that way, I just set up a plain Java project with the exact same code I pasted above and I'm getting a successful login. I'll keep poking around to see what else I find.

from jraw.

I should have probably mentioned that the tests worked fine and a JRE version of my code (Thread not Async ofc.) worked fine. This is probably the most active and the most well written Java RAW compared to all the others I have found and just wanted to try my hand at a Reddit client using the new Lollipop features.

Will anything be done about this issue or should I try a fix myself?

from jraw.

I'm going to try with OAuth as well, maybe I will have some luck there. As soon as I get some time.

from jraw.

I think I found the problem. java.net.CookieStoreImpl stores the cookies in a Map<URI, List<HttpCookie>> (source). After sending the initial login request, the map looks like this (in JSON for simplicity):

{
    "http://ssl.reddit.com": {
        "__cfduid": "<omitted>",
        "secure_session": "<omitted>",
        "reddit_session": "<omitted>"
    }
}

So now when we submit a request to https://www.reddit.com/api/me.json, the CookieStore doesn't have any cookies for www.reddit.com, only ssl.reddit.com and therefore no cookies are sent.

java.net.InMemoryCookieStore (which does not seem to be included in the Android version of the JVM) stores the cookies in an ArrayList for the entire domain, so that when the cookies are retrieved for http://www.reddit.com/api/me.json, they are all sent, even though the login cookies were for http://ssl.reddit.com.

A solution would be to implement our own CookieHandler since InMemoryCookieStore is copyrighted:

/*
 * Copyright (c) 2005, 2008, Oracle and/or its affiliates. All rights reserved.
 * ORACLE PROPRIETARY/CONFIDENTIAL. Use is subject to license terms.
 */

And then we could just do

CookieManager manager = new CookieManager(new OurNewCookieStore(), CookiePolicy.ACCEPT_ALL);
http.setCookieHandler(manager);

from jraw.

I tried to come up with a solution, but with no change. For the heck of it I included InMemomoryCookieStore.java in the Android project and made my RedditClient subclass use it. Even then, the bug persists.

In OkHttp's HttpEngine class, the Cookie header is added with all correct values. If I do

curl -H 'Cookie: <OkHttp's generated value from the CookieStore>' https://www.reddit.com/api/me.json

then everything works fine.

It must be something Android-specific, but not necessarily the CookieStore implementation.

from jraw.

In any case, since cookie authentication has been deprecated (see #47), this issue is no longer a major problem. With the latest commit, this minimal code will successfully authenticate a user on Android:

public class MainActivity extends Activity {
    private static final String TAG = "myapp";
    private static final String REDIRECT_URL = // ...

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        RedditClient reddit = new RedditClient(UserAgent.of(...));
        final OAuthHelper helper = reddit.getOAuthHelper();
        final Credentials credentials = Credentials.installedApp(...);
        WebView webView = (WebView) findViewById(R.id.webview);

        webView.loadUrl(helper.getAuthorizationUrl(credentials.getClientId(), redirectUrl, ableToRefreshToken, scopes).toExternalForm());
        webView.setWebViewClient(new WebViewClient() {
            @Override
            public void onPageStarted(WebView view, String url, Bitmap favicon) {
                if (url.contains("code=")) {
                    new UserChallengeTask(helper, credentials).execute(url);
                }
            }
        });
    }

    private static class UserChallengeTask extends AsyncTask<String, Void, OAuthData> {
        private OAuthHelper helper;
        private Credentials creds;
        public UserChallengeTask(OAuthHelper helper, Credentials creds) {
            this.helper = helper;
            this.creds = creds;
        }

        @Override
        protected OAuthData doInBackground(String... params) {
            try {
                return helper.onUserChallenge(params[0], REDIRECT_URL, creds);
            } catch (NetworkException | OAuthException e) {
                throw new RuntimeException("", e);
            }
        }

        @Override
        protected void onPostExecute(OAuthData oAuthData) {
            Log.i(TAG, oAuthData.toString());
        }
    }
}

from jraw.

Thank you very much kind sir. Excellent job!

from jraw.