interpreting apple watch timestamps about qs_ledger HOT 5 CLOSED

@francescolosterzo If you check the apple_health_data_processor.ipynb, you'll see this line:

convert_tz = lambda x: x.to_pydatetime().replace(tzinfo=pytz.utc).astimezone(pytz.timezone('Asia/Shanghai'))

There is where we set the tz and then subsequent lines parse it to correct timezones. Check that to see if it works for your needs.

To be honest, that code is one of the older parts so I'm not sure if it's up to data to any changes Apple made in their export files and timestamps.

from qs_ledger.

@markwk I am continuing this since now I see other things I would like to ask you and they still fall under the title of this issue.

1. I see that heart rate points are not equally spaced in time, in the night they are less frequent that during the day. I guess Apple understands to the activity level and adapts the sampling time (e.g. during the night a point every few minutes is ok). Or there might be anything else going on here?

2. following your example, I am using the startDate as a reference and I see that data are sometimes not ordered in time, i.e. when I plot my heart rate vs time with matplotlib.pyplot.plot I see the line going back and forth every now and then. Do you have any idea why this is happening? Is it safe to sort the rows of the dataframe by startDate? Or will it mess thing up?

from qs_ledger.

@francescolosterzo you didn't share your code so I'm only guessing but it sounds like you didn't convert it to datetime or process it like the other examples.

I went ahead and added a quick example in code showing and plotting HR now in that notebook. Still a small issue with x-axis labeling or ticks but you shouuld have the idea now.

from qs_ledger.

Hi @markwk
you are right, here is what I do:

1. load the data

df = pd.read_csv('HeartRate.csv', skipinitialspace=True)

2. add a timezone column based on when I went to the US

cList = [
    (datetime.datetime(2019,7,24), datetime.datetime(2019,8,3,12), 'Europe/Zurich'),
    (datetime.datetime(2019,8,3,12,1), datetime.datetime(2019,8,13,12), 'America/Detroit'),
    (datetime.datetime(2019,8,13,12,1), datetime.datetime(2019,9,26), 'Europe/Zurich'),
]

dfList = []
for entry in chunkList:
        startDate, stopDate, tz = entry
        
        this_df = df[ (pd.to_datetime(df['startDate']) >= startDate) & (pd.to_datetime(df['startDate']) <= stopDate) ]
        
        this_df = this_df.assign(timezone = tz)
        
        dfList.append(this_df)
    
    df_out = pd.concat(dfList, ignore_index=True)

3. assign the timezone and get the timezone-naive timestamp (in order to have things smoother)

df['timestamp'] = df.groupby('timezone')['startDate'].apply(lambda x: x.dt.tz_localize('UTC').dt.tz_convert(x.name).dt.tz_localize(None))

This might cause problems on the days in which I actually changed timezone, but this is not the problem right now.

The real problem I was referring to is that if I chart the HR vs timestamp in a single night, I get points flipped in time every now and then. Here is an example:

df[ (df.startDate >= datetime.datetime(2019,7,25,1)) & (df.startDate <= datetime.datetime(2019,7,25,2)) ][['startDate', 'timestamp', 'value']].values

array([ ...
       [Timestamp('2019-07-25 01:19:25'), 1564017565, 69.0],
       [Timestamp('2019-07-25 01:26:50'), 1564018010, 69.0],
       [Timestamp('2019-07-25 01:31:05'), 1564018265, 72.4528],
       [Timestamp('2019-07-25 01:30:15'), 1564018215, 66.0],
       [Timestamp('2019-07-25 01:34:12'), 1564018452, 68.0],
       [Timestamp('2019-07-25 01:43:51'), 1564019031, 68.0],
       ...], dtype=object)

As you can see above the 3rd and the 4th entries are flipped in time. This happens a few times in each night.

So I was wondering if you have any idea about why this happens?

from qs_ledger.

@francescolosterzo Sorry. I haven't seen this error in my data and not sure why it's appearing for you. You should be able to do a new sort on the data to correct it without any issues.

from qs_ledger.