@ydkhatri going over the code to add more test coverage it looks like:

scripts/UnifiedLogReader.py defines DecompressTraceV3Log but this function is not used
this means that the DecompressTraceV3 function in UnifiedLog/Lib.py is also not used.

Could you confirm if my assessment is correct? And if there is additional context I'm not aware of, why these functions should remain in the code base. Otherwise I'll prep a PR to remove them.

VirtualFileSystem.get_virtual_file() will return a VirtualFile for a path to a directory

Unclear from the code and docstring what the intended behavior is

Dsc.ReadFmtStringAndEntriesFromVirtualOffset returns a string on error and a tuple on success

https://github.com/ydkhatri/UnifiedLogReader/blob/master/UnifiedLog/Lib.py#L1442

Please clarify what the return value should be. Or should this method raise on error instead?

pip install unifiedlog does not install the dependencies. Need to add install-requires to setup.py.

From https://github.com/ydkhatri/UnifiedLogReader/blob/master/UnifiedLog/Lib.py#L300

           elif item_id == 0x6102: # uuid
                self.system_boot_uuid = UUID(bytes=buffer[pos:pos+16])
                self.boot_uuid_ts_list = GetBootUuidTimeSyncList(self.ts_list, self.system_boot_uuid)
                if not self.boot_uuid_ts_list:
                    raise ValueError('Could not get Timesync for boot uuid! Cannot parse file..')

If self.boot_uuid_ts_list is an empty list the tracev3 file fails to parse.

I've change this to if self.boot_uuid_ts_list is None:, since GetBootUuidTimeSyncList returns None if it cannot find a Timesync corresponding with the boot identifier.

Let me know if this matched the intended behavior

Hi,

Iam trying to use this script for some IOS Unified Logs (dont know if it would work?)
But when using i get two erros:

Traceback (most recent call last):
File "UnifiedLogReader.py", line 500, in
if not Main():
File "UnifiedLogReader.py", line 456, in Main
logger.addHandler(log_console_handler)
NameError: global name 'logger' is not defined

File "UnifiedLogReader.py", line 498, in
if not Main():
File "UnifiedLogReader.py", line 462, in Main
unified_log_reader = UnifiedLogReader()
File "UnifiedLogReader.py", line 286, in init
self._vfs = virtual_file_system.VirtualFileSystem(
NameError: global name 'virtual_file_system' is not defined

Do you have an idea how to fix it?

Hello,

First, thank you for the great work ! It's amazing, it should help us a lot to parse logs in Windows.

After some try, most of the logs are extracted, but we still have some errors. I didn't open a ticket for each one, but I can if needed. Are they known ? Each one are repeated a bunch of times at different frequency, with different address / offset / values.

ERROR - missing data for log @ 0x4F0E8

Traceback (most recent call last):
  File "UnifiedLogReader-master/UnifiedLog/tracev3_file.py", line 674, in RecreateMsgFromFmtStringAndData
    msg += self._ReadNtSid(raw_data)
  File "UnifiedLogReader-master/UnifiedLog/data_format.py", line 76, in _ReadNtSid
    sub_authorities = struct.unpack('<{}I'.format(num_sub_auth), data[8:8*num_sub_auth])
error: unpack requires a string argument of length 4

UnifiedLogReader-master/UnifiedLog/tracev3_file.py:662: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if unk == 'D': # 0x44
UnifiedLogReader-master/UnifiedLog/tracev3_file.py:667: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif unk == '#': #0x23

ERROR - Unknown value for mbr_details found 0xa3 in log @ 0x16C18E8

ERROR - Data Reference not found for unique_ref=0x1000003535BF4D626 ct=3656559941158!

ERROR - Failed to open file system_logs.logarchive/94/D581CA04CA3AA78087D09B74E2AB67
Traceback (most recent call last):
  File "UnifiedLogReader-master/UnifiedLog/virtual_file.py", line 31, in open
IOError: [Errno 24] Too many open files: 'system_logs.logarchive/94/D581CA04CA3AA78087D09B74E2AB67'

ERROR - Did not read data of type 3, t1=location, t2=_CLDaemonStatusStateTrackerState, length=0x28 from log @ 0x5D18 ct=25128227678
ERROR - Did not read data of type 3, t1=location, t2=_CLLocationManagerStateTrackerState, length=0x48 from log @ 0x61560 ct=46065225628
ERROR - Did not read data of type 2, t1=AppStoreDaemon, t2=LogPushConnection, length=0x2E from log @ 0x7C1048 ct=15495190668348

ERROR - Invalid bounds 0x8F1500 for bc166f11-dd33-32af-af67-de0b2c985aa8

INFO - Unknown custom data object type '{public, location:_CLLocationManagerStateTrackerState}' data size=0x48 in log @ 0x1CE798

when self.uuid_entries is empty and GetUuidEntryFromUuid is called it raises UnboundLocalError: local variable 'b' referenced before assignment

What is the expected behavior? To raise KeyError or return None, or equiv?

To do:

• Have GetUuidEntryFromUuid return None
• remove GetUuidEntryFromUuid
  • pending https://github.com/joachimmetz/UnifiedLogReader/tree/clean1

VirtualFile.get_file_size() returns ValueError if file does not exists?

======================================================================
ERROR: testGetFileSize (virtual_file.VirtualFileTest)
Tests the get_file_size function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "UnifiedLogReader/tests/virtual_file.py", line 24, in testGetFileSize
    file_size = file_entry.get_file_size()
  File "UnifiedLogReader/UnifiedLog/virtual_file.py", line 45, in get_file_size
    raise ValueError('File pointer was invalid. File must be opened before calling get_file_size()')
ValueError: File pointer was invalid. File must be opened before calling get_file_size()

@ydkhatri is this the intended behavior? IMHO an IOError('No such file') or equiv would be more clear.

The exception in Dsc.Parse at https://github.com/ydkhatri/UnifiedLogReader/blob/master/UnifiedLog/Lib.py#L1504 does not sets the result to False hence the method returns True even if the code between the try except fails.

Also the Except is very broad and will catch things like ImportErrors, could you reduce the except to IOError, OSError, and struct related exceptions?

Note that the custom specifier can be larger than 64 characters e.g. %{public,signpost.telemetry:number1,name=SOSSignpostNameSOSCCEnsurePeerRegistration}d

64 is a limit that is used in