Cppparser preprocessor do not expand macros with arguments correctly,about panda3d/panda3d

Comments (94)

mingodad commented on May 30, 2024 1

I've just tested with an amalgamation of Lua-5.4.6 (https://github.com/mingodad/lua-wasm-playground/blob/main/am-lua-5.4.6.c) and it handled it fine, we are making good progress !

from panda3d.

rdb commented on May 30, 2024 1

Hahaha! It's correct, but it's a little bit dumb. Pushed a change that makes it less unwieldy.

from panda3d.

mingodad commented on May 30, 2024 1

Thank you again !
It's really better.

I think that overall you've nailed the preprocessor very well !

from panda3d.

mingodad commented on May 30, 2024 1

Really impressive less than 1/10 of the time now (that's a nice hanging fruit) !
Before

/usr/bin/time dist/Release/GNU-Linux/cppparser-nb  -E -D__PANDA_CC__  -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include sqlite3.c > /dev/null
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Finished parsing.
13.59user 0.03system 0:13.64elapsed 99%CPU (0avgtext+0avgdata 88080maxresident)k
0inputs+0outputs (0major+20916minor)pagefaults 0swaps

After:

/usr/bin/time dist/Release/GNU-Linux/cppparser-nb  -E -D__PANDA_CC__ -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include sqlite3.c > /dev/null
Predefining __PANDA_CC__ as 
Predefining wchar_t as pccwchar_t
Finished parsing.
1.11user 0.04system 0:01.16elapsed 99%CPU (0avgtext+0avgdata 87104maxresident)k
0inputs+0outputs (0major+20682minor)pagefaults 0swaps

from panda3d.

mingodad commented on May 30, 2024 1

Sorry ! I've applied it wrongly before.
It does work !

from panda3d.

rdb commented on May 30, 2024

Thanks for reporting, I checked in a fix.

I also took the opportunity to improve the error reporting when an error occurs within a macro, since that was previously quite tricky to diagnose without recompiling cppparser and sifting through the debug output:

from panda3d.

mingodad commented on May 30, 2024

Here is another:

#define __SYSMACROS_DECL_TEMPL(rtype, name, proto)			     \
  extern rtype gnu_dev_##name proto __THROW __attribute_const__;

#define __SYSMACROS_DECLARE_MAJOR(DECL_TEMPL)			\
  DECL_TEMPL(unsigned int, major, (__dev_t __dev))

__SYSMACROS_DECLARE_MAJOR (__SYSMACROS_DECL_TEMPL)

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading test.c
  test.c:76:50: warning: Not enough arguments for manifest __SYSMACROS_DECL_TEMPL
  __SYSMACROS_DECLARE_MAJOR (__SYSMACROS_DECL_TEMPL)
                                                   ^
    test.c:76:1: error: unknown type 'gnu_dev_'
    __SYSMACROS_DECLARE_MAJOR (__SYSMACROS_DECL_TEMPL)
    ^
Expansion of __SYSMACROS_DECLARE_MAJOR:
 -> extern gnu_dev_ __THROW __attribute_const__; (unsigned int, major, (__dev_t __dev))
    test.c:76:1: error: syntax error
    __SYSMACROS_DECLARE_MAJOR (__SYSMACROS_DECL_TEMPL)
    ^
Expansion of __SYSMACROS_DECLARE_MAJOR:
 -> extern gnu_dev_ __THROW __attribute_const__; (unsigned int, major, (__dev_t __dev))
Error in parsing.

Output from gcc:

gcc -E -P test.c
extern unsigned int gnu_dev_major (__dev_t __dev) __THROW __attribute_const__;

Output from ppstep:

./ppstep test.c 
Preprocessing test.c:0:1.
pp (started)> s
[test.c:76:1]: __SYSMACROS_DECLARE_MAJOR ( __SYSMACROS_DECL_TEMPL )
pp (called)> s
[test.c:76:1]: __SYSMACROS_DECL_TEMPL ( unsigned int , major , ( __dev_t __dev ) )
pp (expanded)> s
[test.c:76:1]: __SYSMACROS_DECL_TEMPL ( unsigned int , major , ( __dev_t __dev ) )
pp (called)> s
[test.c:76:1]: extern unsigned int gnu_dev_major ( __dev_t __dev ) __THROW __attribute_const__ ;
pp (expanded)> s
[test.c:76:1]: extern unsigned int gnu_dev_major ( __dev_t __dev ) __THROW __attribute_const__ ;
pp (rescanned)> s
[test.c:76:1]: extern unsigned int gnu_dev_major ( __dev_t __dev ) __THROW __attribute_const__ ;
pp (rescanned)> s
[test.c:76:51]: extern unsigned int gnu_dev_major ( __dev_t __dev ) __THROW __attribute_const__ ; 

pp (lexed)> s
[test.c:79:1]: extern unsigned int gnu_dev_major ( __dev_t __dev ) __THROW __attribute_const__ ; 

pp (complete)> s

from panda3d.

rdb commented on May 30, 2024

Also fixed that one - macros passed as arguments to other macros were being expanded erroneously if they were function macros used without parentheses.

from panda3d.

mingodad commented on May 30, 2024

Thank you !
Now there is several messages that doesn't seem to be correct with this example:

#define _SC(a) a
#define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")

const char *cr = SQUIRREL_COPYRIGHT;

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading test.c
  test.c:79:101: warning: Too many arguments for manifest _SC
  #define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")
                                                                                                      ^
  test.c:79:9: warning: missing terminating " character
  #define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  test.c:79:9: warning: missing terminating " character
  #define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  test.c:79:9: warning: missing terminating " character
  #define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Finished parsing.
char const *cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte";

gcc output:

gcc -E -P test.c
const char *cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte";

ppstep output :

ppstep test.c 
Preprocessing test.c:0:1.
pp (started)> s
[test.c:81:1]: const
pp (lexed)> s
[test.c:81:7]: const char
pp (lexed)> s
[test.c:81:12]: const char *
pp (lexed)> s
[test.c:81:13]: const char * cr
pp (lexed)> s
[test.c:81:16]: const char * cr =
pp (lexed)> s
[test.c:81:18]: const char * cr = SQUIRREL_COPYRIGHT
pp (called)> s
[test.c:81:18]: const char * cr = _SC ( "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" )
pp (expanded)> s
[test.c:81:18]: const char * cr = _SC ( "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" )
pp (called)> s
[test.c:81:18]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte"
pp (expanded)> s
[test.c:81:18]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte"
pp (rescanned)> s
[test.c:81:18]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte"
pp (rescanned)> s
[test.c:81:36]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" ;
pp (lexed)> s
[test.c:81:37]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" ; 

pp (lexed)> s
[test.c:84:1]: const char * cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" ; 

pp (complete)> s

from panda3d.

mingodad commented on May 30, 2024

After posting the last message and looking at the last line of cppparser output I noticed that it also switched the non macro part const char *cr to char const *cr.

from panda3d.

mingodad commented on May 30, 2024

Doing my experiments this patch seems to fix the string handling problem:

--- <unnamed>
+++ <unnamed>
@@ -2510,6 +2510,26 @@
   p = q + result.size();
 }
 
+static bool checkQuoted(const string &expr, size_t &p)
+{
+  if (expr[p] == '"' || expr[p] == '\'') {
+    // Quoted string or character.
+    int quote_mark = expr[p];
+    p++;
+    while (p < expr.size() && expr[p] != quote_mark && expr[p] != '\n') {
+      if (expr[p] == '\\') {
+        p++;
+      }
+      if (p < expr.size()) {
+        p++;
+      }
+    }
+    p++;
+    return true;
+  }
+  return false;
+}
+
 /**
  *
  */
@@ -2529,20 +2549,8 @@
     }
     args.push_back(expr.substr(q, p - q));
 
-  } else if (expr[p] == '"' || expr[p] == '\'') {
+  } else if(checkQuoted(expr, p)) {
     // Quoted string or character.
-    int quote_mark = expr[p];
-    p++;
-    while (p < expr.size() && expr[p] != quote_mark && expr[p] != '\n') {
-      if (expr[p] == '\\') {
-        p++;
-      }
-      if (p < expr.size()) {
-        p++;
-      }
-    }
-    p++;
-
   } else {
     // Skip paren.
     p++;
@@ -2552,6 +2560,7 @@
     }
     size_t q = p;
     while (p < expr.size() && expr[p] != ')') {
+      checkQuoted(expr, p);
       if (expr[p] == ',') {
         args.push_back(trim_blanks(expr.substr(q, p - q)));
         q = p+1;

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/test.c
  0. START_CPP
  #define _SC(a) a
  #define SQUIRREL_COPYRIGHT	_SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")
Pushing to string ""Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte")"
  0. START_CONST_EXPR
  1. SIMPLE_STRING Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte
  2. ')'
  1. KW_CONST
  2. KW_CHAR
  3. '*'
  4. IDENTIFIER cr
  5. '='
  Pushing to expansion " _SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte") "
    Expanding SQUIRREL_COPYRIGHT to  _SC("Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte") 
    Pushing to expansion " "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" "
      Expanding _SC to  "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte" 
      6. SIMPLE_STRING Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte
      End of input stream, restoring to previous input
    End of input stream, restoring to previous input
  End of input stream, restoring to previous input
7. ';'
8. token 0

Finished parsing.
char const *cr = "Copyright (C) 2003-2017 Alberto Demichelis, Domingo Alvarez Duarte";

from panda3d.

rdb commented on May 30, 2024

I already checked in a fix that also handles the case of nested parentheses correctly.

char const and const char are equivalent, no sense separating out that case in the parser.

from panda3d.

mingodad commented on May 30, 2024

Thank you again !
Now I'm going further on my tests and now stop at this:

#define SQ_OBJECT_RAWINIT() { _unVal.raw = 0; }

#define _REF_TYPE_DECL(type,_class,sym) \
	SQObjectPtr(_class * x) \
	{ \
		SQ_OBJECT_RAWINIT() \
	}

_REF_TYPE_DECL(OT_STRING_UTF8,SQStringUtf8,StrUtf8)

ccpparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/test.c
    /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/test.c:91:1: error: syntax error
    _REF_TYPE_DECL(OT_STRING_UTF8,SQStringUtf8,StrUtf8)
    ^
Expansion of _REF_TYPE_DECL:
 -> SQObjectPtr( SQStringUtf8 * x) { SQ_OBJECT_RAWINIT() }
Error in parsing.

gcc output:

gcc -E -P test.c
SQObjectPtr(SQStringUtf8 * x) { { _unVal.raw = 0; } }

ppstep output:

ppstep test.c 
Preprocessing test.c:0:1.
pp (started)> s
[test.c:91:1]: _REF_TYPE_DECL ( OT_STRING_UTF8 , SQStringUtf8 , StrUtf8 )
pp (called)> s
[test.c:91:1]: SQObjectPtr ( SQStringUtf8 * x ) { SQ_OBJECT_RAWINIT ( ) }
pp (expanded)> s
[test.c:91:1]: SQObjectPtr ( SQStringUtf8 * x ) { SQ_OBJECT_RAWINIT ( ) }
pp (called)> s
[test.c:91:1]: SQObjectPtr ( SQStringUtf8 * x ) { { _unVal . raw = 0 ; } }
pp (expanded)> s
[test.c:91:1]: SQObjectPtr ( SQStringUtf8 * x ) { { _unVal . raw = 0 ; } }
pp (rescanned)> s
[test.c:91:1]: SQObjectPtr ( SQStringUtf8 * x ) { { _unVal . raw = 0 ; } }
pp (rescanned)> s
[test.c:91:52]: SQObjectPtr ( SQStringUtf8 * x ) { { _unVal . raw = 0 ; } } 

pp (lexed)> s
[test.c:94:1]: SQObjectPtr ( SQStringUtf8 * x ) { { _unVal . raw = 0 ; } } 

pp (complete)> s

from panda3d.

rdb commented on May 30, 2024

That doesn't look like a preprocessor issue to me, because if I declare the SQStringUtf8 identifier, it parses fine:

struct SQStringUtf8;

#define SQ_OBJECT_RAWINIT() { _unVal.raw = 0; }

#define _REF_TYPE_DECL(type,_class,sym) \
  SQObjectPtr(_class * x) \
  { \
    SQ_OBJECT_RAWINIT() \
  }

struct SQObjectPtr {
  _REF_TYPE_DECL(OT_STRING_UTF8,SQStringUtf8,StrUtf8)
};

from panda3d.

mingodad commented on May 30, 2024

The problem is with the preprocessor not expanding SQ_OBJECT_RAWINIT() inside the macro _REF_TYPE_DECL the syntax error later on it's correct because this is a cut down from the original source code only for the purpose to check the preprocessor expansion.

from panda3d.

mingodad commented on May 30, 2024

Expansion of _REF_TYPE_DECL:
 -> SQObjectPtr( SQStringUtf8 * x) { SQ_OBJECT_RAWINIT() }

Should be:

Expansion of _REF_TYPE_DECL:
 -> SQObjectPtr( SQStringUtf8 * x) { { _unVal.raw = 0; } }

from panda3d.

rdb commented on May 30, 2024

It just hasn't gotten to that part yet when it's issuing the error. It's tripping over SQStringUtf8.

from panda3d.

mingodad commented on May 30, 2024

Here is the file that I'm testing and getting the error:
am-squilu.cpp.zip

from panda3d.

rdb commented on May 30, 2024

SQStringUtf8 is actually defined further down in the file, which is why it's not seeing it. When I add struct SQStringUtf8; somewhere above it, it gets past that point.

I guess the compiler considers line 494 to be a sufficient forward declaration of this type, and interrogate doesn't. Hmm.

from panda3d.

mingodad commented on May 30, 2024

And here is how to build the am-squilu.cpp:

g++ -DMAKE_SQ_CMD -o am-squilu am-squilu.cpp

from panda3d.

mingodad commented on May 30, 2024

SQStringUtf8 is actually defined further down in the file, which is why it's not seeing it. When I add struct SQStringUtf8; somewhere above it, it gets past that point.

I guess the compiler considers line 494 to be a sufficient forward declaration of this type, and interrogate doesn't. Hmm.

Good point !

from panda3d.

mingodad commented on May 30, 2024

Why it shows the macro not fully expanded (see the SQ_OBJECT_RAWINIT() there) ?

cppparser-nb -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include am-squilu.cpp
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
...
am-squilu.cpp:1377:2: error: syntax error
	_REF_TYPE_DECL(OT_STRING_UTF8,SQStringUtf8,StrUtf8)
 ^
with _REF_TYPE_DECL() expanded to: SQObjectPtr( SQStringUtf8 * x) { SQ_OBJECT_RAWINIT() _type= OT_STRING_UTF8 ; _unVal.pStrUtf8 = x; assert(_unVal.pTable); _unVal.pRefCounted->_uiRef++; } inline SQObjectPtr& operator=( SQStringUtf8 *x) { SQObjectValue old_unVal; bool isRefCounted = ISREFCOUNTED(_type); if(isRefCounted) old_unVal = _unVal; _type = OT_STRING_UTF8 ; SQ_REFOBJECT_INIT() _unVal.pStrUtf8 = x; _unVal.pRefCounted->_uiRef++; if(isRefCounted) __ReleaseRefCounted(old_unVal); return *this; } inline SQStringUtf8 *toStrUtf8 () { assert(_type == OT_STRING_UTF8 ); return _unVal.pStrUtf8 ; }
Error in parsing.

from panda3d.

mingodad commented on May 30, 2024

Also the other internal macros were not expanded ISREFCOUNTED(_type), SQ_REFOBJECT_INIT().

from panda3d.

rdb commented on May 30, 2024

As I said earlier, it's choking on the SQStringUtf8 before it has gotten to the SQ_OBJECT_RAWINIT macro. Whenever the parser sees a token that is a macro, it expands it before continuing. There's no requirement to preprocess the whole line / x tokens ahead.

from panda3d.

mingodad commented on May 30, 2024

Ok, thank you again !

from panda3d.

mingodad commented on May 30, 2024

Interesting that the previous ones like _REF_TYPE_DECL(OT_ARRAY,SQInt8Array,Int8Array) that have SQInt8Array also defined only further down on line 2764 doesn't get flagged with syntax error.

from panda3d.

rdb commented on May 30, 2024

Because SQInt8Array has a forward declaration on line 131.

from panda3d.

mingodad commented on May 30, 2024

I was going to point out my mistake when I posted it, sorry and thank you again !

from panda3d.

mingodad commented on May 30, 2024

Let it that way for now, adding struct SQStringUtf8; after struct SQString; on line 132 and testing it again it stops now here:

    Pushing to expansion " TK_FALSE , "
      Expanding ENUM_TK to  TK_FALSE , 
      End of input stream, restoring to previous input
    36503. IDENTIFIER TK_FALSE
    36504. ','
    Pushing to expansion " TK_"am-squilu.cpp" , "
      Expanding ENUM_TK to  TK_"am-squilu.cpp" , 
      36505. IDENTIFIER TK_
      36506. SIMPLE_STRING am-squilu.cpp
/home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/am-squilu.cpp:3379:5: error: syntax error
    SQ_KEYWORDS_LIST()
    ^
with SQ_KEYWORDS_LIST() expanded to: TK_"am-squilu.cpp" ,
Error in parsing.

from panda3d.

mingodad commented on May 30, 2024

It's expanding ENUM_TK(__FILE__)\ to ENUM_TK("am-squilu.cpp")\ and it shouldn't.

from panda3d.

mingodad commented on May 30, 2024

#define ENUM_TK(tk) TK_##tk
#define SQ_KEYWORDS_LIST() \
	ENUM_TK(__FILE__)

int SQ_KEYWORDS_LIST();

cppparser output:

cppparser-nb -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
  0. START_CPP
  #define ENUM_TK(tk) TK_##tk
  #define SQ_KEYWORDS_LIST() 
	ENUM_TK(__FILE__)
  1. KW_INT
  Pushing to expansion " ENUM_TK(__FILE__) "
    Expanding SQ_KEYWORDS_LIST to  ENUM_TK(__FILE__) 
    Pushing to expansion " TK_"test.c" "
      Expanding ENUM_TK to  TK_"test.c" 
      2. IDENTIFIER TK_
      3. SIMPLE_STRING test.c
test.c:97:5: error: syntax error
int SQ_KEYWORDS_LIST();
    ^
with SQ_KEYWORDS_LIST() expanded to: TK_"test.c"
Error in parsing.

gcc output:

gcc -E -P test.c
int TK___FILE__;

ppstep output :

ppstep test.c 
Preprocessing test.c:0:1.
pp (started)> s
[test.c:97:1]: int
pp (lexed)> s
[test.c:97:5]: int SQ_KEYWORDS_LIST ( )
pp (called)> s
[test.c:97:5]: int ENUM_TK ( __FILE__ )
pp (expanded)> s
[test.c:97:5]: int ENUM_TK ( __FILE__ )
pp (called)> s
[test.c:97:5]: int TK___FILE__
pp (expanded)> s
[test.c:97:5]: int TK___FILE__
pp (rescanned)> s
[test.c:97:5]: int TK___FILE__
pp (rescanned)> s
[test.c:97:23]: int TK___FILE__ ;
pp (lexed)> s
[test.c:97:24]: int TK___FILE__ ; 

pp (lexed)> s
[test.c:100:1]: int TK___FILE__ ; 

pp (complete)> s

from panda3d.

mingodad commented on May 30, 2024

Notice that there is ENUM_TK(__FILE__), ENUM_TK(__FUNCTION__), ENUM_TK(__LINE__) and lloking through the code I didn't see __FUNCTION__ anywhere but that would be OK because it will not expand it.

from panda3d.

rdb commented on May 30, 2024

It's not specific to __FILE__, because this also fails:

#define my_file "test"
#define a(x) a ## x

a(my_file)

from panda3d.

mingodad commented on May 30, 2024

It seems that __FILE__, __LINE__, __FUNCTION__ and maybe others are treated differently because here we also have then somehow specifically tested in several places.

from panda3d.

rdb commented on May 30, 2024

They're not. It fails for any macro. The fix I checked in at the beginning of this thread is incorrect, evidently, it shouldn't expand the arguments before passing them into the macro function. I need to back it out and think about how it should have been fixed.

from panda3d.

rdb commented on May 30, 2024

6.10.3.1 Argument substitution

After the arguments for the invocation of a function-like macro have been identified, argument substitution takes place. A parameter in the replacement list, unless preceded by a # or ## preprocessing token or followed by a ## preprocessing token (see below), is replaced by the corresponding argument after all macros contained therein have been expanded. Before being substituted, each argument’s preprocessing tokens are completely macro replaced as if they formed the rest of the preprocessing file; no other preprocessing tokens are available.

So we need to avoid doing the expansion for arguments that are participating in stringification or token-pasting. Actually, an argument could be used multiple times in a macro, sometimes requiring and sometimes not requiring expansion. Bleh.

from panda3d.

mingodad commented on May 30, 2024

There is this project https://github.com/lpsantil/ucpp that I'm using here https://github.com/mingodad/CG-SQL-Lua-playground and it seems to process fine the last and the previous troubled examples.

Also https://www.boost.org/doc/libs/1_84_0/libs/wave/doc/macro_expansion_process.html that is used by ppstep https://github.com/notfoundry/ppstep seems to have some documentation to help.

from panda3d.

rdb commented on May 30, 2024

Phoo, that was a lot of work, but I think I got it now. Your am-squilu file parses.

To make debugging easier, I added a preprocessor mode to parse_file. Use parse_file -E and it should behave like cc -E.

from panda3d.

rdb commented on May 30, 2024

I forgot about the SQStringUtf8 issue, which is not a preprocessor issue, but cppparser misapplying forward declaration rules.

The rules are pretty bizarre, since this:

struct A {
  struct B;
};

declares B as a subtype of A, whereas this:

struct A {
  struct B *b;
};

actually declares B in the global scope!

Even more mind-bogglingly, this:

namespace N {
  struct A {
    struct B *b;
  };
}

forward-declares B as N::B. 🤯

EDIT: This turns out to be pretty tricky to solve, because the parser doesn't distinguish between a forward declaration and an elaborated type specifier. I'm going to leave it for now, but feel free to create an issue to resolve that if you don't want it to be forgotten.

from panda3d.

mingodad commented on May 30, 2024

Every day is a day to learn anything new !

from panda3d.

mingodad commented on May 30, 2024

What's your system ?
Because here on Ubuntu 18.04 I'm now getting weird errors and Lua-5.4.6 that was before parsing fine now it's not, still looking how to get a minimum test case.

To build it use gcc -Os -DMAKE_LUA_CMD -DLUA_PROGNAME='"lua"' -DLUA_COMPAT_5_3 -DLUA_USE_LINUX -D_XOPEN_SOURCE=500 -o am-lua-5.4.6 am-lua-5.4.6.c -lm -ldl -lreadline.
am-lua-5.4.6.c.zip

from panda3d.

rdb commented on May 30, 2024

I tried and it parses fine for me (after adding some stub definitions for signal.h to parser-inc).

It works both with parse_file -E and the old parse_file mode for me.

I'm on macOS arm64.

Can you share which errors you're getting? Maybe I would have a clue.

from panda3d.

mingodad commented on May 30, 2024

Here is the output of parsing /usr/include/stdio.h:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include /usr/include/stdio.h
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading /usr/include/stdio.h
  Reading /usr/include/x86_64-linux-gnu/bits/libc-header-start.h
    Reading /usr/include/features.h
      Reading /usr/include/stdc-predef.h
      Reading /usr/include/x86_64-linux-gnu/sys/cdefs.h
        Reading /usr/include/x86_64-linux-gnu/bits/wordsize.h
        Reading /usr/include/x86_64-linux-gnu/bits/long-double.h
      Reading /usr/include/x86_64-linux-gnu/gnu/stubs.h
        Reading /usr/include/x86_64-linux-gnu/gnu/stubs-32.h
  Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
  Reading /usr/include/x86_64-linux-gnu/bits/types.h
    Reading /usr/include/features.h
    Reading /usr/include/x86_64-linux-gnu/bits/wordsize.h
    Reading /usr/include/x86_64-linux-gnu/bits/typesizes.h
  Reading /usr/include/x86_64-linux-gnu/bits/types/__FILE.h
  Reading /usr/include/x86_64-linux-gnu/bits/types/FILE.h
  Reading /usr/include/x86_64-linux-gnu/bits/libio.h
    /usr/include/x86_64-linux-gnu/bits/libio.h:32:9: error: "Never include <bits/libio.h> directly; use <stdio.h> instead."
    # error "Never include <bits/libio.h> directly; use <stdio.h> instead."
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Reading /usr/include/x86_64-linux-gnu/bits/_G_config.h
      /usr/include/x86_64-linux-gnu/bits/_G_config.h:8:9: error: "Never include <bits/_G_config.h> directly; use <stdio.h> instead."
      # error "Never include <bits/_G_config.h> directly; use <stdio.h> instead."
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      Reading /usr/include/x86_64-linux-gnu/bits/types.h
      Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
      Reading /usr/include/x86_64-linux-gnu/bits/types/__mbstate_t.h
    Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stdarg.h
  Reading /usr/include/x86_64-linux-gnu/bits/stdio_lim.h
  Reading /usr/include/x86_64-linux-gnu/bits/sys_errlist.h
Error in parsing.

It's like it's not registering the include guards:

#if !defined _STDIO_H && !defined _LIBIO_H
# error "Never include <bits/libio.h> directly; use <stdio.h> instead."
#endif

/usr/include/stdio.h:

#ifndef _STDIO_H
#define _STDIO_H	1
...

from panda3d.

rdb commented on May 30, 2024

Oops, it's working with defined(_STDIO_H) but not with defined _STDIO_H, must be a regression in my last commit. I did refactor code relating to this...

from panda3d.

rdb commented on May 30, 2024

Yup, that was it, fix pushed... guess none of the headers I tested on used this form of defined, sorry about that

from panda3d.

mingodad commented on May 30, 2024

Thank you again !
It's parsing with minor glitches that I'm looking now.

sqlite3.c output that I'm still trying to figure out what's the problem:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include sqlite3.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/sqlite3.c
  Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stdarg.h
...
  Reading /usr/include/x86_64-linux-gnu/sys/mman.h
    Reading /usr/include/features.h
    Reading /usr/include/x86_64-linux-gnu/bits/types.h
    Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
    Reading /usr/include/x86_64-linux-gnu/bits/mman.h
      Reading /usr/include/x86_64-linux-gnu/bits/mman-linux.h
        Reading /usr/include/x86_64-linux-gnu/bits/mman-shared.h
  /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/sqlite3.c:38741:46: error: syntax error
    { "stat",         (sqlite3_syscall_ptr)stat,       0  },
                                               ^
Error in parsing.

from panda3d.

rdb commented on May 30, 2024

You seem to have detailed error messages turned off in bison... even with my old bison 2.3 I'm getting more descriptive error messages than that!

I'm guessing the problem is that stat is both a struct and a function and interrogate does not understand that. This also fails:

struct stat;

void stat();

void *ptr = (void *)stat;

  /private/tmp/test.c:5:25: error: syntax error, unexpected ';', expecting '{' or '('
  void *ptr = (void *)stat;
                          ^

from panda3d.

rdb commented on May 30, 2024

Could it be this simple...?

diff --git a/dtool/src/cppparser/cppScope.cxx b/dtool/src/cppparser/cppScope.cxx
index d091cae756..cc3b362141 100644
--- a/dtool/src/cppparser/cppScope.cxx
+++ b/dtool/src/cppparser/cppScope.cxx
@@ -724,6 +733,12 @@ find_symbol(const string &name, bool recurse) const {
     return _struct_type;
   }
 
+  Functions::const_iterator fi;
+  fi = _functions.find(name);
+  if (fi != _functions.end()) {
+    return (*fi).second;
+  }
+
   Types::const_iterator ti;
   ti = _types.find(name);
   if (ti != _types.end()) {
@@ -741,12 +756,6 @@ find_symbol(const string &name, bool recurse) const {
     return (*vi).second;
   }
 
-  Functions::const_iterator fi;
-  fi = _functions.find(name);
-  if (fi != _functions.end()) {
-    return (*fi).second;
-  }
-
   Using::const_iterator ui;
   for (ui = _using.begin(); ui != _using.end(); ++ui) {
     CPPDeclaration *decl = (*ui)->find_symbol(name, false);

from panda3d.

mingodad commented on May 30, 2024

With the changes you've shown above it goes forward and now stops here:

      Reading /usr/include/x86_64-linux-gnu/bits/mman-linux.h
        Reading /usr/include/x86_64-linux-gnu/bits/mman-shared.h
  /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/sqlite3.c:44350:36: error: unknown literal suffix SQLITE_TEMP_FILE_PREFIX
        sqlite3_snprintf(nBuf, zBuf, "%s/"SQLITE_TEMP_FILE_PREFIX"%llx%c",
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Reading /usr/include/dlfcn.h
    Reading /usr/include/features.h
    Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
    Reading /usr/include/x86_64-linux-gnu/bits/dlfcn.h
  /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/sqlite3.c:137413:18: error: syntax error, unexpected '(', expecting ')'
    (sqlite3_value*(*)(const sqlite3_value*))sqlite3_value_dup,
                   ^
Error in parsing.

Here is the sqlite3 that I'm testing with volatile commented out /*volatile*/:
Uploading sqlite3.zip…

from panda3d.

mingodad commented on May 30, 2024

I've sent the previous message before it finished uploading:
sqlite3.zip

from panda3d.

mingodad commented on May 30, 2024

Here is the minimal failed code:

# define SQLITE_TEMP_FILE_PREFIX "etilqs_"
const char *fmr = "%s/"SQLITE_TEMP_FILE_PREFIX"%llx%c";

gcc output:

gcc -E -P test.c
const char *fmr = "%s/""etilqs_""%llx%c";

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading test.c
  test.c:105:19: error: unknown literal suffix SQLITE_TEMP_FILE_PREFIX
  const char *fmr = "%s/"SQLITE_TEMP_FILE_PREFIX"%llx%c";
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error in parsing.

from panda3d.

rdb commented on May 30, 2024

This is actually invalid in C++11:

#define SUFFIX "bar"

const char *str = "foo"SUFFIX;

❯ clang++ -std=c++11 test.cpp
test.cpp:3:24: error: invalid suffix on literal; C++11 requires a space between literal and identifier [-Wreserved-user-defined-literal]
const char *str = "foo"SUFFIX;
                       ^

Maybe we ought to allow it anyway, and let interrogate parse either C or C++? There might be ambiguity due to the existence of literal suffixes in C++11.

from panda3d.

mingodad commented on May 30, 2024

Sorry I forgot/remember that in C++ it's invalid.

from panda3d.

mingodad commented on May 30, 2024

Add a space between the strings and the MACRO constant and also commenting out the weird cast (sqlite3_value*(*)(const sqlite3_value*))sqlite3_value_dup, it could parse it all (volatile also commented).

Great work !
Thank you again for all help !

I'll continue testing it to see if there is more easy hanging fruits.

from panda3d.

mingodad commented on May 30, 2024

There is also this C++ parser https://github.com/robertoraggi/cplusplus I'll invite the author to look here and if possible give some help.

from panda3d.

rdb commented on May 30, 2024

I checked in a fix for the stat problem and also made a change allowing the C-style spaceless prefix to work. If there's no matching string literal, the parser will check if there's a macro defined by that name, and if so, separate the tokens.

Minimal repro for the function pointer issue:

typedef struct sqlite3_value sqlite3_value;
sqlite3_value *sqlite3_value_dup(const sqlite3_value*);

void *thing = (sqlite3_value*(*)(const sqlite3_value*))sqlite3_value_dup;

It's failing to parse the type name in the return value of the function pointer. It parses fine when the return type is void.

I'm not well-versed with bison. If you would like to look into this, feel free.

from panda3d.

mingodad commented on May 30, 2024

There is this small one that maybe is related to the evaluation of define :

#ifdef __WCHAR_MAX__
# define __WCHAR_MAX	__WCHAR_MAX__
#elif L'\0' - 1 > 0
# define __WCHAR_MAX	(0xffffffffu + L'\0')
#else
# define __WCHAR_MAX	(0x7fffffff + L'\0')
#endif

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/test.c
  /home/mingo/dev/c/A_frameworks/panda3d/cppparser-nb/test.c:109:2: warning: Ignoring invalid expression L'\0' - 1 > 0
  #elif L'\0' - 1 > 0
   ^
Finished parsing.

from panda3d.

mingodad commented on May 30, 2024

I'll look at the weird cast after some more testings.

from panda3d.

rdb commented on May 30, 2024

Haha, it was seeing the L as an undefined macro and expanding it to 0, so it would end up trying to parse 0'\0'.

Turns out this is another special edge case. Apparently, none of the other literal prefixes work in clang's preprocessor, they are all expanded as macros.

Fix pushed.

from panda3d.

mingodad commented on May 30, 2024

Also this small difference when using the new -E:

const char *fmt = "unrecognized token: \"%s\"";

without -E output as expected:

cppparser-nb -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Finished parsing.
char const *fmt = "unrecognized token: \"%s\"";

with -E the string scapped quotes disappear:

cppparser-nb -E -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t

const char * fmt = "unrecognized token: "%s"";
Finished parsing.

from panda3d.

rdb commented on May 30, 2024

This is just a printing problem, when it's printing the AST it escapes properly but when printing the tokens it doesn't.

I'll fix this later.

from panda3d.

mingodad commented on May 30, 2024

It'll be useful because I was trying to pass the output of using -E to gcc -fsyntax-only cppparser-preprocessed-with-E to spot errors.

from panda3d.

rdb commented on May 30, 2024

Fair, fix pushed.

That looks useful. We can also try to do a roundtrip: parse_file -> gcc and compare the result of that with gcc directly.

from panda3d.

mingodad commented on May 30, 2024

Another curiosity is the enum declaration with an initializer:

enum
  {
    _SC_ARG_MAX,
    _SC_UIO_MAXIOV,
    _SC_IOV_MAX = _SC_UIO_MAXIOV,
    _SC_PII_INTERNET_STREAM,
    _SC_PII_INTERNET_DGRAM,
    _SC_PII_OSI_COTS,
    _SC_PII_OSI_CLTS,
  };

cppparser output:

cppparser-nb -v -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Reading test.c
Finished parsing.
enum {
  _SC_ARG_MAX = 0,
  _SC_UIO_MAXIOV = 1,
  _SC_IOV_MAX = ::_SC_UIO_MAXIOV,
  _SC_PII_INTERNET_STREAM = (::_SC_UIO_MAXIOV + 1),
  _SC_PII_INTERNET_DGRAM = ((::_SC_UIO_MAXIOV + 1) + 1),
  _SC_PII_OSI_COTS = (((::_SC_UIO_MAXIOV + 1) + 1) + 1),
  _SC_PII_OSI_CLTS = ((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1),
};

gcc output:

gcc -E -P test.c
enum
  {
    _SC_ARG_MAX,
    _SC_UIO_MAXIOV,
    _SC_IOV_MAX = _SC_UIO_MAXIOV,
    _SC_PII_INTERNET_STREAM,
    _SC_PII_INTERNET_DGRAM,
    _SC_PII_OSI_COTS,
    _SC_PII_OSI_CLTS,
  };

from panda3d.

mingodad commented on May 30, 2024

Here is the partial output :

  _SC_IOV_MAX = ::_SC_UIO_MAXIOV,
  _SC_PII_INTERNET_STREAM = (::_SC_UIO_MAXIOV + 1),
  _SC_PII_INTERNET_DGRAM = ((::_SC_UIO_MAXIOV + 1) + 1),
  _SC_PII_OSI_COTS = (((::_SC_UIO_MAXIOV + 1) + 1) + 1),
  _SC_PII_OSI_CLTS = ((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1),
  _SC_PII_OSI_M = (((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1),
  _SC_T_IOV_MAX = ((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREADS = (((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_SAFE_FUNCTIONS = ((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_GETGR_R_SIZE_MAX = (((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_GETPW_R_SIZE_MAX = ((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_LOGIN_NAME_MAX = (((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_TTY_NAME_MAX = ((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_DESTRUCTOR_ITERATIONS = (((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_KEYS_MAX = ((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_STACK_MIN = (((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_THREADS_MAX = ((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_ATTR_STACKADDR = (((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_ATTR_STACKSIZE = ((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_PRIORITY_SCHEDULING = (((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_PRIO_INHERIT = ((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_PRIO_PROTECT = (((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_THREAD_PROCESS_SHARED = ((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_NPROCESSORS_CONF = (((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_NPROCESSORS_ONLN = ((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_PHYS_PAGES = (((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_AVPHYS_PAGES = ((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_ATEXIT_MAX = (((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_PASS_MAX = ((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_VERSION = (((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_XCU_VERSION = ((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_UNIX = (((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_CRYPT = ((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_ENH_I18N = (((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_SHM = ((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_2_CHAR_TERM = (((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_2_C_VERSION = ((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_2_UPE = (((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_XPG2 = ((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_XPG3 = (((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_XOPEN_XPG4 = ((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_CHAR_BIT = (((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_CHAR_MAX = ((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_CHAR_MIN = (((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_INT_MAX = ((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_INT_MIN = (((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_LONG_BIT = ((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_WORD_BIT = (((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_MB_LEN_MAX = ((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_NZERO = (((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_SSIZE_MAX = ((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_SCHAR_MAX = (((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_SCHAR_MIN = ((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_SHRT_MAX = (((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_SHRT_MIN = ((((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_UCHAR_MAX = (((((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_UINT_MAX = ((((((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
  _SC_ULONG_MAX = (((((((((((((((((((((((((((((((((((((((((((((((((((((((((::_SC_UIO_MAXIOV + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1) + 1),
...

from panda3d.

mingodad commented on May 30, 2024

I think that's not important because the output with -E is usable:

cppparser-nb -E -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include test.c
Predefining __PANDA_CC__ as 
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t

enum
{
  _SC_ARG_MAX,
  _SC_UIO_MAXIOV,
  _SC_IOV_MAX = _SC_UIO_MAXIOV,
  _SC_PII_INTERNET_STREAM,
  _SC_PII_INTERNET_DGRAM,
  _SC_PII_OSI_COTS,
  _SC_PII_OSI_CLTS,
};
Finished parsing.

from panda3d.

rdb commented on May 30, 2024

Great, thank you very much for your help testing and filing issues! 😊

from panda3d.

mingodad commented on May 30, 2024

Not at all !
Good night !

from panda3d.

mingodad commented on May 30, 2024

Now that it's working for the majority of tests I did so far and before dive in the parser itself I want to see if there is any hanging fruit to improve the performance of it.

/usr/bin/time dist/Release/GNU-Linux/cppparser-nb  -E -D__PANDA_CC__ -D__builtin_va_list=void* -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include sqlite3.c > /dev/null
Predefining __PANDA_CC__ as
Predefining __builtin_va_list as void*
Predefining wchar_t as pccwchar_t
Finished parsing.
13.90user 0.07system 0:13.99elapsed 99%CPU (0avgtext+0avgdata 88104maxresident)k
0inputs+0outputs (0major+20917minor)pagefaults 0swaps

/usr/bin/time gcc -E -P sqlite3.c > /dev/null
0.16user 0.01system 0:00.18elapsed 99%CPU (0avgtext+0avgdata 39448maxresident)k
0inputs+0outputs (0major+8519minor)pagefaults 0swaps

Based on yesterday good results working together I'm asking if you are interested/have time to dive in ?

from panda3d.

rdb commented on May 30, 2024

I'm sure that there is a lot to improve in terms of performance. That said, it is fast enough right now for our use cases, so I don't think it's worth it for me to spend a significant amount of time on it. Pull requests are always welcome though!

from panda3d.

mingodad commented on May 30, 2024

Here is a partial view of the profile output:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls   s/call   s/call  name    
  8.20      1.00     1.00 127885800     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_S_right(std::_Rb_tree_node_base*)
  7.34      1.90     0.90 125425478     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_clone_node<std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node>(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > const*, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node&)
  6.07      2.64     0.74    36877     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_copy<std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node>(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > const*, std::_Rb_tree_node_base*, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node&)
  4.71      3.21     0.58 627127390     0.00     0.00  std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const& std::forward<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::remove_reference<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>::type&)
  4.47      3.76     0.55    47961     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*)
  4.26      4.28     0.52 125425478     0.00     0.00  void std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_construct_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&)
  3.77      4.74     0.46 125425478     0.00     0.00  std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>::pair(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&)
  3.03      5.11     0.37 125425478     0.00     0.00  void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&)
  2.79      5.45     0.34 125429611     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::allocate(unsigned long, void const*)
  2.70      5.78     0.33 501718444     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_get_Node_allocator()
  2.30      6.06     0.28 250859345     0.00     0.00  __gnu_cxx::__aligned_membuf<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_addr()
  2.21      6.33     0.27 125425478     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node::operator()<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&) const
  2.21      6.60     0.27 250859345     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_valptr()
  2.21      6.87     0.27 125425478     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_create_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&)
  2.13      7.13     0.26 125429611     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_destroy_node(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*)
  2.13      7.39     0.26 125429611     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_drop_node(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*)
  2.05      7.64     0.25 125429611     0.00     0.00  void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>*)
  1.89      7.87     0.23 250859345     0.00     0.00  __gnu_cxx::__aligned_membuf<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_ptr()
  1.80      8.09     0.22 125429611     0.00     0.00  std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > >::deallocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >&, std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*, unsigned long)
  1.80      8.31     0.22 125429611     0.00     0.00  std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >&, unsigned long)
  1.60      8.50     0.20       96     0.00     0.00  void std::allocator_traits<std::allocator<std::_Rb_tree_node<CPPFile> > >::construct<CPPFile, CPPFile const&>(std::allocator<std::_Rb_tree_node<CPPFile> >&, CPPFile*, CPPFile const&)
  1.56      8.69     0.19 125429611     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::deallocate(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*, unsigned long)
  1.56      8.88     0.19 125429611     0.00     0.00  void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>*)
  1.39      9.05     0.17 251306832     0.00     0.00  operator new(unsigned long, void*)
  1.31      9.21     0.16 125425478     0.00     0.00  void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&>(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&)
  1.23      9.36     0.15 125429611     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_get_node()
  1.15      9.50     0.14 130102870     0.00     0.00  __gnu_cxx::__aligned_membuf<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_ptr() const
  1.15      9.64     0.14 130102870     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_valptr() const
  1.15      9.78     0.14 125433801     0.00     0.00  std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>::~pair()
  1.07      9.91     0.13 125429611     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_put_node(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*)
  0.98     10.03     0.12     9584     0.00     0.00  void std::_Destroy_aux<false>::__destroy<CPPToken*>(CPPToken*, CPPToken*)
  0.90     10.14     0.11 125429611     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::max_size() const
  0.74     10.23     0.09 130102870     0.00     0.00  __gnu_cxx::__aligned_membuf<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::_M_addr() const
  0.74     10.32     0.09 12677581     0.00     0.00  Filename::Filename(Filename const&)
  0.66     10.40     0.08 126825490     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_S_left(std::_Rb_tree_node_base*)
  0.66     10.48     0.08 63316206     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_S_right(std::_Rb_tree_node_base const*)
  0.66     10.56     0.08    56910     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::end() const
  0.57     10.63     0.07 10905348     0.00     0.00  CPPPreprocessor::InputFile::get()
  0.57     10.70     0.07       96     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<CPPFile> >::max_size() const
  0.57     10.77     0.07        1     0.07     0.07  _GLOBAL__sub_I_current_scope
  0.53     10.84     0.07 125516178     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_S_left(std::_Rb_tree_node_base const*)
  0.53     10.90     0.07 23019883     0.00     0.00  bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*)
  0.41     10.95     0.05  7389707     0.00     0.00  Filename::operator=(Filename const&)
  0.41     11.00     0.05  6286208     0.00     0.00  Filename::operator=(char const*)
  0.29     11.04     0.04  3268929     0.00     0.00  CPPPreprocessor::InputFile::peek()
  0.25     11.07     0.03 17301905     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::empty() const
  0.25     11.10     0.03 14273696     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::operator--()
  0.25     11.13     0.03  3694580     0.00     0.00  CPPFile::operator=(CPPFile const&)
  0.25     11.16     0.03     2987     0.00     0.00  PandaFileStreamBuf::decode_newlines(char*, unsigned long, char const*, unsigned long)
  0.25     11.19     0.03    22248     0.00     0.00  __gnu_cxx::new_allocator<std::_List_node<CPPCommentBlock*> >::allocate(unsigned long, void const*)
  0.25     11.22     0.03     9584     0.00     0.00  void std::_Destroy<CPPToken*, CPPToken>(CPPToken*, CPPToken*, std::allocator<CPPToken>&)
  0.25     11.25     0.03       96     0.00     0.00  void __gnu_cxx::new_allocator<std::_Rb_tree_node<CPPFile> >::construct<CPPFile, CPPFile const&>(CPPFile*, CPPFile const&)
  0.25     11.28     0.03                             __gnu_cxx::new_allocator<std::_Rb_tree_node<Filename> >::deallocate(std::_Rb_tree_node<Filename>*, unsigned long)
  0.20     11.30     0.03  4603577     0.00     0.00  std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >::operator()(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> const&) const
  0.20     11.33     0.03  4603577     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_S_key(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > const*)
  0.16     11.35     0.02 14415369     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::_List_iterator(std::__detail::_List_node_base*)
  0.16     11.37     0.02 14380082     0.00     0.00  __gnu_cxx::__aligned_membuf<CPPPreprocessor::InputFile>::_M_ptr()
  0.16     11.39     0.02 14380082     0.00     0.00  std::_List_node<CPPPreprocessor::InputFile>::_M_valptr()
  0.16     11.41     0.02 14287317     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::end()
  0.16     11.43     0.02 10958947     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::back()
  0.16     11.45     0.02 10910022     0.00     0.00  CPPPreprocessor::get()
  0.16     11.47     0.02  6288063     0.00     0.00  Filename::locate_extension()
  0.16     11.49     0.02  6286769     0.00     0.00  Filename::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)
  0.16     11.51     0.02  4601653     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_impl::~_Vector_impl()
  0.16     11.53     0.02  4601653     0.00     0.00  void std::_Destroy<CPPAttributeList::Attribute*>(CPPAttributeList::Attribute*, CPPAttributeList::Attribute*)
  0.16     11.55     0.02   636025     0.00     0.00  CPPToken::output_code(std::ostream&) const
  0.16     11.57     0.02   218251     0.00     0.00  CPPPreprocessor::check_keyword(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.16     11.59     0.02    47961     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::~new_allocator()
  0.16     11.61     0.02    36877     0.00     0.00  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::new_allocator(__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > const&)
  0.16     11.63     0.02     9584     0.00     0.00  void std::_Destroy<CPPToken*>(CPPToken*, CPPToken*)
  0.12     11.64     0.02  1481409     0.00     0.00  std::allocator_traits<std::allocator<CPPAttributeList::Attribute> >::select_on_container_copy_construction(std::allocator<CPPAttributeList::Attribute> const&)
  0.12     11.66     0.02    39043     0.00     0.00  CPPPreprocessor::InputFile::connect_input(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.08     11.67     0.01 18559074     0.00     0.00  Filename::~Filename()
  0.08     11.68     0.01 14380082     0.00     0.00  __gnu_cxx::__aligned_membuf<CPPPreprocessor::InputFile>::_M_addr()
  0.08     11.69     0.01  6288246     0.00     0.00  Filename::locate_hash()
  0.08     11.70     0.01  6286568     0.00     0.00  Filename::set_text()
  0.08     11.71     0.01  6286208     0.00     0.00  Filename::Filename(char const*)
  0.08     11.72     0.01  4601667     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_impl_data::_Vector_impl_data()
  0.08     11.73     0.01  3763468     0.00     0.00  CPPPreprocessor::skip_comment(int)
  0.08     11.74     0.01  3309585     0.00     0.00  std::reverse_iterator<std::_List_iterator<CPPPreprocessor::InputFile> >::reverse_iterator(std::_List_iterator<CPPPreprocessor::InputFile>)
  0.08     11.75     0.01  3291839     0.00     0.00  std::reverse_iterator<std::_List_iterator<CPPPreprocessor::InputFile> >::operator*() const
  0.08     11.76     0.01  3254892     0.00     0.00  CPPPreprocessor::peek()
  0.08     11.77     0.01  3195033     0.00     0.00  CPPFile::CPPFile(CPPFile const&)
  0.08     11.78     0.01  3183359     0.00     0.00  std::_List_const_iterator<CPPPreprocessor::InputFile>::_List_const_iterator(std::__detail::_List_node_base const*)
  0.08     11.79     0.01  3164378     0.00     0.00  std::_List_node<CPPPreprocessor::InputFile>::_M_valptr() const
  0.08     11.80     0.01  3120174     0.00     0.00  CPPAttributeList::CPPAttributeList()
  0.08     11.81     0.01  3110846     0.00     0.00  cppyystype::cppyystype()
  0.08     11.82     0.01  2369988     0.00     0.00  cppyyltype::cppyyltype(cppyyltype const&)
  0.08     11.83     0.01  2209344     0.00     0.00  cppyyltype::operator=(cppyyltype const&)
  0.08     11.84     0.01  2143064     0.00     0.00  CPPToken::~CPPToken()
  0.08     11.85     0.01  1654575     0.00     0.00  std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::size() const
  0.08     11.86     0.01  1481437     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_M_get_Tp_allocator() const
  0.08     11.87     0.01  1481409     0.00     0.00  bool __gnu_cxx::operator!=<CPPAttributeList::Attribute const*, std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > >(__gnu_cxx::__normal_iterator<CPPAttributeList::Attribute const*, std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > > const&, __gnu_cxx::__normal_iterator<CPPAttributeList::Attribute const*, std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > > const&)
  0.08     11.88     0.01  1481409     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_base(unsigned long, std::allocator<CPPAttributeList::Attribute> const&)
  0.08     11.89     0.01  1481409     0.00     0.00  std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::vector(std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > const&)
  0.08     11.90     0.01  1481386     0.00     0.00  cppyystype::cppyystype(cppyystype const&)
  0.08     11.91     0.01  1174858     0.00     0.00  CPPPreprocessor::get_line_number() const
  0.08     11.92     0.01   666351     0.00     0.00  CPPToken::CPPToken(int, int, int, CPPFile const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyystype const&)
  0.08     11.93     0.01   661678     0.00     0.00  CPPToken::CPPToken(CPPToken const&)
  0.08     11.94     0.01   336549     0.00     0.00  std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::end() const
  0.08     11.95     0.01   335226     0.00     0.00  std::_Vector_base<CPPNameComponent, std::allocator<CPPNameComponent> >::_M_get_Tp_allocator() const
  0.08     11.96     0.01   335226     0.00     0.00  std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::_S_relocate(CPPNameComponent*, CPPNameComponent*, CPPNameComponent*, std::allocator<CPPNameComponent>&)
  0.08     11.97     0.01   335226     0.00     0.00  CPPToken const& std::forward<CPPToken const&>(std::remove_reference<CPPToken const&>::type&)
  0.08     11.98     0.01   290771     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >*, std::_Rb_tree_node_base*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.08     11.99     0.01   252703     0.00     0.00  CPPPreprocessor::get_identifier(int)
  0.08     12.00     0.01   172404     0.00     0.00  void std::allocator_traits<std::allocator<CPPToken> >::destroy<CPPToken>(std::allocator<CPPToken>&, CPPToken*)
  0.08     12.01     0.01   167613     0.00     0.00  std::_Vector_base<CPPNameComponent, std::allocator<CPPNameComponent> >::_Vector_impl_data::_Vector_impl_data()
  0.08     12.02     0.01   167613     0.00     0.00  void std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::_M_realloc_insert<CPPNameComponent>(__gnu_cxx::__normal_iterator<CPPNameComponent*, std::vector<CPPNameComponent, std::allocator<CPPNameComponent> > >, CPPNameComponent&&)
  0.08     12.03     0.01   167613     0.00     0.00  std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::begin()
  0.08     12.04     0.01    56255     0.00     0.00  CPPPreprocessor::get_number(int)
  0.08     12.05     0.01    50844     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::operator++()
  0.08     12.06     0.01    45133     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::begin() const
  0.08     12.07     0.01    36877     0.00     0.00  std::_Rb_tree_node_base::_S_maximum(std::_Rb_tree_node_base*)
  0.08     12.08     0.01    36877     0.00     0.00  std::_Rb_tree_node_base::_S_minimum(std::_Rb_tree_node_base*)
  0.08     12.09     0.01    36877     0.00     0.00  std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_copy<std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node>(std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > const&, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_Alloc_node&)
  0.08     12.10     0.01    13853     0.00     0.00  CPPPreprocessor::expand_manifests(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, CPPManifest*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > > const&, bool) const
  0.08     12.11     0.01     4190     0.00     0.00  CPPManifest::CPPManifest(CPPPreprocessor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&)
  0.08     12.12     0.01        1     0.01    12.03  CPPPreprocessor::preprocess_file(Filename const&)
  0.08     12.13     0.01                             std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<CPPManifest*, std::allocator<CPPManifest*> > > >::operator*() const
  0.08     12.14     0.01                             std::allocator_traits<std::allocator<CPPAttributeList::Attribute> >::deallocate(std::allocator<CPPAttributeList::Attribute>&, CPPAttributeList::Attribute*, unsigned long)
  0.08     12.15     0.01                             CPPDeclaration** std::__copy_move_a2<true, CPPDeclaration**, CPPDeclaration**>(CPPDeclaration**, CPPDeclaration**, CPPDeclaration**)
  0.04     12.15     0.01  6083090     0.00     0.00  std::allocator<CPPAttributeList::Attribute>::~allocator()
  0.04     12.16     0.01  3164378     0.00     0.00  __gnu_cxx::__aligned_membuf<CPPPreprocessor::InputFile>::_M_ptr() const
  0.04     12.16     0.01  3120174     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_impl::_Vector_impl()
  0.04     12.17     0.01  2972195     0.00     0.00  std::allocator<CPPAttributeList::Attribute>::allocator(std::allocator<CPPAttributeList::Attribute> const&)
  0.04     12.17     0.01   346332     0.00     0.00  __gnu_cxx::__normal_iterator<CPPAttributeList::Attribute*, std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > >::base() const
  0.04     12.18     0.01   342923     0.00     0.00  std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> > >::_M_begin()
  0.04     12.18     0.01   335226     0.00     0.00  std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::_S_do_relocate(CPPNameComponent*, CPPNameComponent*, CPPNameComponent*, std::allocator<CPPNameComponent>&, std::integral_constant<bool, true>)
  0.04     12.19     0.01   167613     0.00     0.00  std::allocator_traits<std::allocator<CPPNameComponent> >::allocate(std::allocator<CPPNameComponent>&, unsigned long)
  0.04     12.19     0.01     4572     0.00     0.00  std::enable_if<std::__and_<std::allocator_traits<pallocator_array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::__construct_helper<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::type>::value, void>::type std::allocator_traits<pallocator_array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_S_construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(pallocator_array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.04     12.20     0.01                             void __gnu_cxx::new_allocator<CPPStructType*>::destroy<CPPStructType*>(CPPStructType**)
  0.04     12.20     0.01                             std::vector<CPPDeclaration*, std::allocator<CPPDeclaration*> >::operator[](unsigned long)
...
  0.00     12.20     0.00        1     0.00     0.00  void std::_Destroy<CPPScope**, CPPScope*>(CPPScope**, CPPScope**, std::allocator<CPPScope*>&)
  0.00     12.20     0.00        1     0.00     0.00  void std::_Destroy<int*>(int*, int*)
  0.00     12.20     0.00        1     0.00     0.00  void std::_Destroy<int*, int>(int*, int*, std::allocator<int>&)

 %         the percentage of the total running time of the
time       program used by this function.

cumulative a running sum of the number of seconds accounted
 seconds   for by this function and those listed above it.

 self      the number of seconds accounted for by this
seconds    function alone.  This is the major sort for this
           listing.

calls      the number of times this function was invoked, if
           this function is profiled, else blank.

 self      the average number of milliseconds spent in this
ms/call    function per call, if this function is profiled,
	   else blank.

 total     the average number of milliseconds spent in this
ms/call    function and its descendents per call, if this
	   function is profiled, else blank.

name       the name of the function.  This is the minor sort
           for this listing. The index shows the location of
	   the function in the gprof listing. If the index is
	   in parenthesis it shows where it would appear in
	   the gprof listing if it were to be printed.

Copyright (C) 2012-2018 Free Software Foundation, Inc.

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.

		     Call graph (explanation follows)


granularity: each sample hit covers 2 byte(s) for 0.08% of 12.20 seconds

index % time    self  children    called     name
                                                 <spontaneous>
[1]     98.6    0.00   12.03                 main [1]
                0.01   12.02       1/1           CPPPreprocessor::preprocess_file(Filename const&) [2]
                0.00    0.00       3/3           predefine_macro(CPPParser&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [380]
                0.00    0.00       5/6286208     Filename::Filename(char const*) [70]
                0.00    0.00       4/4           DSearchPath::append_directory(Filename const&) [437]
                0.00    0.00       5/18559074     Filename::~Filename() [192]
                0.00    0.00       2/2           std::vector<CPPFile::Source, std::allocator<CPPFile::Source> >::push_back(CPPFile::Source&&) [485]
                0.00    0.00       1/1           preprocess_argv(int&, char**&) [1452]
                0.00    0.00       1/1           CPPPreprocessor::set_verbose(int) [1518]
                0.00    0.00       1/1           CPPScope::write(std::ostream&, int, CPPScope*) const [1548]
-----------------------------------------------
                0.01   12.02       1/1           main [1]
[2]     98.6    0.01   12.02       1         CPPPreprocessor::preprocess_file(Filename const&) [2]
                0.01   11.96  636026/636026      CPPPreprocessor::get_next_token() <cycle 1> [40]
                0.02    0.00  636025/636025      CPPToken::output_code(std::ostream&) const [148]
                0.00    0.02  636025/1460524     CPPToken::operator=(CPPToken const&) [124]
                0.00    0.01  636026/2143064     CPPToken::~CPPToken() [128]
                0.00    0.00       1/1           CPPPreprocessor::init_cpp(CPPFile const&) [275]
                0.00    0.00       1/183         Filename::make_canonical() [261]
                0.00    0.00       1/3143193     CPPFile::CPPFile(Filename const&, Filename const&, CPPFile::Source) [110]
                0.00    0.00       1/12677581     Filename::Filename(Filename const&) [75]
                0.00    0.00       1/6134948     CPPFile::~CPPFile() [228]
                0.00    0.00       1/18559074     Filename::~Filename() [192]
                0.00    0.00  636026/636026      CPPToken::is_eof() const [550]
                0.00    0.00   86290/86290       indent(std::ostream&, int) [618]
                0.00    0.00       1/183         std::set<CPPFile, std::less<CPPFile>, std::allocator<CPPFile> >::find(CPPFile const&) [1073]
                0.00    0.00       1/183         std::set<CPPFile, std::less<CPPFile>, std::allocator<CPPFile> >::end() const [1070]
                0.00    0.00       1/183         std::operator!=(std::_Rb_tree_const_iterator<CPPFile> const&, std::_Rb_tree_const_iterator<CPPFile> const&) [1081]
                0.00    0.00       1/4792        CPPPreprocessor::get_error_count() const [882]
-----------------------------------------------
[3]     98.1    0.01   11.96  636026+1054754 <cycle 1 as a whole> [3]
                0.00    9.45   34252             CPPPreprocessor::expand_manifest(CPPManifest const*, cppyyltype const&) <cycle 1> [4]
                0.00    0.41    3418             CPPPreprocessor::parse_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CPPScope*, CPPScope*, cppyyltype const&) <cycle 1> [36]
                0.00    0.39    4190             CPPPreprocessor::handle_define_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [38]
                0.00    0.36  656886             CPPPreprocessor::get_next_token() <cycle 1> [40]
                0.00    0.32   10823             CPPPreprocessor::process_directive(int) <cycle 1> [42]
                0.01    0.31  252703             CPPPreprocessor::get_identifier(int) <cycle 1> [43]
                0.00    0.23  686347             CPPPreprocessor::internal_get_next_token() <cycle 1> [52]
                0.00    0.16    1373             CPPPreprocessor::handle_if_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [60]
                0.00    0.15    2712             CPPPreprocessor::skip_false_if_block(bool) <cycle 1> [61]
                0.00    0.09    4791             CPPExpressionParser::parse_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CPPPreprocessor const&) <cycle 1> [76]
                0.00    0.08    4791             cppyyparse() <cycle 1> [88]
                0.00    0.01    1572             CPPPreprocessor::handle_ifdef_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [229]
                0.00    0.01    1271             CPPPreprocessor::handle_ifndef_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [234]
                0.00    0.00   20860             cppyylex(cppyystype*, cppyyltype*) <cycle 1> [272]
                0.00    0.00    4791             parse_const_expr(CPPPreprocessor*, CPPScope*, CPPScope*) <cycle 1> [952]
-----------------------------------------------
...

from panda3d.

mingodad commented on May 30, 2024

Ok fair enough !
Thank you again for your great work !

from panda3d.

rdb commented on May 30, 2024

Wow, that's really the vast majority of the time spent handling the manifests map... maybe making a copy of the manifests map every time wasn't such a good idea after all. If my changes introduced a significant performance regression, I might look into this a bit after all.

from panda3d.

mingodad commented on May 30, 2024

My first view on it called my attention that each CPPToken have a copy of struct cppyyltype and the later has a copy of CPPFile I'm not sure that each CPPToken/struct cppyyltype really need it (probably a pointer/index to CPPPreprocessor::_files that also can have a shadow vector storing all CPPPreprocessor::Files (to not have dangling pointers/indexes) and everywhere else only refers to pointer/index instead of copying it every where (the same principle can be applied to other places).

from panda3d.

mingodad commented on May 30, 2024

The same for class cppyystype having an CPPAttributeList maybe a pointer instead could be used.
And of course you've nailed it when making copies everywhere seems to be hurting a bit.

But again as you've said if it's good enough for your use case it's a bit too much spend more time on it.

For me I'm looking at it as an exercise in code analysis/improvement.

from panda3d.

rdb commented on May 30, 2024

I just pushed a change that reduces the time spent parsing sqlite from 14s to 0.7s on my machine. Okay, now it's usable enough. 😅

Most of the gains are from changing the CPPManifest map from an std::map to an std::unordered_map and from using a separate set of ignores rather than copying this map on every expansion.

I didn't see the copying take up a significant time in the profile, but I made a change allowing the compiler to generate more optimized code for CPPFile copying/moving, which may help a little bit. You're right that there are more optimizations possible, but it doesn't seem worth the effort we really see this part take time in the seconds range in a typical run.

from panda3d.

rdb commented on May 30, 2024

Oh, by the way, I also checked in changes to fix the volatile parsing issue, and adding support for __restrict (it's just ignored for now) and __builtin_va_list which is just an opaque built-in type that was expected by my system headers.

from panda3d.

mingodad commented on May 30, 2024

Wow ! 👍
You are really fast an know this codebase !
Thank you again for your great work !

It seems that somehow you've forgot to update cppBison.h with KW_BUILTIN_VA_LIST/KW_RESTRICT added to it:

g++ -DHAVE_IOS_BINARY=1 -DPHAVE_DIRENT_H=1 -DPHAVE_GLOB_H=1 -DPHAVE_LOCKF=1 -DPHAVE_UTIME_H=1 -DBUILDING_DTOOL_DTOOLBASE   -c -O2 -I../dtool/src/cppparser -I../dtool/src/dtoolbase -I../build/include -I../build/cmake/dtool/src/dtoolbase -I../dtool/src/dtoolutil -I../build -std=c++14 -MMD -MP -MF "build/Release/GNU-Linux/_ext/39562175/cppPreprocessor.o.d" -o build/Release/GNU-Linux/_ext/39562175/cppPreprocessor.o ../dtool/src/cppparser/cppPreprocessor.cxx
../dtool/src/cppparser/cppPreprocessor.cxx: In static member function ‘static int CPPPreprocessor::check_keyword(const string&)’:
../dtool/src/cppparser/cppPreprocessor.cxx:2747:43: error: ‘KW_BUILTIN_VA_LIST’ was not declared in this scope
 2747 |   if (name == "__builtin_va_list") return KW_BUILTIN_VA_LIST;
      |                                           ^~~~~~~~~~~~~~~~~~
../dtool/src/cppparser/cppPreprocessor.cxx:2821:36: error: ‘KW_RESTRICT’ was not declared in this scope; did you mean ‘KW_STRUCT’?
 2821 |   if (name == "__restrict") return KW_RESTRICT;
      |                                    ^~~~~~~~~~~
      |                                    KW_STRUCT
../dtool/src/cppparser/cppPreprocessor.cxx:2822:38: error: ‘KW_RESTRICT’ was not declared in this scope; did you mean ‘KW_STRUCT’?
 2822 |   if (name == "__restrict__") return KW_RESTRICT;
      |                                      ^~~~~~~~~~~
      |                                      KW_STRUCT

from panda3d.

mingodad commented on May 30, 2024

Wopps ! My bad ! I need to copy/move cppBison.xxx.prebuild, sorry about that.

from panda3d.

mingodad commented on May 30, 2024

There is a small glitch when I parse <stdio.h> on my machine, somehow the line_number shown bellow get out of sync and several declarations goes on the same line (still valid) and doing some experimenting I could get the output a bit better with changes shown bellow.

------------------- dtool/src/cppparser/cppPreprocessor.cxx -------------------
index f79bcd6ca..d83944da5 100644
@@ -263,6 +263,7 @@ preprocess_file(const Filename &filename) {
     if (token._token == '}') {
       nesting -= 1;
     }
+    //std::cout << "===(" << token._lval.str << ":" << line_number << ":" << token._lloc.first_line << ")===\n";
     if (token._lloc.first_line > line_number) {
       // Token is on a different line, so insert a newline.
       std::cout << "\n";
@@ -279,6 +280,7 @@ preprocess_file(const Filename &filename) {
     next_space = (token._token != '(' && token._token != '~');
     token.output_code(std::cout);
     token = get_next_token();
+    if(token._lloc.first_line < line_number) line_number = token._lloc.last_line;
   }
   std::cout << "\n";

Before:

static void sqlite3HashInit (Hash *);
static void * sqlite3HashInsert (Hash *, const char * pKey, void * pData);
static void * sqlite3HashFind (const Hash *, const char * pKey);
static void sqlite3HashClear (Hash *); typedef long unsigned int size_t; typedef unsigned char __u_char; typedef unsigned short int __u_short; typedef unsigned int __u_int; typedef unsigned long int __u_long; typedef signed char __int8_t; typedef unsigned char __uint8_t; typedef signed short int __int16_t; typedef unsigned short int __uint16_t; typedef signed int __int32_t; typedef unsigned int __uint32_t; typedef signed long long int __int64_t; typedef unsigned long long int __uint64_t; typedef long long int __quad_t; typedef unsigned long long int __u_quad_t; typedef long long int __intmax_t; typedef unsigned long long int __uintmax_t; typedef __u_quad_t __dev_t; typedef unsigned int __uid_t; typedef unsigned int __gid_t; typedef unsigned long int __ino_t; typedef __u_quad_t __ino64_t; typedef unsigned int __mode_t; typedef unsigned int __nlink_t; typedef long int __off_t; typedef __quad_t __off64_t; typedef int __pid_t; typedef struct { int __val [ 2 ]; } __fsid_t; typedef long int __clock_t; typedef unsigned long int __rlim_t; typedef __u_quad_t __rlim64_t; typedef unsigned int __id_t; typedef long int __time_t; typedef unsigned int __useconds_t; typedef long int __suseconds_t; typedef int __daddr_t; typedef int __key_t; typedef int __clockid_t; typedef void * __timer_t; typedef long int __blksize_t; typedef long int __blkcnt_t; typedef __quad_t __blkcnt64_t; typedef unsigned long int __fsblkcnt_t; typedef __u_quad_t __fsblkcnt64_t; typedef unsigned long int __fsfilcnt_t; typedef __u_quad_t __fsfilcnt64_t; typedef int __fsword_t; typedef int __ssize_t; typedef long int __syscall_slong_t; typedef unsigned long int __syscall_ulong_t; typedef __off64_t __loff_t; typedef char * __caddr_t; typedef int __intptr_t; typedef unsigned int __socklen_t; typedef int __sig_atomic_t; struct _IO_FILE; typedef struct _IO_FILE __FILE; struct _IO_FILE; typedef struct _IO_FILE FILE; typedef struct { int __count; union { unsigned int __wch; char __wchb [ 4 ]; } __value; } __mbstate_t; typedef struct { __off_t __pos; __mbstate_t __state; } _G_fpos_t; typedef struct { __off64_t __pos; __mbstate_t __state; } _G_fpos64_t; struct _IO_jump_t; struct _IO_FILE; typedef void _IO_lock_t; struct _IO_marker { struct _IO_marker * _next; struct _IO_FILE * _sbuf; int _pos; }; enum __codecvt_result { __codecvt_ok, __codecvt_partial, __codecvt_error, __codecvt_noconv }; struct _IO_FILE { int _flags; char * _IO_read_ptr; char * _IO_read_end; char * _IO_read_base; char * _IO_write_base; char * _IO_write_ptr; char * _IO_write_end; char * _IO_buf_base; char * _IO_buf_end; char * _IO_save_base; char * _IO_backup_base; char * _IO_save_end; struct _IO_marker * _markers; struct _IO_FILE * _chain; int _fileno; int _flags2; __off_t _old_offset; unsigned short _cur_column; signed char _vtable_offset; char _shortbuf [ 1 ]; _IO_lock_t * _lock; __off64_t _offset; void * __pad1; void * __pad2; void * __pad3; void * __pad4; size_t __pad5; int _mode; char _unused2 [ 15 * sizeof (int) - 4 * sizeof (void *) - sizeof (size_t) ]; }; typedef struct _IO_FILE _IO_FILE; struct _IO_FILE_plus; extern struct _IO_FILE_plus _IO_2_1_stdin_; extern struct _IO_FILE_plus _IO_2_1_stdout_; extern struct _IO_FILE_plus _IO_2_1_stderr_; typedef __ssize_t __io_read_fn (void * __cookie, char * __buf, size_t __nbytes); typedef __ssize_t __io_write_fn (void * __cookie, const char * __buf, size_t __n); typedef int __io_seek_fn (void * __cookie, __off64_t * __pos, int __w); typedef int __io_close_fn (void * __cookie); extern int __underflow (_IO_FILE *); extern int __uflow (_IO_FILE *); extern int __overflow (_IO_FILE *, int); extern int _IO_getc (_IO_FILE * __fp); extern int _IO_putc (int __c, _IO_FILE * __fp); extern int _IO_feof (_IO_FILE * __fp); extern int _IO_ferror (_IO_FILE * __fp); extern int _IO_peekc_locked (_IO_FILE * __fp); extern void _IO_flockfile (_IO_FILE *); extern void _IO_funlockfile (_IO_FILE *); extern int _IO_ftrylockfile (_IO_FILE *); extern int _IO_vfscanf (_IO_FILE *, const char *, __gnuc_va_list, int *); extern int _IO_vfprintf (_IO_FILE *, const char *, __gnuc_va_list); extern __ssize_t _IO_padn (_IO_FILE *, int, __ssize_t); extern size_t _IO_sgetn (_IO_FILE *, void *, size_t); extern __off64_t _IO_seekoff (_IO_FILE *, __off64_t, int, int); extern __off64_t _IO_seekpos (_IO_FILE *, __off64_t, int); extern void _IO_free_backup_area (_IO_FILE *); typedef __off64_t off_t; typedef __off64_t off64_t; typedef _G_fpos64_t fpos_t; typedef _G_fpos64_t fpos64_t; extern struct _IO_FILE * stdin; extern struct _IO_FILE * stdout; extern struct _IO_FILE * stderr; extern int remove (const char * __filename); extern int rename (const char * __old, const char * __new); extern FILE * tmpfile64 (void); extern char * tmpnam (char * __s); extern char * tempnam (const char * __dir, const char * __pfx); extern int fclose (FILE * __stream); extern int fflush (FILE * __stream); extern FILE * fopen64 (const char * __filename, const char * __modes); extern FILE * freopen64 (const char * __filename, const char * __modes, FILE * __stream); extern FILE * fdopen (int __fd, const char * __modes); extern void setbuf (FILE * __stream, char * __buf); extern int setvbuf (FILE * __stream, char * __buf, int __modes, size_t __n); extern int fprintf (FILE * __stream, const char * __format, ...); extern int printf (const char * __format, ...); extern int sprintf (char * __s, const char * __format, ...); extern int vfprintf (FILE * __s, const char * __format, __gnuc_va_list __arg); extern int vprintf (const char * __format, __gnuc_va_list __arg); extern int vsprintf (char * __s, const char * __format, __gnuc_va_list __arg); extern int snprintf (char * __s, size_t __maxlen, const char * __format, ...); extern int vsnprintf (char * __s, size_t __maxlen, const char * __format, __gnuc_va_list __arg); extern int fscanf (FILE * __stream, const char * __format, ...); extern int scanf (const char * __format, ...); extern int sscanf (const char * __s, const char * __format, ...); extern int __isoc99_fscanf (FILE * __stream, const char * __format, ...); extern int __isoc99_scanf (const char * __format, ...); extern int __isoc99_sscanf (const char * __s, const char * __format, ...); extern int vfscanf (FILE * __s, const char * __format, __gnuc_va_list __arg); extern int vscanf (const char * __format, __gnuc_va_list __arg); extern int vsscanf (const char * __s, const char * __format, __gnuc_va_list __arg); extern int __isoc99_vfscanf (FILE * __s, const char * __format, __gnuc_va_list __arg); extern int __isoc99_vscanf (const char * __format, __gnuc_va_list __arg); extern int __isoc99_vsscanf (const char * __s, const char * __format, __gnuc_va_list __arg); extern int fgetc (FILE * __stream); extern int getc (FILE * __stream); extern int getchar (void); extern int getc_unlocked (FILE * __stream); extern int getchar_unlocked (void); extern int fputc (int __c, FILE * __stream); extern int putc (int __c, FILE * __stream); extern int putchar (int __c); extern int putc_unlocked (int __c, FILE * __stream); extern int putchar_unlocked (int __c); extern char * fgets (char * __s, int __n, FILE * __stream); extern char * gets (char * __s); extern int fputs (const char * __s, FILE * __stream); extern int puts (const char * __s); extern int ungetc (int __c, FILE * __stream); extern size_t fread (void * __ptr, size_t __size, size_t __n, FILE * __stream); extern size_t fwrite (const void * __ptr, size_t __size, size_t __n, FILE * __s); extern int fseek (FILE * __stream, long int __off, int __whence); extern long int ftell (FILE * __stream); extern void rewind (FILE * __stream); extern int fseeko64 (FILE * __stream, __off64_t __off, int __whence); extern __off64_t ftello64 (FILE * __stream); extern int fgetpos64 (FILE * __stream, fpos64_t * __pos); extern int fsetpos64 (FILE * __stream, const fpos64_t * __pos); extern void clearerr (FILE * __stream); extern int feof (FILE * __stream); extern int ferror (FILE * __stream); extern void perror (const char * __s); extern int fileno (FILE * __stream); extern FILE * popen (const char * __command, const char * __modes); extern int pclose (FILE * __stream); extern char * ctermid (char * __s); extern void flockfile (FILE * __stream); extern int ftrylockfile (FILE * __stream); extern void funlockfile (FILE * __stream); typedef int pccwchar_t; typedef enum { P_ALL, P_PID, P_PGID } idtype_t; typedef float _Float32; typedef double _Float64; typedef double _Float32x; typedef long double _Float64x; typedef struct { int quot; int rem; } div_t; typedef struct { long int quot; long int rem; } ldiv_t; typedef struct { long long int quot; long long int rem; } lldiv_t; extern size_t __ctype_get_mb_cur_max (void); extern double atof (const char * __nptr); extern int atoi (const char * __nptr); extern long int atol (const char * __nptr); extern long long int atoll (const char * __nptr); extern double strtod (const char * __nptr, char * * __endptr); extern float strtof (const char * __nptr, char * * __endptr); extern long double strtold (const char * __nptr, char * * __endptr); extern long int strtol (const char * __nptr, char * * __endptr, int __base); extern unsigned long int strtoul (const char * __nptr, char * * __endptr, int __base); extern long long int strtoll (const char * __nptr, char * * __endptr, int __base); extern unsigned long long int strtoull (const char * __nptr, char * * __endptr, int __base); extern char * l64a (long int __n); extern long int a64l (const char * __s); typedef __loff_t loff_t; typedef __ino64_t ino_t; typedef __ino64_t ino64_t; typedef __dev_t dev_t; typedef __gid_t gid_t; typedef __mode_t mode_t; typedef __nlink_t nlink_t; typedef __uid_t uid_t; typedef __pid_t pid_t; typedef __id_t id_t; typedef __ssize_t ssize_t; typedef __key_t key_t; typedef __clock_t clock_t; typedef __clockid_t clockid_t; typedef __time_t time_t; typedef __timer_t timer_t; typedef __useconds_t useconds_t; typedef __suseconds_t suseconds_t; typedef __int8_t int8_t; typedef __int16_t int16_t; typedef __int32_t int32_t; typedef __int64_t int64_t; typedef unsigned char u_int8_t; typedef unsigned short int u_int16_t; typedef unsigned int u_int32_t; typedef unsigned long long int u_int64_t; typedef int register_t; typedef __blksize_t blksize_t; typedef __blkcnt64_t blkcnt_t; typedef __fsblkcnt64_t fsblkcnt_t; typedef __fsfilcnt64_t fsfilcnt_t; typedef __blkcnt64_t blkcnt64_t; typedef __fsblkcnt64_t fsblkcnt64_t; typedef __fsfilcnt64_t fsfilcnt64_t; struct __pthread_rwlock_arch_t { unsigned int __readers; unsigned int __writers; unsigned int __wrphase_futex; unsigned int __writers_futex; unsigned int __pad3; unsigned int __pad4; unsigned char __flags; unsigned char __shared; signed char __rwelision; unsigned char __pad2; int __cur_writer; }; typedef struct __pthread_internal_slist { struct __pthread_internal_slist * __next; } __pthread_slist_t; struct __pthread_mutex_s { int __lock; unsigned int __count; int __owner; int __kind; unsigned int __nusers; union { struct { short __espins; short __eelision; } __elision_data; __pthread_slist_t __list; }; }; struct __pthread_cond_s { union { unsigned long long int __wseq; struct { unsigned int __low; unsigned int __high; } __wseq32; }; union { unsigned long long int __g1_start; struct { unsigned int __low; unsigned int __high; } __g1_start32; }; unsigned int __g_refs [ 2 ]; unsigned int __g_size [ 2 ]; unsigned int __g1_orig_size; unsigned int __wrefs; unsigned int __g_signals [ 2 ]; }; typedef unsigned long int pthread_t; typedef union { char __size [ 4 ]; int __align; } pthread_mutexattr_t; typedef union { char __size [ 4 ]; int __align; } pthread_condattr_t; typedef unsigned int pthread_key_t; typedef int pthread_once_t; union pthread_attr_t { char __size [ 36 ]; long int __align; }; typedef union pthread_attr_t pthread_attr_t; typedef union { struct __pthread_mutex_s __data; char __size [ 24 ]; long int __align; } pthread_mutex_t; typedef union { struct __pthread_cond_s __data; char __size [ 48 ]; long long int __align; } pthread_cond_t; typedef union { struct __pthread_rwlock_arch_t __data; char __size [ 32 ]; long int __align; } pthread_rwlock_t; typedef union { char __size [ 8 ]; long int __align; } pthread_rwlockattr_t; typedef volatile int pthread_spinlock_t; typedef union { char __size [ 20 ]; long int __align; } pthread_barrier_t; typedef union { char __size [ 4 ]; int __align; } pthread_barrierattr_t; extern long int random (void); extern void srandom (unsigned int __seed); extern char * initstate (unsigned int __seed, char * __statebuf, size_t __statelen); extern char * setstate (char * __statebuf); extern int rand (void); extern void srand (unsigned int __seed); extern int rand_r (unsigned int * __seed); extern double drand48 (void); extern double erand48 (unsigned short int __xsubi [ 3 ]); extern long int lrand48 (void); extern long int nrand48 (unsigned short int __xsubi [ 3 ]); extern long int mrand48 (void); extern long int jrand48 (unsigned short int __xsubi [ 3 ]); extern void srand48 (long int __seedval); extern unsigned short int * seed48 (unsigned short int __seed16v [ 3 ]); extern void lcong48 (unsigned short int __param [ 7 ]); extern void * malloc (size_t __size); extern void * calloc (size_t __nmemb, size_t __size); extern void * realloc (void * __ptr, size_t __size); extern void free (void * __ptr); extern int posix_memalign (void * * __memptr, size_t __alignment, size_t __size); extern void abort (void); extern int atexit (void (* __func) (void)); extern void exit (int __status); extern void _Exit (int __status); extern char * getenv (const char * __name); extern int putenv (char * __string); extern int setenv (const char * __name, const char * __value, int __replace); extern int unsetenv (const char * __name); extern char * mktemp (char * __template); extern int mkstemp64 (char * __template); extern int system (const char * __command); extern char * realpath (const char * __name, char * __resolved); typedef int (* __compar_fn_t) (const void *, const void *); extern void * bsearch (const void * __key, const void * __base, size_t __nmemb, size_t __size, __compar_fn_t __compar); extern void qsort (void * __base, size_t __nmemb, size_t __size, __compar_fn_t __compar); extern int abs (int __x); extern long int labs (long int __x); extern long long int llabs (long long int __x); extern div_t div (int __numer, int __denom); extern ldiv_t ldiv (long int __numer, long int __denom); extern lldiv_t lldiv (long long int __numer, long long int __denom); extern char * ecvt (double __value, int __ndigit, int * __decpt, int * __sign); extern char * fcvt (double __value, int __ndigit, int * __decpt, int * __sign); extern char * gcvt (double __value, int __ndigit, char * __buf); extern int mblen (const char * __s, size_t __n); extern int mbtowc (pccwchar_t * __pwc, const char * __s, size_t __n); extern int wctomb (char * __s, pccwchar_t __wchar); extern size_t mbstowcs (pccwchar_t * __pwcs, const char * __s, size_t __n); extern size_t wcstombs (char * __s, const pccwchar_t * __pwcs, size_t __n); extern int getsubopt (char * * __optionp, char * const * __tokens, char * * __valuep); extern void setkey (const char * __key); extern int posix_openpt (int __oflag); extern int grantpt (int __fd); extern int unlockpt (int __fd); extern char * ptsname (int __fd); extern void * memcpy (void * __dest, const void * __src, size_t __n); extern void * memmove (void * __dest, const void * __src, size_t __n); extern void * memccpy (void * __dest, const void * __src, int __c, size_t __n); extern void * memset (void * __s, int __c, size_t __n); extern int memcmp (const void * __s1, const void * __s2, size_t __n); extern void * memchr (const void * __s, int __c, size_t __n); extern char * strcpy (char * __dest, const char * __src); extern char * strncpy (char * __dest, const char * __src, size_t __n); extern char * strcat (char * __dest, const char * __src); extern char * strncat (char * __dest, const char * __src, size_t __n); extern int strcmp (const char * __s1, const char * __s2); extern int strncmp (const char * __s1, const char * __s2, size_t __n); extern int strcoll (const char * __s1, const char * __s2); extern size_t strxfrm (char * __dest, const char * __src, size_t __n); extern char * strdup (const char * __s); extern char * strchr (const char * __s, int __c); extern char * strrchr (const char * __s, int __c); extern size_t strcspn (const char * __s, const char * __reject); extern size_t strspn (const char * __s, const char * __accept); extern char * strpbrk (const char * __s, const char * __accept); extern char * strstr (const char * __haystack, const char * __needle); extern char * strtok (char * __s, const char * __delim); extern char * __strtok_r (char * __s, const char * __delim, char * * __save_ptr); extern char * strtok_r (char * __s, const char * __delim, char * * __save_ptr); extern size_t strlen (const char * __s); extern char * strerror (int __errnum); extern int __xpg_strerror_r (int __errnum, char * __buf, size_t __buflen); typedef long int ptrdiff_t;
typedef sqlite_int64 i64;
typedef sqlite_uint64 u64;
...
typedef struct { int __count;
union { unsigned int __wch;
char __wchb [ 4 ];
} __value;
} __mbstate_t;

After:

static void sqlite3HashInit (Hash *);
static void * sqlite3HashInsert (Hash *, const char * pKey, void * pData);
static void * sqlite3HashFind (const Hash *, const char * pKey);
static void sqlite3HashClear (Hash *);
typedef long unsigned int size_t;
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
...
typedef struct
{
  int __count;
  union
  {
    unsigned int __wch;
    char __wchb [ 4 ];
  } __value;
} __mbstate_t;

Maybe you still have it fresh in your mind and know a better fix ?

from panda3d.

rdb commented on May 30, 2024

I think I must have a different stdio.h, since mine doesn't show the same issue. Maybe we should just use last_line instead of first_line altogether in that method, or does that introduce different problems?

It's really just a hack to make the output look readable, since the token stream doesn't have newlines (or any whitespace for that matter).

from panda3d.

mingodad commented on May 30, 2024

If I understood you correctly you mean:

------------------- dtool/src/cppparser/cppPreprocessor.cxx -------------------
index f79bcd6ca..2e1531a8b 100644
@@ -263,7 +263,8 @@ preprocess_file(const Filename &filename) {
     if (token._token == '}') {
       nesting -= 1;
     }
-    if (token._lloc.first_line > line_number) {
+    if (token._lloc.last_line > line_number) {
       // Token is on a different line, so insert a newline.
       std::cout << "\n";
       line_number = token._lloc.first_line;
@@ -279,6 +280,7 @@ preprocess_file(const Filename &filename) {
     next_space = (token._token != '(' && token._token != '~');
     token.output_code(std::cout);
     token = get_next_token();
   }
   std::cout << "\n";

With the above I'm getting the same output as Before here #1638 (comment) , so it doesn't fix anything.

from panda3d.

mingodad commented on May 30, 2024

The thing is <stdio.h> includes <bits/types.h> and in the later everything goes on the same line, but if I parse <bits/types.h> there is no problem, it's like line_number get updated incrementally then token = get_next_token(); returns to the previous include to continue from there (and it's in a line_number smaller than the previous included file but somehow there is a missing check to fix it).

cppparser-nb -v -E -D__PANDA_CC__ -Dwchar_t=pccwchar_t -S/usr/include -S/usr/lib/gcc/x86_64-linux-gnu/9/include /usr/include/stdio.h > /dev/null
Predefining __PANDA_CC__ as 
Predefining wchar_t as pccwchar_t
Reading /usr/include/stdio.h
  Reading /usr/include/x86_64-linux-gnu/bits/libc-header-start.h
    Reading /usr/include/features.h
      Reading /usr/include/stdc-predef.h
      Reading /usr/include/x86_64-linux-gnu/sys/cdefs.h
        Reading /usr/include/x86_64-linux-gnu/bits/wordsize.h
        Reading /usr/include/x86_64-linux-gnu/bits/long-double.h
      Reading /usr/include/x86_64-linux-gnu/gnu/stubs.h
        Reading /usr/include/x86_64-linux-gnu/gnu/stubs-32.h
  Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
  Reading /usr/include/x86_64-linux-gnu/bits/types.h
    Reading /usr/include/features.h
    Reading /usr/include/x86_64-linux-gnu/bits/wordsize.h
    Reading /usr/include/x86_64-linux-gnu/bits/typesizes.h
  Reading /usr/include/x86_64-linux-gnu/bits/types/__FILE.h
  Reading /usr/include/x86_64-linux-gnu/bits/types/FILE.h
  Reading /usr/include/x86_64-linux-gnu/bits/libio.h
    Reading /usr/include/x86_64-linux-gnu/bits/_G_config.h
      Reading /usr/include/x86_64-linux-gnu/bits/types.h
      Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h
      Reading /usr/include/x86_64-linux-gnu/bits/types/__mbstate_t.h
    Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stdarg.h
  Reading /usr/lib/gcc/x86_64-linux-gnu/9/include/stdarg.h
  Reading /usr/include/x86_64-linux-gnu/bits/stdio_lim.h
  Reading /usr/include/x86_64-linux-gnu/bits/sys_errlist.h
Finished parsing.

from panda3d.

rdb commented on May 30, 2024

Oh, is it because its in a different file? What if we increment the line number every time the file changes?

Try this:

diff --git a/dtool/src/cppparser/cppPreprocessor.cxx b/dtool/src/cppparser/cppPreprocessor.cxx
index 9c9967bc78..a0a655fd1e 100644
--- a/dtool/src/cppparser/cppPreprocessor.cxx
+++ b/dtool/src/cppparser/cppPreprocessor.cxx
@@ -385,14 +385,17 @@ preprocess_file(const Filename &filename) {
   int nesting = 0;
   bool next_space = false;
   CPPToken token = get_next_token();
+  CPPFile cur_file = file;
   while (!token.is_eof()) {
     if (token._token == '}') {
       nesting -= 1;
     }
-    if (token._lloc.first_line > line_number) {
+    if (token._lloc.first_line > line_number ||
+        token._lloc.file != cur_file) {
       // Token is on a different line, so insert a newline.
       std::cout << "\n";
       line_number = token._lloc.first_line;
+      cur_file = token._lloc.file;
       indent(std::cout, nesting * 2);
     }
     else if (next_space && token._token != ';' && token._token != ':' && token._token != ',' && token._token != ')') {

from panda3d.

mingodad commented on May 30, 2024

That fix the problem !
Thank you again for your great work !
You as a father knows your child very well !

from panda3d.

mingodad commented on May 30, 2024

Maybe there is another way to do it without making so many copies of CPPFile ?

from panda3d.

rdb commented on May 30, 2024

Did the change slow down the parse appreciably?

I think CPPFile could be turned into a wrapper around a pointer, but I'm only going to spend time on this if it is a measurable problem.

from panda3d.

mingodad commented on May 30, 2024

Well it works for now, maybe when another revision is needed .
No it doesn't slow down in any perceptible way when parsing sqlite3.c.
Don't worry about it.
Thank you again !

from panda3d.

rdb commented on May 30, 2024

This version of the patch should work without copying CPPFile for every line.

diff --git a/dtool/src/cppparser/cppPreprocessor.cxx b/dtool/src/cppparser/cppPreprocessor.cxx
index 9c9967bc78..00185487a2 100644
--- a/dtool/src/cppparser/cppPreprocessor.cxx
+++ b/dtool/src/cppparser/cppPreprocessor.cxx
@@ -385,6 +385,7 @@ preprocess_file(const Filename &filename) {
   int nesting = 0;
   bool next_space = false;
   CPPToken token = get_next_token();
+  CPPFile cur_file = token._lloc.file;
   while (!token.is_eof()) {
     if (token._token == '}') {
       nesting -= 1;
@@ -405,6 +406,12 @@ preprocess_file(const Filename &filename) {
     next_space = (token._token != '(' && token._token != '~');
     token.output_code(std::cout);
     token = get_next_token();
+
+    if (token._lloc.file != cur_file) {
+      // Switched to a new file, reset the line number.
+      cur_file = token._lloc.file;
+      line_number = 0;
+    }
   }
   std::cout << "\n";

I will commit it shortly.

from panda3d.

mingodad commented on May 30, 2024

No, with that change almost everything goes in a new line.

from panda3d.

mingodad commented on May 30, 2024

Only for curiosity this is how the profile looks like (parsing sqlite3.c like before):

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls   s/call   s/call  name    
  7.32      0.09     0.09 10910014     0.00     0.00  CPPPreprocessor::get()
  4.88      0.15     0.06 12677559     0.00     0.00  Filename::Filename(Filename const&)
  4.47      0.21     0.06 17301879     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::empty() const
  4.07      0.26     0.05 10905340     0.00     0.00  CPPPreprocessor::InputFile::get()
  4.07      0.31     0.05  6286196     0.00     0.00  Filename::operator=(char const*)
  3.25      0.35     0.04     2987     0.00     0.00  PandaFileStreamBuf::decode_newlines(char*, unsigned long, char const*, unsigned long)
  2.85      0.38     0.04 23612344     0.00     0.00  bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*)
  2.44      0.41     0.03 14380058     0.00     0.00  __gnu_cxx::__aligned_membuf<CPPPreprocessor::InputFile>::_M_addr()
  2.44      0.44     0.03 14380058     0.00     0.00  std::_List_node<CPPPreprocessor::InputFile>::_M_valptr()
  2.44      0.47     0.03 14287297     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::end()
  2.44      0.50     0.03 14273675     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::operator--()
  2.44      0.53     0.03  2405919     0.00     0.00  cppyyltype::cppyyltype()
  2.44      0.56     0.03   252702     0.00     0.00  CPPPreprocessor::get_identifier(int)
  2.44      0.59     0.03   218251     0.00     0.00  CPPPreprocessor::check_keyword(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  1.63      0.61     0.02 18559042     0.00     0.00  Filename::~Filename()
  1.63      0.63     0.02  6288051     0.00     0.00  Filename::locate_basename()
  1.63      0.65     0.02  5934413     0.00     0.00  Filename::operator=(Filename const&)
  1.63      0.67     0.02  4601648     0.00     0.00  void std::_Destroy<CPPAttributeList::Attribute*>(CPPAttributeList::Attribute*, CPPAttributeList::Attribute*)
  1.63      0.69     0.02  3135980     0.00     0.00  __gnu_cxx::__normal_iterator<CPPAttributeList::Attribute const*, std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > >::__normal_iterator(CPPAttributeList::Attribute const* const&)
  1.63      0.71     0.02  2369985     0.00     0.00  cppyyltype::cppyyltype(cppyyltype const&)
  1.63      0.73     0.02   277168     0.00     0.00  std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long) const
  1.63      0.75     0.02    36500     0.00     0.00  CPPPreprocessor::skip_c_comment(int)
  1.63      0.77     0.02     4791     0.00     0.00  cppyyparse()
  1.63      0.79     0.02     2712     0.00     0.00  CPPPreprocessor::skip_false_if_block(bool)
  1.22      0.81     0.02 14301608     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::operator*() const
  0.81      0.82     0.01 14415345     0.00     0.00  std::_List_iterator<CPPPreprocessor::InputFile>::_List_iterator(std::__detail::_List_node_base*)
  0.81      0.83     0.01 10958938     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::back()
  0.81      0.84     0.01  6290090     0.00     0.00  Filename::get_pattern() const
  0.81      0.85     0.01  6134938     0.00     0.00  CPPFile::~CPPFile()
  0.81      0.86     0.01  4601648     0.00     0.00  void std::_Destroy_aux<false>::__destroy<CPPAttributeList::Attribute*>(CPPAttributeList::Attribute*, CPPAttributeList::Attribute*)
  0.81      0.87     0.01  4601648     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_impl::~_Vector_impl()
  0.81      0.88     0.01  4601648     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::~_Vector_base()
  0.81      0.89     0.01  4601648     0.00     0.00  std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::~vector()
  0.81      0.90     0.01  3763466     0.00     0.00  CPPPreprocessor::skip_comment(int)
  0.81      0.91     0.01  3309574     0.00     0.00  std::reverse_iterator<std::_List_iterator<CPPPreprocessor::InputFile> >::reverse_iterator(std::_List_iterator<CPPPreprocessor::InputFile>)
  0.81      0.92     0.01  3268919     0.00     0.00  CPPPreprocessor::InputFile::peek()
  0.81      0.93     0.01  3254883     0.00     0.00  CPPPreprocessor::peek()
  0.81      0.94     0.01  3195028     0.00     0.00  CPPFile::CPPFile(CPPFile const&)
  0.81      0.95     0.01  3164368     0.00     0.00  __gnu_cxx::__aligned_membuf<CPPPreprocessor::InputFile>::_M_addr() const
  0.81      0.96     0.01  3138217     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::end() const
  0.81      0.97     0.01  3120171     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_impl::_Vector_impl()
  0.81      0.98     0.01  3120171     0.00     0.00  std::_Vector_base<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::_Vector_base()
  0.81      0.99     0.01  3093085     0.00     0.00  std::__cxx11::list<CPPPreprocessor::InputFile, std::allocator<CPPPreprocessor::InputFile> >::back() const
  0.81      1.00     0.01  3093085     0.00     0.00  std::_List_const_iterator<CPPPreprocessor::InputFile>::operator--()
  0.81      1.01     0.01  2143061     0.00     0.00  CPPToken::~CPPToken()
  0.81      1.02     0.01  1481407     0.00     0.00  std::allocator_traits<std::allocator<CPPAttributeList::Attribute> >::select_on_container_copy_construction(std::allocator<CPPAttributeList::Attribute> const&)
  0.81      1.03     0.01  1481407     0.00     0.00  std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> >::vector(std::vector<CPPAttributeList::Attribute, std::allocator<CPPAttributeList::Attribute> > const&)
  0.81      1.04     0.01   815034     0.00     0.00  CPPToken::CPPToken(int, cppyyltype const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyystype const&)
  0.81      1.05     0.01   636084     0.00     0.00  bool std::operator!=<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.81      1.06     0.01   636024     0.00     0.00  CPPToken::output_code(std::ostream&) const
  0.81      1.07     0.01   541643     0.00     0.00  std::__detail::_Mod_range_hashing::operator()(unsigned long, unsigned long) const
  0.81      1.08     0.01   396700     0.00     0.00  unsigned long const& std::min<unsigned long>(unsigned long const&, unsigned long const&)
  0.81      1.09     0.01   335226     0.00     0.00  std::allocator_traits<std::allocator<CPPNameComponent> >::max_size(std::allocator<CPPNameComponent> const&)
  0.81      1.10     0.01   335226     0.00     0.00  std::vector<CPPNameComponent, std::allocator<CPPNameComponent> >::_S_do_relocate(CPPNameComponent*, CPPNameComponent*, CPPNameComponent*, std::allocator<CPPNameComponent>&, std::integral_constant<bool, true>)
  0.81      1.11     0.01   253684     0.00     0.00  std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, true>::_M_next() const
  0.81      1.12     0.01   252848     0.00     0.00  std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, CPPManifest*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  0.81      1.13     0.01   234392     0.00     0.00  std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<CPPManifest const*, false> >, true>::_S_get(std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<CPPManifest const*, false> >, true>&)
  0.81      1.14     0.01   213423     0.00     0.00  __gnu_cxx::__normal_iterator<CPPManifest::ExpansionNode const*, std::vector<CPPManifest::ExpansionNode, std::allocator<CPPManifest::ExpansionNode> > >::__normal_iterator(CPPManifest::ExpansionNode const* const&)
  0.81      1.15     0.01   169511     0.00     0.00  CPPIdentifier::get_scope(CPPScope*, CPPScope*, CPPPreprocessor*) const
  0.81      1.16     0.01   167613     0.00     0.00  std::_Vector_base<CPPNameComponent, std::allocator<CPPNameComponent> >::_Vector_base()
  0.81      1.17     0.01    85093     0.00     0.00  std::operator!=(std::_List_iterator<CPPPreprocessor::InputFile> const&, std::_List_iterator<CPPPreprocessor::InputFile> const&)
  0.81      1.18     0.01    10823     0.00     0.00  CPPPreprocessor::should_ignore_preprocessor() const
  0.81      1.19     0.01                             std::allocator_traits<std::allocator<std::_Rb_tree_node<Filename> > >::deallocate(std::allocator<std::_Rb_tree_node<Filename> >&, std::_Rb_tree_node<Filename>*, unsigned long)
  0.81      1.20     0.01                             void std::vector<CPPClosureType::Capture, std::allocator<CPPClosureType::Capture> >::_M_realloc_insert<CPPClosureType::Capture const&>(__gnu_cxx::__normal_iterator<CPPClosureType::Capture*, std::vector<CPPClosureType::Capture, std::allocator<CPPClosureType::Capture> > >, CPPClosureType::Capture const&)
  0.41      1.20     0.01   675297     0.00     0.00  std::char_traits<char>::compare(char const*, char const*, unsigned long)
  0.41      1.21     0.01   335226     0.00     0.00  CPPNameComponent* std::__relocate_a<CPPNameComponent*, CPPNameComponent*, std::allocator<CPPNameComponent> >(CPPNameComponent*, CPPNameComponent*, CPPNameComponent*, std::allocator<CPPNameComponent>&)
...
  0.00      1.23     0.00        1     0.00     0.00  void std::_Destroy<int*, int>(int*, int*, std::allocator<int>&)

 %         the percentage of the total running time of the
time       program used by this function.

cumulative a running sum of the number of seconds accounted
 seconds   for by this function and those listed above it.

 self      the number of seconds accounted for by this
seconds    function alone.  This is the major sort for this
           listing.

calls      the number of times this function was invoked, if
           this function is profiled, else blank.

 self      the average number of milliseconds spent in this
ms/call    function per call, if this function is profiled,
	   else blank.

 total     the average number of milliseconds spent in this
ms/call    function and its descendents per call, if this
	   function is profiled, else blank.

name       the name of the function.  This is the minor sort
           for this listing. The index shows the location of
	   the function in the gprof listing. If the index is
	   in parenthesis it shows where it would appear in
	   the gprof listing if it were to be printed.

Copyright (C) 2012-2018 Free Software Foundation, Inc.

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.

		     Call graph (explanation follows)


granularity: each sample hit covers 2 byte(s) for 0.81% of 1.23 seconds

index % time    self  children    called     name
                                                 <spontaneous>
[1]     94.3    0.00    1.16                 main [1]
                0.00    1.16       1/1           CPPPreprocessor::preprocess_file(Filename const&) [2]
                0.00    0.00       2/2           predefine_macro(CPPParser&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [343]
                0.00    0.00       4/4           DSearchPath::append_directory(Filename const&) [370]
                0.00    0.00       2/2           std::vector<CPPFile::Source, std::allocator<CPPFile::Source> >::push_back(CPPFile::Source&&) [381]
                0.00    0.00       5/6286196     Filename::Filename(char const*) [14]
                0.00    0.00       5/18559042     Filename::~Filename() [67]
                0.00    0.00       1/1           preprocess_argv(int&, char**&) [1557]
                0.00    0.00       1/1           CPPPreprocessor::set_verbose(int) [1623]
                0.00    0.00       1/1           CPPScope::write(std::ostream&, int, CPPScope*) const [1653]
-----------------------------------------------
                0.00    1.16       1/1           main [1]
[2]     94.3    0.00    1.16       1         CPPPreprocessor::preprocess_file(Filename const&) [2]
                0.07    1.05  636025/636025      CPPPreprocessor::get_next_token() <cycle 1> [6]
                0.00    0.01  636024/636024      CPPFile::operator!=(CPPFile const&) const [85]
                0.00    0.01  636025/2143061     CPPToken::~CPPToken() [32]
                0.01    0.00  636024/636024      CPPToken::output_code(std::ostream&) const [104]
                0.00    0.00  636024/1460522     CPPToken::operator=(CPPToken const&) [124]
                0.00    0.00       1/183         Filename::make_canonical() [263]
                0.00    0.00       1/1           CPPPreprocessor::init_cpp(CPPFile const&) [344]
                0.00    0.00      71/2966933     CPPFile::operator=(CPPFile const&) [71]
                0.00    0.00       1/3195028     CPPFile::CPPFile(CPPFile const&) [36]
                0.00    0.00       1/3143187     CPPFile::CPPFile(Filename const&, Filename const&, CPPFile::Source) [57]
                0.00    0.00       2/6134938     CPPFile::~CPPFile() [59]
                0.00    0.00       1/12677559     Filename::Filename(Filename const&) [19]
                0.00    0.00       1/18559042     Filename::~Filename() [67]
                0.00    0.00  636025/636025      CPPToken::is_eof() const [478]
                0.00    0.00   88159/88159       indent(std::ostream&, int) [565]
                0.00    0.00       1/183         std::set<CPPFile, std::less<CPPFile>, std::allocator<CPPFile> >::find(CPPFile const&) [1154]
                0.00    0.00       1/183         std::set<CPPFile, std::less<CPPFile>, std::allocator<CPPFile> >::end() const [1151]
                0.00    0.00       1/183         std::operator!=(std::_Rb_tree_const_iterator<CPPFile> const&, std::_Rb_tree_const_iterator<CPPFile> const&) [1162]
                0.00    0.00       1/4792        CPPPreprocessor::get_error_count() const [929]
-----------------------------------------------
[3]     90.8    0.07    1.05  636025+1054750 <cycle 1 as a whole> [3]
                0.03    0.24  252702             CPPPreprocessor::get_identifier(int) <cycle 1> [5]
                0.00    0.25  656885             CPPPreprocessor::get_next_token() <cycle 1> [6]
                0.00    0.22  686345             CPPPreprocessor::internal_get_next_token() <cycle 1> [7]
                0.02    0.18    2712             CPPPreprocessor::skip_false_if_block(bool) <cycle 1> [8]
                0.02    0.07    4791             cppyyparse() <cycle 1> [13]
                0.00    0.05   34251             CPPPreprocessor::expand_manifest(CPPManifest const*, cppyyltype const&) <cycle 1> [28]
                0.00    0.02   10823             CPPPreprocessor::process_directive(int) <cycle 1> [62]
                0.00    0.01    3418             CPPPreprocessor::parse_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CPPScope*, CPPScope*, cppyyltype const&) <cycle 1> [131]
                0.00    0.00    4190             CPPPreprocessor::handle_define_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [167]
                0.00    0.00    1373             CPPPreprocessor::handle_if_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [174]
                0.00    0.00    4791             CPPExpressionParser::parse_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CPPPreprocessor const&) <cycle 1> [175]
                0.00    0.00   20860             cppyylex(cppyystype*, cppyyltype*) <cycle 1> [201]
                0.00    0.00    1572             CPPPreprocessor::handle_ifdef_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [244]
                0.00    0.00    1271             CPPPreprocessor::handle_ifndef_directive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cppyyltype const&) <cycle 1> [249]
                0.00    0.00    4791             parse_const_expr(CPPPreprocessor*, CPPScope*, CPPScope*) <cycle 1> [1003]
-----------------------------------------------
...

from panda3d.

rdb commented on May 30, 2024

So get() takes a lot of time (it's just called very often). Maybe the list of files could be a hand-rolled singly-linked list (pointer to the last element with a pointer to the previous one in it) or some other data structure.

Linked lists are usually a bad idea but in this case we just need to optimize for accessing the last element, which they're fine for.

The Filename copy also takes a lot of time, which is due to the CPPFile copying. There's another easy change I can make that allows it to use move semantics more often, which will help a bit.

That said, I'm not going to be tempted to spend much time on this.

from panda3d.

rdb commented on May 30, 2024

Okay, I made some more changes. It seems to be marginally faster now. Perhaps some kind of buffering might help speed up the result of get().

That said, time for me to move on to other tasks... Thanks for all the help!

from panda3d.

mingodad commented on May 30, 2024

Again thank you !

from panda3d.

Cppparser preprocessor do not expand macros with arguments correctly about panda3d HOT 94 CLOSED

Comments (94)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent