goccy / p5-compiler-lexer Goto Github PK

View Code? Open in Web Editor NEW

46.0 9.0 18.0 43.58 MB

Lexical Analyzer for Perl5

License: Other

Perl 91.60% Makefile 0.07% C++ 8.12% XS 0.21%

p5-compiler-lexer's Introduction

NAME

Compiler::Lexer - Lexical Analyzer for Perl5

SYNOPSIS

use Compiler::Lexer;
use Data::Dumper;

my $filename = $ARGV[0];
open my $fh, '<', $filename or die "Cannot open $filename: $!";
my $script = do { local $/; <$fh> };

my $lexer = Compiler::Lexer->new($filename);
my $tokens = $lexer->tokenize($script);
print Dumper $tokens;

my $modules = $lexer->get_used_modules($script);
print Dumper $modules;

METHODS

my $lexer = Compiler::Lexer->new($options);

create new instance. You can create object from $options in hash reference.

options list
- filename
- verbose : includes token of Pod, Comment and WhiteSpace
$lexer->tokenize($script);

get token objects includes parameter of 'name' or 'type' or 'line' and so on. This method requires perl source code in string.
$lexer->set_library_path(['path1', 'path2' ...])

set libraries path for reading recursively. Default paths are @INC.
$lexer->recursive_tokenize($script)

get hash reference like { 'module_nameA' => [], 'module_nameB' => [] ... }. This method requires per source code in string.
$lexer->get_used_modules($script);

get names of used module. This method requires perl source code in string.

AUTHOR

Masaaki Goshima (goccy) <goccy(at)cpan.org>

CONTRIBUTORS

tokuhirom: Tokuhiro Matsuno

LICENSE AND COPYRIGHT

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

p5-compiler-lexer's People

Contributors

Stargazers

Watchers

Forkers

tokuhirom rurban myoan yuki-kimoto moznion dolmen syohex rvandam ncstang doyleyoung y3i12 toddr code-hex zippelmitz gugod karupanerura clayne maxbyz

p5-compiler-lexer's Issues

Cannot recognize Regular expression literal with 'not'

use Compiler::Lexer;
use YAML;

my $tokens = Compiler::Lexer->new->tokenize('not /\d/');
print YAML::Dump($tokens);

First / is recognized as div operator.


---
- !!perl/hash:Compiler::Lexer::Token
  data: not
  has_warnings: 0
  kind: 5
  line: 1
  name: AlphabetNot
  stype: 0
  type: 69
- !!perl/hash:Compiler::Lexer::Token
  data: /
  has_warnings: 0
  kind: 1
  line: 1
  name: Div
  stype: 0
  type: 4
- !!perl/hash:Compiler::Lexer::Token
  data: \
  has_warnings: 0
  kind: 1
  line: 1
  name: Ref
  stype: 0
  type: 10
- !!perl/hash:Compiler::Lexer::Token
  data: d
  has_warnings: 1
  kind: 24
  line: 1
  name: Key
  stype: 0
  type: 122
- !!perl/hash:Compiler::Lexer::Token
  data: /
  has_warnings: 0
  kind: 1
  line: 1
  name: Div
  stype: 0
  type: 4

! /\d/ works well.

Support 5.010_001 style numeric literal

In perl5, 5.010_001 is same as 5.010001. But Compiler::Lexer does not analyze properly.

5.010 as Double and _001 as 'Key'.

SEGV while parsing

gdb backtrace:

#0  0x00007ffff7318425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff731bb8b in __GI_abort () at abort.c:91
#2  0x00007ffff735639e in __libc_message (do_abort=2, fmt=0x7ffff7460008 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
#3  0x00007ffff7360b96 in malloc_printerr (action=3, str=0x7ffff7460118 "double free or corruption (!prev)",
    ptr=<optimized out>) at malloc.c:5018
#4  0x00007ffff62e6af7 in Lexer::clearContext (this=0xe6f0e0) at src/compiler/lexer/Compiler_lexer.cpp:121
#5  0x00007ffff62da250 in XS_Compiler__Lexer_tokenize (cv=<optimized out>) at src/Compiler-Lexer.xs:62
#6  0x000000000049f676 in Perl_pp_entersub () at pp_hot.c:2875
#7  0x0000000000497fc3 in Perl_runops_standard () at run.c:42
#8  0x0000000000439eb8 in S_run_body (oldscope=1) at perl.c:2467
#9  perl_run (my_perl=<optimized out>) at perl.c:2383
#10 0x000000000041da75 in main (argc=2, argv=0x7fffffffdc98, env=0x7fffffffdcb0) at perlmain.c:114

Reproducable code:

#!/usr/bin/env perl
use strict;
use warnings;
use utf8;
use 5.010000;
use autodie;
use Test::More;
use Compiler::Lexer;
use LWP::Simple;

print "Perl: $]\n";
print "Compiler::Lexer: $Compiler::Lexer::VERSION\n";

mirror('https://raw.github.com/perl6/roast/master/S32-io/file-tests.t', 'file-test.t');
mirror('https://raw.github.com/perl6/roast/master/S32-io/path.t', 'path.t');

my @files = qw(
    file-test.t
    path.t
);
for (@files) {
    warn "$_\n";
    my $src = do {
        open my $fh, '<', $_;
        local $/;
        <$fh>;
    };
    Compiler::Lexer->new('-')->tokenize($src);
}

Running result:

% perl f.pl
Perl: 5.018000
Compiler::Lexer: 0.13
file-test.t
path.t
perl: malloc.c:3806: _int_malloc: Assertion `victim->fd_nextsize->bk_nextsize == victim' failed.
zsh: abort      perl f.pl

Div operator has misrecognized as the RegDelim

When we use div operator with any string (yes, I know it is stupid mistake but valid syntax), the operator is recognized as the RegDelim token.

Following is the reproduce code;

Compiler::Lexer->new->tokenize("if ('' + 1  || '' - 1  || '' * 1  || '' / 1) {}");

Do you have any plans to distinguish quoted heredoc from non-quoted heredoc?

Now, following two code are treated the same.

print <<END_QUOTE;
Four score and seven years ago...
END_QUOTE

print <<'END_QUOTE';
Four score and seven years ago...
END_QUOTE

Both of tags of the beginning of here document are discriminated as HereDocumentRawTag.
We cannot know the difference of here document tag is quoted or not.

Do you have any plan to support this?

`x=` still hasn't been recognized

at version 0.19

Code to reproduce is following;

Compiler::Lexer->new->tokenize('$foo x= 3')

Result;

  bless( {
           'data' => '$foo',
           'has_warnings' => 0,
           'kind' => 24,
           'line' => 1,
           'name' => 'GlobalVar',
           'stype' => 0,
           'type' => 193
         }, 'Compiler::Lexer::Token' ),
  bless( {
           'data' => 'x',
           'has_warnings' => 0,
           'kind' => 1,
           'line' => 1,
           'name' => 'StringMul',
           'stype' => 0,
           'type' => 19
         }, 'Compiler::Lexer::Token' ),
  bless( {
           'data' => '=',
           'has_warnings' => 0,
           'kind' => 2,
           'line' => 1,
           'name' => 'Assign',
           'stype' => 0,
           'type' => 65
         }, 'Compiler::Lexer::Token' ),
  bless( {
           'data' => '3',
           'has_warnings' => 0,
           'kind' => 24,
           'line' => 1,
           'name' => 'Int',
           'stype' => 0,
           'type' => 170
         }, 'Compiler::Lexer::Token' )

Segmentation fault occurred when semicolon-less at end of the script

$ echo 'print' | perl -MCompiler::Lexer -MData::Dumper -E 'say Dumper(Compiler::Lexer->new("")->get_used_modules(<>))'

The result is

Segmentation fault

In the case of echo 'print;', it is normally run.

In addition,the code is still correct.

$ echo 'print' | perl -cw
- syntax OK

If `format` is used as key of hash, Compiler::Lexer misdetects

Hi!

Sample code to reproduce;

%hash = ( format => 'baz' );

Bad result

This format is not declaration of Format, but it is not...

bless( {
    data => "format",
    has_warnings => 0,
    kind => 3,
    line => 2,
    name => "FormatDecl",
    stype => 0,
    type => 179
}, 'Compiler::Lexer::Token' )

Expected

bless( {
    data => "format",
    has_warnings => 0,
    kind => 24,
    line => 2,
    name => "Key",
    stype => 0,
    type => 119
}, 'Compiler::Lexer::Token' ),

How?

'self is not of type Compiler::Lexer' error

The test script fails even after applying a tiny fix below:

> perl "-MExtUtils::Command::MM" "-e" "test_harness(1, 'inc', 'blib/lib', 'blib/arch')" t/*.t
t/Lexer.t .. 
1..1
ok 1 - use Compiler::Lexer;
self is not of type Compiler::Lexer at t/Lexer.t line 9.
# Looks like your test exited with 255 just after 1.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 1 subtests passed

--- a/MANIFEST
+++ b/MANIFEST
@@ -1,8 +1,8 @@
 Changes
-Lexer.xs
 Makefile.PL
 MANIFEST
 ppport.h
 README
 t/Lexer.t
-lib/Lexer.pm
+lib/Compiler/Lexer.pm
+src/Compiler-Lexer.xs
diff --git a/t/Lexer.t b/t/Lexer.t
index fc500e2..99ad5c7 100644
--- a/t/Lexer.t
+++ b/t/Lexer.t
@@ -6,7 +6,7 @@ BEGIN { use_ok('Compiler::Lexer') };

 my $name = $0;
 #use modules tests
-print Dumper Lexer::get_used_modules($name, <<'SCRIPT');
+print Dumper Compiler::Lexer::get_used_modules($name, <<'SCRIPT');
 use Test::Module;
 my $hash = { use => "value" };
 $hash->{use};

That kind of naming issue also seems to remain on your p5-Compiler-Tools-CopyPasteDetector.

Some assign operator are not supported

It looks

x=
<<=
>>=

are not supported.

recursive_tokenize('foo => {') segfaults

Found by accident while minimizing another crash case.

Occurs SEGV

Here is a sample code to reproduce:

m/ (?{ _foo() }) /smx;

sub _foo {
    return 'bar';
}

Get "comment" and "pod" token

tokenize() method cannot interpret "comment" and "pod" token.
Considering the origins of this module, this may be obvious. But if they are acquirable, I will think that it is convenient.

How do you think this?

%- is variable name

Compiler::Lexer tokenizes "%-" as Mod, Sub. But it's variable name in perl5.

Does Token.pm not needed?

https://github.com/goccy/p5-Compiler-Lexer/blob/master/lib/Compiler/Lexer/Token.pm
This file is empty. I guess you need to remove this file in repository.

Segfault when there is a combination of newlines and none

It appears that an input string with at least 15 new-lines which is then modified to have no trailing new-line causes the tokenizer to dump core.

use Compiler::Lexer;
my $string = "\n" x 15;
$string = "#";
Compiler::Lexer->new->tokenize($string);

Outputs:

*** glibc detected *** perl: free(): invalid next size (fast): 0x0000000013a82290 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2b7ad4b1e4af]
/lib64/libc.so.6(cfree+0x4b)[0x2b7ad4b227ab]
<...>lib/perl5/x86_64-linux-thread-multi/auto/Compiler/Lexer/Lexer.so(_ZN5Lexer12clearContextEv+0x29)[0x2b7ad88094a9]
<...>lib/perl5/x86_64-linux-thread-multi/auto/Compiler/Lexer/Lexer.so[0x2b7ad8804645]
perl(Perl_pp_entersub+0x58f)[0x48900f]
perl(Perl_runops_standard+0xe)[0x4875fe]
perl(perl_run+0x305)[0x431665]
perl(main+0xcd)[0x41d46d]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b7ad4aca9f4]
perl(sin+0xb1)[0x41d2e9]

My setup:

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:
Platform:
osname=linux, osvers=2.6.18-400.1.1.el5, archname=x86_64-linux-thread-multi
uname='linux 2.6.18-400.1.1.el5 #1 smp thu dec 18 00:59:53 est 2014 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dprefix=.plenv/versions/5.10.1 -de -Dusedevel -Dusethreads -A'eval:scriptdir=.plenv/versions/5.10.1/bin''
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.1.2 20080704 (Red Hat 4.1.2-54)', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64 /usr/local/lib64
libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=libc-2.5.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.5'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_USE_DEVEL
USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
Built under linux
Compiled at Mar 4 2015 13:42:57

'package Foo' cause SEGV

Following code causes segmentation fault.

Compiler::Lexer->new('-')->tokenize('package Foo')

Here is a stack trace.

gdb$ bt
#0  0x00007ffff6aa4818 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff6d19c36 in operator==<char, std::char_traits<char>, std::allocator<char> > (__rhs=0x7ffff6d1e955 "=>", __lhs=...)
    at /usr/include/c++/4.6/bits/basic_string.h:2462
#2  Annotator::annotateKey (this=<optimized out>, ctx=0x7fffffffe0b0, tk=<optimized out>) at src/compiler/lexer/Compiler_annotator.cpp:113
#3  0x00007ffff6d1a10f in Annotator::annotate (this=0x772200, ctx=0x7fffffffe0b0, tk=0x9b5020)
    at src/compiler/lexer/Compiler_annotator.cpp:48
#4  0x00007ffff6d1be75 in Lexer::annotateTokens (this=<optimized out>, ctx=0x7fffffffe0b0) at src/compiler/lexer/Compiler_lexer.cpp:173
#5  0x00007ffff6d1c685 in Lexer::tokenize (this=0x9a0bd0, script=<optimized out>) at src/compiler/lexer/Compiler_lexer.cpp:148
#6  0x00007ffff6d12457 in XS_Compiler__Lexer_tokenize (cv=<optimized out>) at src/Compiler-Lexer.xs:45
#7  0x000000000049f676 in Perl_pp_entersub () at pp_hot.c:2875
#8  0x0000000000497fc3 in Perl_runops_standard () at run.c:42
#9  0x0000000000439eb8 in S_run_body (oldscope=1) at perl.c:2467
#10 perl_run (my_perl=<optimized out>) at perl.c:2383
#11 0x000000000041da75 in main (argc=3, argv=0x7fffffffe4a8, env=0x7fffffffe4c8) at perlmain.c:114

Why does it put a whitespace between tokens for `args`?

at https://github.com/goccy/p5-Compiler-Lexer/blob/master/src/compiler/lexer/Compiler_lexer.cpp#L665

Are any abstruse reasons in here?

Should improve the strategy to determine the size of pool of the TokenManager

This point at issue, Compiler::Lexer cannot allocate tokens that don't have visible character length.

For example;

s///;

If this code is tokenized by Compiler::Lexer, it will occur SEGV. Because this code should be tokenized like so;

s (RegReplace)
/ (RegDelim)
'' (RegReplaceFrom)
/ (RegMiddleDelim)
'' (RegReplaceTo)
/ (RegDelim)
; (SemiColon)

7 tokens.
Now, this lexer assures memory which has length of sizeof(Token) * (strlen(script) + 1). But in this case its length is sizeof(Token) * 6, so short. Because RegReplaceFrom and RegReplaceTo don't have length so they are not counted by strlen(script).

ref:

p5-Compiler-Lexer/src/compiler/lexer/Compiler_lexer.cpp

Line 22 in e68db12

tmgr = new TokenManager(script_size);

https://github.com/goccy/p5-Compiler-Lexer/blob/master/src/compiler/lexer/Compiler_manager.cpp#L9

I think this strategy to determine the size of pool has several problems. So I think it should be improved.
Then I thought some ideas to improve,

Pre-allocate excess of memory from the beginning
Use realloc(3) and enlarge on demand (means when it is lacked)
Not treat empty something as a Token.

1 is really ad-hok and not beautiful, but easy way. 2 might be expensive, so it impair the performance. 3 is simple, but it is inconvenience and breaks backward compatibility.
From my perspective, hybrid method of 1 and 2 is good.

How do you think?

P.S.
However, Compiler::Lexer allocates the memory for white space characters, each characters which are in a string literal, and etc. Means it has redundant memory, so I don't think this matter appear frequently...

RegOpt of `m` has been misrecognized as match operator

Code to reproduce

Compiler::Lexer->new->tokenize(<<'...');
/foo/m;
/bar/;
...

Result

[
  bless( {
    data => "/",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegDelim",
    stype => 0,
    type => 151
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => "foo",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegExp",
    stype => 0,
    type => 186
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => "/",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegDelim",
    stype => 0,
    type => 151
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => "m",
    has_warnings => 0,
    kind => 29,
    line => 1,
    name => "RegOpt",
    stype => 0,
    type => 144
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => ";",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegDelim",
    stype => 0,
    type => 151
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => "\n/bar/",
    has_warnings => 0,
    kind => 24,
    line => 2,
    name => "RegExp",
    stype => 0,
    type => 186
  }, 'Compiler::Lexer::Token' ),
  bless( {
    data => ";",
    has_warnings => 0,
    kind => 24,
    line => 2,
    name => "RegDelim",
    stype => 0,
    type => 151
  }, 'Compiler::Lexer::Token' )
]

The option of regular expression of m has been misrecognized as match operator (even if it is handled as RegOpt).

In this case, this m becomes m;;, and semicolons are treated as RegDelim.
I'm expecting as follows;

m is RegOpt
; is SemiColon, not RegDelim

Some perl5 expressions cause segmentation fault

Hi. I really appreciate your challenges including Compiler::Tools::CopyPasteDetector, gperl, and so on.

I found some sentences cause segmentation faults when I call get_groups_by_syntax_level() by following codes.

use strict;
use warnings;
use Compiler::Lexer;

my $lexer = Compiler::Lexer->new("");
#my $tokens = $lexer->tokenize("sub hoge ($) {}");
my $tokens = $lexer->tokenize("sub hoge { +3-2 }");
$lexer->get_groups_by_syntax_level($$tokens, Compiler::Lexer::SyntaxType::T_Stmt);

I ran this script on perl v5.14.2 built for darwin-2level.
The version of my OS X is 10.8.2.

Cannot recognize rightly the heredoc tag as bareword and lowercase

Compiler::Lexer->new->tokenize("<<end\nend");

This code returns as follows;

bless({
    'data' => '<<',
    'has_warnings' => 0,
    'kind' => 1,
    'line' => 1,
    'name' => 'LeftShift',
    'stype' => 0,
    'type' => 47
}, 'Compiler::Lexer::Token' ),
bless({
    'data' => 'end',
    'has_warnings' => 1,
    'kind' => 24,
    'line' => 2,
    'name' => 'Key',
    'stype' => 0,
    'type' => 119
}, 'Compiler::Lexer::Token' ),
bless({
    'data' => 'end',
    'has_warnings' => 1,
    'kind' => 24,
    'line' => 2,
    'name' => 'Key',
    'stype' => 0,
    'type' => 119
}, 'Compiler::Lexer::Token' )

I expected both of end are HereDocumentRawTag and HereDocumentEnd, however not.

If tag name is CAPITAL CASE, it has no problem. These are HereDocumentRawTag and HereDocumentEnd.

Compiler::Lexer->new->tokenize("<<END\nEND");

How?

`$foo-1` cannot tokenize certainly

Now, It has tokenized as $foo-1 as Var.

I think it should be Var of $foo, - of Sub and 1 as Int.

How?

Will not compile on Solaris 11 with gcc

Hit the following issue compiling on Solaris 11:

cpan[5]> test Compiler::Lexer
Running test for module 'Compiler::Lexer'
Running Build for G/GO/GOCCY/Compiler-Lexer-0.17.tar.gz
Has already been unwrapped into directory /home/jhall/.cpan/build/Compiler-Lexer-0.17-1z7dih

CPAN.pm: Going to build G/GO/GOCCY/Compiler-Lexer-0.17.tar.gz

Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'Compiler-Lexer' version '0.17'
Merging cpanfile prereqs to MYMETA.yml
Merging cpanfile prereqs to MYMETA.json
Building Compiler-Lexer
gcc -Isrc -I/usr/perl5/5.12/lib/i86pc-solaris-64int/CORE -fPIC -Iinclude -Wno-missing-field-initializers -g3 -c -DPTR_IS_LONG -I/usr/gnu/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV -O2 -o src/compiler/util/Compiler_token.o src/compiler/util/Compiler_token.cpp
In file included from src/compiler/util/Compiler_token.cpp:1:0:
include/lexer.hpp: In member function ‘TokenInfo TokenManager::getTokenInfo(const char*)’:
include/lexer.hpp:55:67: error: ‘strlen’ was not declared in this scope
error building src/compiler/util/Compiler_token.o from 'src/compiler/util/Compiler_token.cpp' at /export/code/perl5/lib/site_perl/ExtUtils/CBuilder/Base.pm line 175, line 1.
GOCCY/Compiler-Lexer-0.17.tar.gz
./Build --extra_linker_flags -L/export/code/perl5/lib --extra_linker_flags -L/export/code/perl5/lib -- NOT OK
Running Build test
Can't test without successful make
Failed during this command:
GOCCY/Compiler-Lexer-0.17.tar.gz : make NO

This is my environment:

perlgcc2 -V
Summary of my perl5 (revision 5 version 12 subversion 5) configuration:

Platform:
osname=solaris, osvers=2.11, archname=i86pc-solaris-64int
uname='sunos localhost 5.11 i86pc i386 i86pc'
config_args='-Dprefix=/usr/perl5/5.12 -Dprivlib=/usr/perl5/5.12/lib -Dsitelib=/usr/perl5/site_perl/5.12 -Dvendorprefix=/usr/perl5/5.12 -Dvendorlib=/usr/perl5/vendor_perl/5.12 -Dlibperl=libperl.so -Duseshrplib -Dusedtrace -Duse64bitint -Dcc=cc -Doptimize=-xO4 -Dcf_email=[email protected] -Dcf_by=perl-bugs -Dmyhostname=localhost -Dmksymlinks -O -de'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-DPTR_IS_LONG -I/usr/gnu/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV',
optimize='-O2 ',
cppflags='-DPTR_IS_LONG -I/usr/gnu/include'
ccversion='Sun C 5.10 SunOS_i386 Patch 142363-07 2010/12/09', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/lib -L/usr/ccs/lib -L/lib -L/usr/gnu/lib '
libpth=/usr/lib /usr/ccs/lib /lib /usr/gnu/lib
libs=-lsocket -lnsl -lgdbm -ldb -ldl -lm -lc
perllibs=-lsocket -lnsl -ldl -lm -lc
libc=/lib/libc.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -R /usr/perl5/5.12/lib/i86pc-solaris-64int/CORE'
cccdlflags='-fPIC', lddlflags='-G -L/usr/lib -L/usr/ccs/lib -L/lib -L/usr/gnu/lib'

Characteristics of this binary (from libperl):
Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
PERL_USE_SAFE_PUTENV USE_64_BIT_INT USE_LARGE_FILES
USE_PERLIO USE_PERL_ATOF
Locally applied patches:
7111771 Problem with utility/perl
7125218 Problem with utility/perl
7030196 Problem with utility/perl
15820486 Problem with utility/perl
15880426 Problem with utility/perl
16417744 Problem with utility/perl
16427786 Problem with utility/perl
Built under solaris
Compiled at Dec 3 2013 00:45:56
%ENV:
PERL5LIB="/export/code/perl5/lib/PerlGcc2:/export/code/perl5/lib:/export/code/perl5/lib/site_perl:/export/code/perl5/lib/perl5"
PERL5_OVERRIDE_CONFIG="1"
@inc:
/export/code/perl5/lib/PerlGcc2
/export/code/perl5/lib/i86pc-solaris-64int
/export/code/perl5/lib
/export/code/perl5/lib/site_perl/5.12.5/i86pc-solaris-64int
/export/code/perl5/lib/site_perl/5.12.5
/export/code/perl5/lib/site_perl
/export/code/perl5/lib/perl5/i86pc-solaris-64int
/export/code/perl5/lib/perl5
/usr/perl5/site_perl/5.12/i86pc-solaris-64int
/usr/perl5/site_perl/5.12
/usr/perl5/vendor_perl/5.12/i86pc-solaris-64int
/usr/perl5/vendor_perl/5.12
/usr/perl5/5.12/lib/i86pc-solaris-64int
/usr/perl5/5.12/lib
.

Any suggestions?

Detects empty regex as defined-or

my @chars = split //, $what;

This // should be RegExp. But Compiler::Lexer says:

          bless( {
                   'stype' => 0,
                   'kind' => 1,
                   'name' => 'DefaultOperator',
                   'has_warnings' => 0,
                   'data' => '//',
                   'type' => 57,
                   'line' => 5
                 }, 'Compiler::Lexer::Token' ),

Cannot tokenize SQL::Translator::Parser::DB2::Grammer

Following code does not finish.

#!/usr/bin/env perl
use strict;
use warnings;
use utf8;
use 5.010000;
use autodie;

use Compiler::Lexer;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new();
my $res = $ua->get('http://cpansearch.perl.org/src/FREW/SQL-Translator-0.11016/lib/SQL/Translator/Parser/DB2/Grammar.pm');
$res->is_success or die;

Compiler::Lexer->new('-')->tokenize($res->content);

assert failure from t/recursive_tokenize.t

CentOS release 6.5 with gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC):

./Build test
t/Lexer.t ............... ok
t/format.t .............. ok
t/issue_32.t ............ ok
t/issue_35.t ............ ok
t/issue_38.t ............ ok
t/issue_39.t ............ ok
t/issue_40.t ............ ok
t/issue_42.t ............ ok
t/issue_43.t ............ ok
t/issue_44.t ............ ok
t/issue_45.t ............ ok
t/issue_48.t ............ ok
t/issue_53.t ............ ok
t/issue_reports.t ....... ok
t/package.t ............. ok
t/perl6.t ............... ok
t/recursive_tokenize.t .. perl: src/compiler/lexer/Compiler_lexer.cpp:314: bool Lexer::isExpr(Token_, Token_, Enum::Token::Type::Type, Enum::Token::Kind::Kind): Assertion `tk->tks[0]->info.type == LeftBrace' failed.
t/recursive_tokenize.t .. No subtests run
t/verbose.t ............. ok

Test Summary Report

t/recursive_tokenize.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
Files=18, Tests=28, 4 wallclock secs ( 0.09 usr 0.04 sys + 2.89 cusr 0.91 csys = 3.93 CPU)
Result: FAIL
Failed 1/18 test programs. 0/28 subtests failed.

perl -v says
...

This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux

See also #67 but I don't know if that's related.

Strangely enough I tried the same Compiler::Lexer 0.22 in OS X, and there it works fine.

'$#x' is 'ArraySize + StringMul'

correct : $#x -> ArraySize + Key
irregal : $#x -> ArraySize + StringMul

Column number information

I noticed that the returned token object contains line number information but no column number information. Is there an undocumented feature to access the column number information?

And thanks 👍

Segmentation fault in tokenize

Following script cause segmentation fault in my machine.

v5.16.2 built for x86_64-linux.

use strict;
use warnings;
use utf8;

use Compiler::Lexer;

for (
    ['package Foo 3 { }', '5.012'],
    ['package Foo 3', '5.012'],
    ['require mro', '5.010'],
    ['use mro', '5.010'],
) {
    # warn $_->[0];
    my $lexer = Compiler::Lexer->new('-');
    my $tokens2 = $lexer->tokenize($_->[0]);
    my @tokens = map { @$$_ } $tokens2;
}

print "Done\n";

Here is backtrace.

gdb$ bt
#0  0x00007ffff6a94818 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff6d12cfe in std::operator==<char, std::char_traits<char>, std::allocator<char> > (__lhs=...,
    __rhs=0x7ffff6d1d7f2 "=>") at /usr/include/c++/4.6/bits/basic_string.h:2462
#2  0x00007ffff6d13d36 in Annotator::annotateKey (this=0x893960, ctx=0x893910, tk=0x893880)
    at src/compiler/lexer/Compiler_annotator.cpp:137
#3  0x00007ffff6d13503 in Annotator::annotate (this=0x893960, ctx=0x893910, tk=0x893880)
    at src/compiler/lexer/Compiler_annotator.cpp:47
#4  0x00007ffff6d18101 in Lexer::annotateTokens (this=0x893690, tokens=0x7979a0)
    at src/compiler/lexer/Compiler_lexer.cpp:225
#5  0x00007ffff6d0ab4a in XS_Compiler__Lexer_tokenize (cv=0x8fdb50) at src/Compiler-Lexer.xs:46
#6  0x000000000049aebc in Perl_pp_entersub () at pp_hot.c:2778
#7  0x0000000000493b43 in Perl_runops_standard () at run.c:41
#8  0x000000000043a06a in S_run_body (oldscope=1) at perl.c:2402
#9  perl_run (my_perl=<optimized out>) at perl.c:2320
#10 0x000000000041f36d in main (argc=2, argv=0x7fffffffe628, env=0x7fffffffe640) at perlmain.c:120

qoutewords parsed as regex

use Compiler::Lexer 0.13;
use Data::Dumper;
my $lexer = Compiler::Lexer->new( $0 );
my $tokens = $lexer->tokenize( 'qw( foo bar baz )' );
say Dumper ($tokens );

$VAR1 = [
bless( {
'has_warnings' => 0,
'line' => 1,
'stype' => 0,
'kind' => 30,
'type' => 147,
'data' => 'qw',
'name' => 'RegList'
}, 'Compiler::Lexer::Token' ),
bless( {
'name' => 'RegDelim',
'data' => '(',
'type' => 151,
'kind' => 24,
'line' => 1,
'stype' => 0,
'has_warnings' => 0
}, 'Compiler::Lexer::Token' ),
bless( {
'type' => 186,
'name' => 'RegExp',
'data' => ' foo bar baz ',
'has_warnings' => 0,
'line' => 1,
'stype' => 0,
'kind' => 24
}, 'Compiler::Lexer::Token' ),
bless( {
'type' => 151,
'name' => 'RegDelim',
'data' => ')',
'has_warnings' => 0,
'line' => 1,
'stype' => 0,
'kind' => 24
}, 'Compiler::Lexer::Token' )
];

Can't recognize `package Foo'Bar`

Package name can contain single quotation (called apostrophe). It is used as NAMESPACE RESOLVER.

e.g.

package Foo'Bar;
say __PACKAGE__; # => Foo::Bar

This module can't recognize it.

lexer thinks ${ is always a ScalarDereference

Which is not true.

perl -wle '$foo = 42; print ${foo}'
42

perl -Mstrict -MCompiler::Lexer -MData::Dumper -wle 'print Dumper(Compiler::Lexer->new->tokenize(q[${foo}]))'
$VAR1 = [
bless( {
'kind' => 23,
'has_warnings' => 0,
'stype' => 0,
'name' => 'ScalarDereference',
'data' => '${',
'type' => 115,
'line' => 1
}, 'Compiler::Lexer::Token' ),
bless( {
'kind' => 24,
'has_warnings' => 1,
'stype' => 0,
'name' => 'Key',
'data' => 'foo',
'type' => 122,
'line' => 1
}, 'Compiler::Lexer::Token' ),
bless( {
'kind' => 22,
'has_warnings' => 0,
'stype' => 0,
'name' => 'RightBrace',
'data' => '}',
'type' => 110,
'line' => 1
}, 'Compiler::Lexer::Token' )
];

could not parse the v-string

It could not parse the v-string correctly such as use v5.18.0.

$ echo 'use v5.18.0;' | perl -MCompiler::Lexer -MData::Dumper -E 'say Dumper(Compiler::Lexer->new("")->get_used_modules(<>))'

The result is

$VAR1 = [
          {
            'args' => '  .  18.0',
            'name' => 'v5'
          }
        ];

Can build on windows x64 with activeperl 5.20.1 x64

Here is the log:

--> Working on Compiler::Lexer
Fetching http://www.cpan.org/authors/id/G/GO/GOCCY/Compiler-Lexer-0.21.tar.gz ... OK
Compiler-Lexer-0.21/Build.PL
Compiler-Lexer-0.21/Changes
Compiler-Lexer-0.21/LICENSE
Compiler-Lexer-0.21/META.json
Compiler-Lexer-0.21/README.md
Compiler-Lexer-0.21/builder/MyBuilder.pm
Compiler-Lexer-0.21/cpanfile
Compiler-Lexer-0.21/cpanfile.snapshot
Compiler-Lexer-0.21/example/benchmark.pl
Compiler-Lexer-0.21/example/sample.pl
Compiler-Lexer-0.21/gen/double_charactor_operator.gperf
Compiler-Lexer-0.21/gen/gen_constants.yaml
Compiler-Lexer-0.21/gen/gen_decl.pl
Compiler-Lexer-0.21/gen/reserved_keywords.gperf
Compiler-Lexer-0.21/gen/test_generator.pl
Compiler-Lexer-0.21/gen/triple_charactor_operator.gperf
Compiler-Lexer-0.21/include/common.hpp
Compiler-Lexer-0.21/include/gen_token.hpp
Compiler-Lexer-0.21/include/keyword.hpp
Compiler-Lexer-0.21/include/lexer.hpp
Compiler-Lexer-0.21/include/token.hpp
Compiler-Lexer-0.21/lib/Compiler/Lexer.pm
Compiler-Lexer-0.21/lib/Compiler/Lexer/Constants.pm
Compiler-Lexer-0.21/lib/Compiler/Lexer/Token.pm
Compiler-Lexer-0.21/minil.toml
Compiler-Lexer-0.21/src/Compiler-Lexer.xs
Compiler-Lexer-0.21/src/compiler/lexer/Compiler_annotator.cpp
Compiler-Lexer-0.21/src/compiler/lexer/Compiler_lexer.cpp
Compiler-Lexer-0.21/src/compiler/lexer/Compiler_manager.cpp
Compiler-Lexer-0.21/src/compiler/lexer/Compiler_scanner.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_double_charactor_operator.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_gen_token_decl.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_reserved_keyword.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_token.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_triple_charactor_operator.cpp
Compiler-Lexer-0.21/src/compiler/util/Compiler_util.cpp
Compiler-Lexer-0.21/src/typemap
Compiler-Lexer-0.21/t/Lexer.t
Compiler-Lexer-0.21/t/format.t
Compiler-Lexer-0.21/t/issue_32.t
Compiler-Lexer-0.21/t/issue_35.t
Compiler-Lexer-0.21/t/issue_38.t
Compiler-Lexer-0.21/t/issue_39.t
Compiler-Lexer-0.21/t/issue_40.t
Compiler-Lexer-0.21/t/issue_42.t
Compiler-Lexer-0.21/t/issue_43.t
Compiler-Lexer-0.21/t/issue_44.t
Compiler-Lexer-0.21/t/issue_45.t
Compiler-Lexer-0.21/t/issue_reports.t
Compiler-Lexer-0.21/t/package.t
Compiler-Lexer-0.21/t/perl6.t
Compiler-Lexer-0.21/t/recursive_tokenize.t
Compiler-Lexer-0.21/t/verbose.t
Compiler-Lexer-0.21/typemap
Compiler-Lexer-0.21/META.yml
Compiler-Lexer-0.21/MANIFEST
Configuring Compiler-Lexer-0.21 ... Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'Compiler-Lexer' version '0.21'
cp META.json MYMETA.json
cp META.yml MYMETA.yml
OK
Building and testing Compiler-Lexer-0.21 ... Building Compiler-Lexer
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\lexer\Compiler_annotator.o" "src\compiler\lexer\Compiler_annotator.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\lexer\Compiler_lexer.o" "src\compiler\lexer\Compiler_lexer.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\lexer\Compiler_manager.o" "src\compiler\lexer\Compiler_manager.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\lexer\Compiler_scanner.o" "src\compiler\lexer\Compiler_scanner.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_double_charactor_operator.o" "src\compiler\util\Compiler_double_charactor_operator.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_gen_token_decl.o" "src\compiler\util\Compiler_gen_token_decl.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_reserved_keyword.o" "src\compiler\util\Compiler_reserved_keyword.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_token.o" "src\compiler\util\Compiler_token.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_triple_charactor_operator.o" "src\compiler\util\Compiler_triple_charactor_operator.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "src\compiler\util\Compiler_util.o" "src\compiler\util\Compiler_util.cpp"
d:\devel\perl\perl\site\bin\gcc.exe -c -I"src" -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields -Iinclude -Wno-missing-field-initializers -g3 -xc++ -s -O2 "-DXS_VERSION=\"0.21\"" "-DVERSION=\"0.21\"" -I"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -I"C:\MinGW\x86_64-w64-mingw32\include" -o "lib\Compiler\Lexer.o" "lib\Compiler\Lexer.c"
ExtUtils::Mkbootstrap::Mkbootstrap('blib\arch\auto\Compiler\Lexer\Lexer.bs')
Generating script 'lib\Compiler\Lexer.lds'
d:\devel\perl\perl\site\bin\dlltool.exe --def "lib\Compiler\Lexer.def" --output-exp "lib\Compiler\Lexer.exp"
d:\devel\perl\perl\site\bin\g++.exe -o "blib\arch\auto\Compiler\Lexer\Lexer.dll" -Wl,--base-file,"lib\Compiler\Lexer.base" -Wl,--image-base,0x3e4b0000 -mdll -s -static-libgcc -static-libstdc++ -L"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -L"C:\MinGW\x86_64-w64-mingw32\lib" "lib\Compiler\Lexer.lds" -lstdc++ "lib\Compiler\Lexer.exp"
d:\devel\perl\perl\site\bin\dlltool.exe --def "lib\Compiler\Lexer.def" --output-exp "lib\Compiler\Lexer.exp" --base-file "lib\Compiler\Lexer.base"
d:\devel\perl\perl\site\bin\g++.exe -o "blib\arch\auto\Compiler\Lexer\Lexer.dll" -Wl,--image-base,0x3e4b0000 -mdll -s -static-libgcc -static-libstdc++ -L"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -L"C:\MinGW\x86_64-w64-mingw32\lib" "lib\Compiler\Lexer.lds" -lstdc++ "lib\Compiler\Lexer.exp"
Generating script 'lib\Compiler\Lexer.lds'
d:\devel\perl\perl\site\bin\dlltool.exe --def "lib\Compiler\Lexer.def" --output-exp "lib\Compiler\Lexer.exp"
d:\devel\perl\perl\site\bin\g++.exe -o "blib\arch\auto\Compiler\Lexer\Lexer.dll" -Wl,--base-file,"lib\Compiler\Lexer.base" -Wl,--image-base,0x3e4b0000 -mdll -s -static-libgcc -static-libstdc++ -L"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -L"C:\MinGW\x86_64-w64-mingw32\lib" "lib\Compiler\Lexer.lds" -lstdc++ "lib\Compiler\Lexer.exp"
d:\devel\perl\perl\site\bin\dlltool.exe --def "lib\Compiler\Lexer.def" --output-exp "lib\Compiler\Lexer.exp" --base-file "lib\Compiler\Lexer.base"
d:\devel\perl\perl\site\bin\g++.exe -o "blib\arch\auto\Compiler\Lexer\Lexer.dll" -Wl,--image-base,0x3e4b0000 -mdll -s -static-libgcc -static-libstdc++ -L"D:\devel\perl-activestate-x64-5.20.1.2000\lib\CORE" -L"C:\MinGW\x86_64-w64-mingw32\lib" "lib\Compiler\Lexer.lds" -lstdc++ "lib\Compiler\Lexer.exp"
t\Lexer.t ............... 
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/1 subtests 
t\format.t .............. 
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/1 subtests 
t\issue_32.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_35.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_38.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_39.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_40.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_42.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_43.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_44.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_45.t ............ 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\issue_reports.t ....... 
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/1 subtests 
t\package.t ............. 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\perl6.t ............... 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\recursive_tokenize.t .. 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
t\verbose.t ............. 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 

Test Summary Report
-------------------
t\Lexer.t             (Wstat: 65280 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 255
t\format.t            (Wstat: 65280 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_32.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_35.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_38.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_39.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_40.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_42.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_43.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_44.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_45.t          (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\issue_reports.t     (Wstat: 65280 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\package.t           (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\perl6.t             (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\recursive_tokenize.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t\verbose.t           (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=16, Tests=3,  3 wallclock secs ( 0.05 usr +  0.05 sys =  0.09 CPU)
Result: FAIL
FAIL

memory corruption in recursive_tokenize.t

Compiler::Lexer 0.19

$ perl -Mblib t/recursive_tokenize.t
*** glibc detected *** perl: malloc(): memory corruption (fast): 0x0a7759d8 ***

Aborted on tokenizing “^/”

Environment

Mac OS X 10.9.4
Perl 5.18.1
clang-503.0.40

Code to reproduce

Compiler::Lexer->new->tokenize('^/');

If you cannot reproduce with the above, please try this;

while (1) {
    Compiler::Lexer->new->tokenize('^/');
}

Output

perl5.18.1(56819,0x7fff7818a310) malloc: *** error for object 0x7ffcbb101208: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

And also aborts on tokenizing only / (e.g. Compiler::Lexer->new->tokenize('/');).

`/!/` is regexp but...

use strict;
use warnings;
use Compiler::Lexer;
use Test::More;

my $src = "( sub { /!/ }, '//' )";
my $lexer = Compiler::Lexer->new('-');
my $tokens = $lexer->tokenize($src);

my @dor = grep { $_->name eq 'DefaultOperator' && $_->data eq '//' } @$tokens;
is 0+@dor, 0;

done_testing;

`q'foobar'` has been analyzed by mistake

Now, q'foobar' has been analyzed such;

bless( {
    data => "qfoobar",
    has_warnings => 0,
    kind => 24,
    line => 8,
    name => "RawString",
    stype => 0,
    type => 170
}, 'Compiler::Lexer::Token' ),

data of this result has wrong value and it doesn't associated with RegQuote.
In my feeling, correct result is expected like so;

bless( {
    data => "q",
    has_warnings => 0,
    kind => 30,
    line => 1,
    name => "RegQuote",
    stype => 0,
    type => 142
}, 'Compiler::Lexer::Token' ),
bless( {
    data => "'",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegDelim",
    stype => 0,
    type => 148
}, 'Compiler::Lexer::Token' ),
bless( {
    data => "foobar",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegExp",
    stype => 0,
    type => 183
}, 'Compiler::Lexer::Token' ),
bless( {
    data => "'",
    has_warnings => 0,
    kind => 24,
    line => 1,
    name => "RegDelim",
    stype => 0,
    type => 148
}, 'Compiler::Lexer::Token' ),

This recognition is correct?

And also q"foobar", qq'foobar', qq"foobar", qw'foobar' and qw"foobar" have been analyzed by mistake in appearance.
Of course, q{foobar} and such have no problem. This problem occurs when quotes are used as delimiter.

lexer thinks ${^FOO} contains BitXOr

It's just a funny variable name.

perl -Mstrict -MCompiler::Lexer -MData::Dumper -wle 'print Dumper(Compiler::Lexer->new->tokenize(q[${^FOO}]))'
$VAR1 = [
bless( {
'kind' => 23,
'has_warnings' => 0,
'stype' => 0,
'name' => 'ScalarDereference',
'data' => '${',
'type' => 115,
'line' => 1
}, 'Compiler::Lexer::Token' ),
bless( {
'kind' => 1,
'has_warnings' => 0,
'stype' => 0,
'name' => 'BitXOr',
'data' => '^',
'type' => 17,
'line' => 1
}, 'Compiler::Lexer::Token' ),
bless( {
'kind' => 24,
'has_warnings' => 1,
'stype' => 0,
'name' => 'Key',
'data' => 'FOO',
'type' => 122,
'line' => 1
}, 'Compiler::Lexer::Token' ),
bless( {
'kind' => 22,
'has_warnings' => 0,
'stype' => 0,
'name' => 'RightBrace',
'data' => '}',
'type' => 110,
'line' => 1
}, 'Compiler::Lexer::Token' )
];

(The ScalarDereference is a bug, too, opened #65 for it.)

Parsing v-string

  perl -e 'use Data::Dumper; use Compiler::Lexer; print Dumper(Compiler::Lexer->new("-")->tokenize("v0.0.1"))'

Returns

    $VAR1 = \[
                bless( {
                         'name' => 'Key',
                         'kind' => 21,
                         'stype' => 0,
                         'data' => 'v0',
                         'type' => 114,
                         'has_warnings' => 1,
                         'line' => 1
                       }, 'Compiler::Lexer::Token' ),
                bless( {
                         'stype' => 0,
                         'kind' => 1,
                         'name' => 'StringAdd',
                         'has_warnings' => 0,
                         'line' => 1,
                         'data' => '.',
                         'type' => 9
                       }, 'Compiler::Lexer::Token' ),
                bless( {
                         'type' => 162,
                         'data' => '0.1',
                         'line' => 1,
                         'has_warnings' => 0,
                         'name' => 'Double',
                         'kind' => 21,
                         'stype' => 0
                       }, 'Compiler::Lexer::Token' )
              ];

But perl5 parse "v0.0.1" as a v-string.

Free memory before global destruction (was: no need to use safe_free() ?)

safe_malloc() is used in src/compiler/lexer/Compiler_lexer.cpp and src/compiler/util/Compiler_token.cpp, but apparently safe_free() is not used in anywhere.

Recognize the v5.20 post dereference syntax

v5.20 has new syntax to recognize after a Pointer (->).

I've sent a pull request. I think I've done the right thing.

suspect naming of quote operators: RegQuote etc.

I think the naming of these lexemes is a bit suspect:

q RegQuote
qq RegDoubleQuote
qw RegList
qx RegExec

(qr is RegDecl, which is fine, kind of, see below)

and the corresponding delimiters are always called RegDelim.

This is very confusing since the q/qq/qw/qx have nothing to do with Regular expressions, and neither do the brackets in qq[...].

Also, the delimiters are always called RegDelim, regardless of whether they open/begin or close/end the string. Is this intentional/good?

I would suggest renaming, e.g.

q QuoteSingle
qq QuoteDouble
qw QuoteList
qx QuoteExec

(and qx could either stay RegDecl, or if we want consistency, renamed as QuoteRegex)

and the delimiters could be renamed QuoteOpen and QuoteClose (or Begin and End?).
If the delimiters are not named as paired, then QuoteDelim, instead of RegDelim.

Lexer struggles

The folowing code let the lexer struggle:

package mypackage;

sub get_average {
        my ($sum, $count) = @_;
        return($sum / $count);
}

sub get_ascii_art {
        my $polizist = q@
          ,
 __  _.-"` `'-.
/||\'._ __{}_(
||||  |'--.__\
|  L.(   ^_\^
\ .-' |   _ |
| |   )\___/
|  \-'`:._]
\__/;      '-.
@;  #"

}
1;

compilation warning from Compiler_lexer.cpp

src/compiler/lexer/Compiler_lexer.cpp: In member function ‘void Lexer::parseSpecificStmt(Token*)’:
src/compiler/lexer/Compiler_lexer.cpp:512: warning: comparison between ‘enum Enum::Token::Type::Type’ and ‘enum Enum::Parser::Syntax::Type’

gcc -v says:

gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)

Access to `y` via allow, it breaks chain structure

e.g.

Compiler::Lexer->new->tokenize('$x->y()->z')

bless( {
        'data' => '$x',
        'has_warnings' => 0,
        'kind' => 24,
        'line' => 1,
        'name' => 'GlobalVar',
        'stype' => 0,
        'type' => 190
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => '->',
        'has_warnings' => 0,
        'kind' => 1,
        'line' => 1,
        'name' => 'Pointer',
        'stype' => 0,
        'type' => 122
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => 'y',
        'has_warnings' => 0,
        'kind' => 31,
        'line' => 1,
        'name' => 'RegAllReplace',
        'stype' => 0,
        'type' => 151
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => '(',
        'has_warnings' => 0,
        'kind' => 24,
        'line' => 1,
        'name' => 'RegDelim',
        'stype' => 0,
        'type' => 148
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => '',
        'has_warnings' => 0,
        'kind' => 24,
        'line' => 1,
        'name' => 'RegReplaceFrom',
        'stype' => 0,
        'type' => 153
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => ')',
        'has_warnings' => 0,
        'kind' => 24,
        'line' => 1,
        'name' => 'RegMiddleDelim',
        'stype' => 0,
        'type' => 150
    }, 'Compiler::Lexer::Token' ),
bless( {
        'data' => '->z',
        'has_warnings' => 1,
        'kind' => 24,
        'line' => 1,
        'name' => 'Key',
        'stype' => 0,
        'type' => 119
    }, 'Compiler::Lexer::Token' )

Expected y() is subroutine call, but it handled as RegAllReplace (:point_down:) and breaks following structure (:point_up:).

bless( {
        'data' => 'y',
        'has_warnings' => 0,
        'kind' => 31,
        'line' => 1,
        'name' => 'RegAllReplace',
        'stype' => 0,
        'type' => 151
    }, 'Compiler::Lexer::Token' ),

The similar problem occurs on the following code;

Compiler::Lexer->new->tokenize('$x->y->z')

segfault on Compiler::Lexer->new('-')->recursive_tokenize('{}');

Yes.

Token type of after pointer token

$var->foo()
foo is the method token
$var->_foo()
_foo is the key token

Are these behavior correct? I feel _foo might be method token.
How about?

`glibc detected free() invalid next size (fast)` on CentOS release 6.5

Environment

CentOS release 6.5 (Final)
Linux 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
perl 5.16.3
Compiler::Lexer 0.18

Code to reproduce

#!/usr/local/bin/env perl

use Compiler::Lexer;
my $script = "\n1;\nEND {}\n";
my $lexer = Compiler::Lexer->new();
my $tokens = $lexer->tokenize($script);

Dump

*** glibc detected *** perl: free(): invalid pointer: 0x0000000001a6ea70 ***
======= Backtrace: =========
/lib64/libc.so.6[0x307b276166]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E+0x6d)[0x7ffd5cb3758d]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E+0x44)[0x7ffd5cb37564]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E+0x44)[0x7ffd5cb37564]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E+0x44)[0x7ffd5cb37564]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE8_M_eraseEPSt13_Rb_tree_nodeIS2_E+0x44)[0x7ffd5cb37564]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(_ZN5Lexer8tokenizeEPc+0x38f)[0x7ffd5cb36c9f]
/home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so(+0x1c542)[0x7ffd5cb33542]
perl(Perl_pp_entersub+0x618)[0x497ac8]
perl(Perl_runops_standard+0x13)[0x4961a3]
perl(perl_run+0x2c8)[0x434878]
perl(main+0x10c)[0x41e10c]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x307b21ed1d]
perl[0x41df39]
======= Memory map: ========
00400000-0055a000 r-xp 00000000 fc:03 1708854                            /home/moznion/.plenv/versions/5.16.3/bin/perl5.16.3
00759000-0075e000 rw-p 00159000 fc:03 1708854                            /home/moznion/.plenv/versions/5.16.3/bin/perl5.16.3
0075e000-0075f000 rw-p 00000000 00:00 0
01717000-01b66000 rw-p 00000000 00:00 0                                  [heap]
307ae00000-307ae20000 r-xp 00000000 fc:03 3276920                        /lib64/ld-2.12.so
307b01f000-307b020000 r--p 0001f000 fc:03 3276920                        /lib64/ld-2.12.so
307b020000-307b021000 rw-p 00020000 fc:03 3276920                        /lib64/ld-2.12.so
307b021000-307b022000 rw-p 00000000 00:00 0
307b200000-307b38b000 r-xp 00000000 fc:03 3277005                        /lib64/libc-2.12.so
307b38b000-307b58a000 ---p 0018b000 fc:03 3277005                        /lib64/libc-2.12.so
307b58a000-307b58e000 r--p 0018a000 fc:03 3277005                        /lib64/libc-2.12.so
307b58e000-307b58f000 rw-p 0018e000 fc:03 3277005                        /lib64/libc-2.12.so
307b58f000-307b594000 rw-p 00000000 00:00 0
307ba00000-307ba02000 r-xp 00000000 fc:03 3276896                        /lib64/libdl-2.12.so
307ba02000-307bc02000 ---p 00002000 fc:03 3276896                        /lib64/libdl-2.12.so
307bc02000-307bc03000 r--p 00002000 fc:03 3276896                        /lib64/libdl-2.12.so
307bc03000-307bc04000 rw-p 00003000 fc:03 3276896                        /lib64/libdl-2.12.so
307be00000-307be83000 r-xp 00000000 fc:03 3277212                        /lib64/libm-2.12.so
307be83000-307c082000 ---p 00083000 fc:03 3277212                        /lib64/libm-2.12.so
307c082000-307c083000 r--p 00082000 fc:03 3277212                        /lib64/libm-2.12.so
307c083000-307c084000 rw-p 00083000 fc:03 3277212                        /lib64/libm-2.12.so
307d200000-307d271000 r-xp 00000000 fc:03 3277450                        /lib64/libfreebl3.so
307d271000-307d470000 ---p 00071000 fc:03 3277450                        /lib64/libfreebl3.so
307d470000-307d472000 r--p 00070000 fc:03 3277450                        /lib64/libfreebl3.so
307d472000-307d473000 rw-p 00072000 fc:03 3277450                        /lib64/libfreebl3.so
307d473000-307d477000 rw-p 00000000 00:00 0
307da00000-307da02000 r-xp 00000000 fc:03 3282188                        /lib64/libutil-2.12.so
307da02000-307dc01000 ---p 00002000 fc:03 3282188                        /lib64/libutil-2.12.so
307dc01000-307dc02000 r--p 00001000 fc:03 3282188                        /lib64/libutil-2.12.so
307dc02000-307dc03000 rw-p 00002000 fc:03 3282188                        /lib64/libutil-2.12.so
307de00000-307de07000 r-xp 00000000 fc:03 3277451                        /lib64/libcrypt-2.12.so
307de07000-307e007000 ---p 00007000 fc:03 3277451                        /lib64/libcrypt-2.12.so
307e007000-307e008000 r--p 00007000 fc:03 3277451                        /lib64/libcrypt-2.12.so
307e008000-307e009000 rw-p 00008000 fc:03 3277451                        /lib64/libcrypt-2.12.so
307e009000-307e037000 rw-p 00000000 00:00 0
307ea00000-307ea16000 r-xp 00000000 fc:03 3282198                        /lib64/libnsl-2.12.so
307ea16000-307ec15000 ---p 00016000 fc:03 3282198                        /lib64/libnsl-2.12.so
307ec15000-307ec16000 r--p 00015000 fc:03 3282198                        /lib64/libnsl-2.12.so
307ec16000-307ec17000 rw-p 00016000 fc:03 3282198                        /lib64/libnsl-2.12.so
307ec17000-307ec19000 rw-p 00000000 00:00 0
3792400000-37924e8000 r-xp 00000000 fc:03 6029965                        /usr/lib64/libstdc++.so.6.0.13
37924e8000-37926e8000 ---p 000e8000 fc:03 6029965                        /usr/lib64/libstdc++.so.6.0.13
37926e8000-37926ef000 r--p 000e8000 fc:03 6029965                        /usr/lib64/libstdc++.so.6.0.13
37926ef000-37926f1000 rw-p 000ef000 fc:03 6029965                        /usr/lib64/libstdc++.so.6.0.13
37926f1000-3792706000 rw-p 00000000 00:00 0
7ffd5c2f1000-7ffd5c2f9000 r-xp 00000000 fc:03 1968393                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Data/Dumper/Dumper.so
7ffd5c2f9000-7ffd5c4f8000 ---p 00008000 fc:03 1968393                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Data/Dumper/Dumper.so
7ffd5c4f8000-7ffd5c4f9000 rw-p 00007000 fc:03 1968393                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Data/Dumper/Dumper.so
7ffd5c4f9000-7ffd5c4fd000 r-xp 00000000 fc:03 1968510                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/IO/IO.so
7ffd5c4fd000-7ffd5c6fc000 ---p 00004000 fc:03 1968510                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/IO/IO.so
7ffd5c6fc000-7ffd5c6fd000 rw-p 00003000 fc:03 1968510                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/IO/IO.so
7ffd5c6fd000-7ffd5c700000 r-xp 00000000 fc:03 1968500                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Fcntl/Fcntl.so
7ffd5c700000-7ffd5c900000 ---p 00003000 fc:03 1968500                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Fcntl/Fcntl.so
7ffd5c900000-7ffd5c901000 rw-p 00003000 fc:03 1968500                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Fcntl/Fcntl.so
7ffd5c901000-7ffd5c917000 r-xp 00000000 fc:03 3277230                    /lib64/libgcc_s-4.4.7-20120601.so.1
7ffd5c917000-7ffd5cb16000 ---p 00016000 fc:03 3277230                    /lib64/libgcc_s-4.4.7-20120601.so.1
7ffd5cb16000-7ffd5cb17000 rw-p 00015000 fc:03 3277230                    /lib64/libgcc_s-4.4.7-20120601.so.1
7ffd5cb17000-7ffd5cb45000 r-xp 00000000 fc:03 3539649                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so
7ffd5cb45000-7ffd5cd45000 ---p 0002e000 fc:03 3539649                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so
7ffd5cd45000-7ffd5cd59000 rw-p 0002e000 fc:03 3539649                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/site_perl/5.16.3/x86_64-linux/auto/Compiler/Lexer/Lexer.so
7ffd5cd59000-7ffd5cd5f000 r-xp 00000000 fc:03 1968419                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/List/Util/Util.so
7ffd5cd5f000-7ffd5cf5e000 ---p 00006000 fc:03 1968419                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/List/Util/Util.so
7ffd5cf5e000-7ffd5cf5f000 rw-p 00005000 fc:03 1968419                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/List/Util/Util.so
7ffd5cf5f000-7ffd5cf61000 r-xp 00000000 fc:03 1969986                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Cwd/Cwd.so
7ffd5cf61000-7ffd5d160000 ---p 00002000 fc:03 1969986                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Cwd/Cwd.so
7ffd5d160000-7ffd5d161000 rw-p 00001000 fc:03 1969986                    /home/moznion/.plenv/versions/5.16.3/lib/perl5/5.16.3/x86_64-linux/auto/Cwd/Cwd.so
7ffd5d161000-7ffd62ff2000 r--p 00000000 fc:03 6032906                    /usr/lib/locale/locale-archive
7ffd62ff2000-7ffd62ff7000 rw-p 00000000 00:00 0
7ffd62fff000-7ffd63001000 rw-p 00000000 00:00 0
7fff33883000-7fff33898000 rw-p 00000000 00:00 0                          [stack]
7fff3389d000-7fff3389e000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

NOTE

If we write use strict; additionally into reproducing code, it works well. But I don't know whether or not related it....

goccy / p5-compiler-lexer Goto Github PK

p5-compiler-lexer's Introduction

NAME

SYNOPSIS

METHODS

AUTHOR

CONTRIBUTORS

LICENSE AND COPYRIGHT

p5-compiler-lexer's People

Contributors

Stargazers

Watchers

Forkers

p5-compiler-lexer's Issues

Sample code to reproduce;

Bad result

Expected

Code to reproduce

Result

Test Summary Report

Environment

Code to reproduce

Output

Environment

Code to reproduce

Dump

NOTE

Recommend Projects

Recommend Topics

Recommend Org