Using gegen and reg within program returns error 198 about stata-gtools HOT 55 CLOSED

I don't get this issue. Can you run

clear all
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
        disp "`level', $_level"
	gegen test = sum(price)
        disp "`level', $_level"
	reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78

I don't think gtools ever sets a local called "level" but I could be wrong.

from stata-gtools.

disp gives no output, just the usual error message.

I don't know if this is helpful, but I noted the strange behavior that after a run with gegen, running the test program with egen again returned the same error message. Only after restarting Stata the egen version runs fine again.
Even more strange, on Win 10 the program runs fine ?!

from stata-gtools.

As I noted, I couldn't reproduce the issue, which is why I was trying to figure out if level really was defined by gegen. Ah, ok, I think this, perhaps:

clear all
log using test, text
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
        disp "`level', $_level"
	cap noi gegen test = sum(price)
        disp "`level', $_level"
        exit `=_rc'
	reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78
log close _all

And send the log file? You say you are using parallel, right? Is the first snippet you posted the exact code you were running?

from stata-gtools.

What is the OS where it gave the error? OSX? Win7?

from stata-gtools.

running on arch linux. This is the log file:
test.log

from stata-gtools.

I'm on arch as well, so I'm not sure what's failing. gegen definitely doesn't do anything to level, which is what I thought, but that was your original error...

Can you run

version
which gegen
which gtools

?

from stata-gtools.

. version
version 15.1

. which gegen
/home/pp/ado/plus/g/gegen.ado
*! version 0.13.1 02May2018 Mauricio Caceres Bravo, [email protected]
*! implementation -egen- using C for faster processing

. which gtools
/home/pp/ado/plus/g/gtools.ado
*! version 0.13.1 02May2018 Mauricio Caceres Bravo, [email protected]
*! Program for managing the gtools package installation

The starting point of this issue was gvegayon/parallel#66

from stata-gtools.

Thanks! I'll look over that issue thread and see if there's any clues there. I upgraded to parallel 0.19 and still no issue. Will keep you posted.

from stata-gtools.

I've tried it on 2 other Linux systems w/o error. It's really strange to me because we are using the same OS and gtools version anyway... Can you run:

clear all
log using test, text
which reg
which regress
which _get_diopts
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
	egen test = sum(price)
        reg `varlist' if `touse'
	gegen test = sum(price), replace
        reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78
log close _all

Sorry I'm basically going blind here, since I don't get the same issue you do. Hopefully we can find the root cause of the problem, though.

from stata-gtools.

Thanks for all your efforts. I noticed that I can reproduce the error even with this code:

clear all
log using test, text
which reg
which regress
which _get_diopts
sysuse auto , clear

egen test = mean(price)
reg price weight
gegen test2 = mean(price)
reg price weight

log close _all

log: test.log

Removing all ados in PLUS and reinstalling only gtools didn't help either. Might want to try reinstalling Stata itself?

from stata-gtools.

Does this happen with other gtools commands or is it just gegen? So if you run gunique price or gtop price does it also give an error with reg? (Note: Maybe try regress and _regress instead of reg?)

from stata-gtools.

gtop and gunique produce the same error, also with regress or _regress.

from stata-gtools.

Can I have the log for

findfile gtools_unix.plugin
!ldd `r(fn)'
clear all
sysuse auto
global GTOOLS_CALLER gegen
_gtools_internal price, gfunction(hash) gen(test) debug(9) v bench(3)
_regress price weight

? I expect you will get the same error. Hopefully the log has a clue...

from stata-gtools.

here is the log: test.log

from stata-gtools.

log for

clear all
sysuse auto
global GTOOLS_CALLER gegen
_gtools_internal price, gfunction(hash) gen(test)
set trace on
_get_diopts diopts options,

?

from stata-gtools.

test.log

from stata-gtools.

This is very strange; there is nothing in the source file that suggests where level gets assigned to its default value of 95. It should be blank throughout... Could you edit _get_diopts.ado?

findfile _get_diopts.ado
!cp `r(fn)' `r(fn)'.bak

Could you add

disp "`level'"

to GetDiopts, before and after the first syntax statement (for me about line 81). Also after the last opts_exclusive (for me around line 96) as well as before and after the second syntax statement (for me around line 157, right before the level error.

Then re-run the latest snippet. Thank you for all your help debugging.

from stata-gtools.

log: test.log

from stata-gtools.

Ok, I think this at least gives a clue, since level is getting assigned to its default value and that turns out to be "95,0" for some bizarre reason... Although you said you you get the same error when you pass level. Ok, could you run:

clear all
sysuse auto
regress price weight, level(90)
gegen test = mean(price)
creturn list
disp "`c(level)', `c(clevel)'"
set trace on
_get_diopts diopts options, level(89)

(keep the modifications to _get_diopts.ado).

from stata-gtools.

log: test.log

from stata-gtools.

Oh, wow, so it's not setting the default. It's actually appending ",0" to whatever you pass. That is exceedingly bizarre. Can you try:

capture program drop test
program test
    syntax, [level(cilevel)]
    disp "`level'"
end

clear all
set obs 10
gen x = 1
test
gegen y = sum(x)
test

from stata-gtools.

Yeah, bizarre is probably the right description. Here the output:

. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1

. test
95

. gegen y = sum(x)

. test
95,0

from stata-gtools.

Can you try that with the develop branch? net uninstall gtools; then from a fresh Stata session can you run:

local github "https://raw.githubusercontent.com"
net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)

clear all
capture program drop test
program test
    disp "`0'"
    syntax, [foo(cilevel)]
    disp "`foo'"
end

set obs 10
gen x = 1
test
gegen y = sum(x)
test

Can you also try

clear all
set obs 10
gen x = 1
syntax, [foo(cilevel)]
disp "`foo'"
gegen y = sum(x)
syntax, [foo(cilevel)]
disp "`foo'"

from stata-gtools.

same result after installing dev branch:

. gen x = 1
. test
95
. gegen y = sum(x)
. test
95,0

and for the second program

. gen x = 1
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. gegen y = sum(x)
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

from stata-gtools.

By the way, I was unsure if some local configurations caused the error. Therefore, I installed Stata MP 15.1 + gtools in a fresh Ubuntu 18.04 virtualbox environment. Too bad, I was able to reproduce the same error.

from stata-gtools.

Tried fresh 18.04 in virtualbox as well and no error for me. I have no clue what's up! I'm really sorry. Un_n

• Ubuntu 18.04
• Virtualbox version 5.2.12 r122571
• Stata 15.1 SE

. version
version 15.1

. !uname -a

Linux mauricio-vb 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

. clear all

. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1

. syntax, [foo(cilevel)]

. disp "`foo'"
95

. gegen y = sum(x)

. syntax, [foo(cilevel)]

. disp "`foo'"
95

from stata-gtools.

Can you try again? Gtools 1.0 just hit the develop branch. While it doesn't address this specifically, the plugin has changed enough that I'm hoping whatever internals were messing it up are not there anymore (on Stata 15 it should use a newer version of the plugin interface, for example; make sure you have version 1.0.0 by running which gtools).

from stata-gtools.

same same ...

. which gegen
/home/pp/ado/plus/g/gegen.ado
*! version 1.0.0 21Jul2018 Mauricio Caceres Bravo, [email protected]
*! implementation -egen- using C for faster processing

. clear all
. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. gegen y = sum(x)
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

from stata-gtools.

Btw is there a workaround or is this issue persistent? For example,

clear all
set obs 10
gen x = 1
syntax, [foo(cilevel)]
disp "`foo'"
gegen y = sum(x)
syntax, [foo(cilevel)]
syntax, [foo(cilevel)]
disp "`foo'"

or

clear all
program gtools_check, plugin using("gtools_unix_v3.plugin")
set obs 10
gen x = 1
syntax, [foo(cilevel)]
disp "`foo'"
gegen y = sum(x)
plugin call gtools_check, check
syntax, [foo(cilevel)]
disp "`foo'"

or

clear all
set obs 10
gen x = 1
syntax, [foo(cilevel)]
disp "`foo'"
gegen y = sum(x)
global GTOOLS_CALLER ghash
_gtools_internal
syntax, [foo(cilevel)]
disp "`foo'"

from stata-gtools.

sry, none of the above examples work. The second disp always shows 95,0.
Maybe I should cross-prost the issue at statalist.org?

from stata-gtools.

Well, it would make sense to crosspost if this wasn't specific to gtools, but regular egen, etc. calls don't have this issue, right? Can you download test1.plugin, test2.plugin, and spookyhash.dll from here. Then try to run:

clear all
program test1, plugin using(test1.plugin) 
plugin call test1
syntax, [foo(cilevel)]
disp "`foo'"

program test2, plugin using(test2.plugin) 
plugin call test2
syntax, [foo(cilevel)]
disp "`foo'"

global GTOOLS_CALLER ghash
_gtools_internal
syntax, [foo(cilevel)]
disp "`foo'"

Does the "95,0" show in after all 3 or just the last one?

from stata-gtools.

I have tried running the test plugins, but get the following error:

. which gtools
/home/pp/ado/plus/g/gtools.ado
*! version 1.0.2 08Aug2018 Mauricio Caceres Bravo, [email protected]
*! Program for managing the gtools package installation
. !ls -l
insgesamt 296
-rw-r--r-- 1 pp users    382 14. Aug 07:10 Makefile
-rw-r--r-- 1 pp users   7176 14. Aug 07:10 spookyhash_api.h
-rw-r--r-- 1 pp users  18944 14. Aug 07:10 spookyhash.dll
-rw-r--r-- 1 pp users    199 14. Aug 07:10 stplugin.c
-rw-r--r-- 1 pp users   5701 14. Aug 07:10 stplugin.h
-rw-r--r-- 1 pp users    433 14. Aug 07:10 test1.c
-rw-r--r-- 1 pp users 119132 14. Aug 07:10 test1.plugin
-rw-r--r-- 1 pp users    765 14. Aug 07:10 test2.c
-rw-r--r-- 1 pp users 120343 14. Aug 07:10 test2.plugin
-rw-r--r-- 1 pp users    123 14. Aug 07:10 test.do
. do "/tmp/SD10587.000000"
. clear all
. program test1, plugin using(test1.plugin) 
./test1.plugin: Ungültiger ELF-Header
Could not load plugin: ./test1.plugin
r(9999);
end of do-file
r(9999);

from stata-gtools.

Ok, that one is on me; really sorry. I had compiled that for Windows and I forgot you're also on Arch. I re-posted the files as a ZIP here. Can you try again? You can just do do test.do for the test script on that folder.

from stata-gtools.

this is the output:

. do test.do
. clear all
. program test1, plugin using(test1.plugin)
. plugin call test1
Hello World
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. 
. program test2, plugin using(test2.plugin)
. plugin call test2
foo:      0.00
hash: 12485269540145512313, 5018404763475985066
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. 
. set obs 1
number of observations (_N) was 0, now 1
. global GTOOLS_CALLER ghash
. _gtools_internal
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

from stata-gtools.

I was really hoping it was the plugin interface in general, but it seems to be something in gtools.

Can you add

syntax, [foo(cilevel)]
disp "`foo'"

right before and right after line 2157 of _gtools_internal.ado (it would be in your adopath, for me ~/ado/plus/_/_gtools_internal.ado) and re-run only the third chunk of the snippet with set trace on then post the log?

The idea is to know whether it's something in the Sata code or the C code.

from stata-gtools.

here is the log file: log.log

from stata-gtools.

I've come across a similar (the same?) issue while using ppmlhdfe and gegen in the same do-file: using gegen before ppmlhdfe causes ppmlhdfe to fail with "invalid syntax" while using egen causes no such problem. This issue only occurs when using Stata on our linux cluster; the code with gegen worked without issues on my MacOS X laptop. (This causes no problems for me as I can simply use egen, but I thought it might be helpful to know that the issue might be Linux-specific. Although of course many other variables also differ between the cluster and my laptop...)

from stata-gtools.

If you run ppmlhdfe with the verbose(1) option, or if you run "set trace" beforehand, where in the program is it reporting an error?

Ppmlhdfe doesn't use C plugins, so on principle it should be like any other programs

from stata-gtools.

Using verbose(1) does not change the output (it still just says "invalid syntax"). The output from set trace on is attached; I believe it fails in _get_diopts. (I don't think this is an issue with ppmlhdfe; rather I think ppmlhdfe in my example fails in the same way as reg fails above. The error is presumably in gegen, or even more low-level)

output.log

from stata-gtools.

For whatever reason it seems that there are corner cases where gegen causes the default cilevel to be set to 95,0 instead of just 95, leading to syntax errors.

It is very puzzling to me because I never set ci level in any gtools programs and I cannot reproduce the error (be it in my Linux laptop or the Linux servers I use).

from stata-gtools.

@ppoppitz @lmusolff If you still get this bug, can you try the develop branch? (gtools version 1.3.2; gtools, upgrade branch(develop))

I just got a report that I suspect is related (#53). It occurs to me that maybe if I save the state of level before running the plugin and restore it after then maybe I can bypass the problem.

from stata-gtools.

sry, I still get the same errors with version 1.3.2 14Feb2019

from stata-gtools.

@ppoppitz Can you try gtools version 1.3.3? gtools, upgrade branch(develop); I removed a dependency that might have been causing the problem. In #53 it looks like gegen no longer sets level to 95,0; hope it's the same for you.

from stata-gtools.

I verified the code from #53 gives me the same error as there with gtools 1.1.2, but when I try to upgrade I get the following error:

. gtools, upgrade branch(develop)
checking gtools consistency and verifying not already installed...
could not copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop

/build/fasterxtile.sthlp
(no action taken)
--Break--
r(1);

from stata-gtools.

Try net uninstall gtools and then

local github "https://raw.githubusercontent.com"
net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)

If you get the same error, what happens if you run

do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl

You should see

. .h gquantiles
invalid syntax
r(198);

Let me know if you don't.

from stata-gtools.

Here's the output from those commands:

. net uninstall gtools
package not found
r(111);

. local github "https://raw.githubusercontent.com"

. net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)
checking gtools consistency and verifying not already installed...
could not copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp
(no action taken)
--Break--
r(1);

. do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl
file https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl not found
r(601);

from stata-gtools.

That was a typo on my part, I meant do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp

In any case, maybe try this:

copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/src/github-issues/40/gtools-1.3.3.zip gtools.zip
unzipfile gtools.zip
net install gtools, from(`c(pwd)'/gtools/) replace

from stata-gtools.

The former gives me r(1), but the second successfully upgraded to "version 1.3.3 15Feb2019". And now I can run the code from #53 without an error :) (I'll try my own at some point but I doubt it would differ, and it is part of a big job that I can't really isolate so need to schedule a run for)

Thanks for looking into this!

from stata-gtools.

I think I'll ask Stata why that file can't be copied on some systems. You're not the first person to have this issue. Can you run

copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp fasterxtile.sthlp

and let me know the error code, if any?

from stata-gtools.

Yes, I vaguely remember having trouble installing it the normal way back when I did because I think of the same error. From that command I get:

. copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp fasterxtile.sthlp
--Break--
r(1);

EDIT: I just ran the same command on my MacOsX Laptop and it gives me the same r(1). But maybe this should be a different issue; let me know if you'd like me to open one.

from stata-gtools.

Can you re-try? I want to know if this had anything to do with the fact it's a tiny file (14 bytes). I just modified the file to be a copy of gquantiles.sthlp (what's your Stata version, by the way?)

from stata-gtools.

Oh wow, apparently that did it: the copy command executes now without problem (both on my laptop and the server). My Stata version is 13.1.

from stata-gtools.

Ok, great. I'm just going to have my build file copy gquantiles.sthlp into fasterxtile.sthlp and bypass this altogether. Thanks for testing it out!

from stata-gtools.

Of course, no worries! Thanks for writing & maintaining this awesome package!

from stata-gtools.

Awesome, it's working here too. Thanks again for your efforts!

from stata-gtools.