facebookarchive / bootstrapped Goto Github PK

Generate bootstrapped confidence intervals for A/B testing in Python.

License: Other

Python 99.87% Shell 0.13%

bootstrapped's Introduction

bootstrapped - confidence intervals made easy

bootstrapped is a Python library that allows you to build confidence intervals from data. This is useful in a variety of contexts - including during ad-hoc a/b test analysis.

Motivating Example - A/B Test

Imagine we own a website and think changing the color of a 'subscribe' button will improve signups. One method to measure the improvement is to conduct an A/B test where we show 50% of people the old version and 50% of the people the new version. We can use the bootstrap to understand how much the button color improves responses and give us the error bars associated with the test - this will give us lower and upper bounds on how good we should expect the change to be!

The Gist - Mean of a Sample

Given a sample of data - we can generate a bunch of new samples by 're-sampling' from what we have gathered. We calculate the mean for each generated sample. We can use the means from the generated samples to understand the variation in the larger population and can construct error bars for the true mean.

bootstrapped - Benefits

Efficient computation of confidence intervals
Functions to handle single populations and a/b tests
Functions to understand statistical power
Multithreaded support to speed-up bootstrap computations
Dense and sparse array support

Example Usage

import numpy as np
import bootstrapped.bootstrap as bs
import bootstrapped.stats_functions as bs_stats

mean = 100
stdev = 10

population = np.random.normal(loc=mean, scale=stdev, size=50000)

# take 1k 'samples' from the larger population
samples = population[:1000]

print(bs.bootstrap(samples, stat_func=bs_stats.mean))
>> 100.08  (99.46, 100.69)

print(bs.bootstrap(samples, stat_func=bs_stats.std))
>> 9.49  (9.92, 10.36)

Extended Examples

Bootstrap Intro
Bootstrap A/B Testing
More notebooks can be found in the examples/ directory

Requirements

bootstrapped requires numpy. The power analysis functions require matplotlib and pandas.

Installation

pip install bootstrapped

How bootstrapped works

bootstrapped provides pivotal (aka empirical) based confidence intervals based on bootstrap re-sampling with replacement. The percentile method is also available.

For more information please see:

Bootstrap confidence intervals (good intro)
An introduction to Bootstrap Methods
The Bootstrap, Advanced Data Analysis
When the bootstrap dosen't work
(book) An Introduction to the Bootstrap
(book) Bootstrap Methods and their Application

See the CONTRIBUTING file for how to help out.

Contributors

Spencer Beecher, Don van der Drift, David Martin, Lindsay Vass, Sergey Goder, Benedict Lim, and Matt Langner.

Special thanks to Eytan Bakshy.

License

bootstrapped is BSD-licensed. We also provide an additional patent grant.

bootstrapped's People

Contributors

Stargazers

Watchers

Forkers

brianb1 samspaulding vibster sam-bristow benthomasson yunweidashuju ml-ai-nlp-ir malayandi spencebeecher rlugojr rezaprimasatya keizar901 jdrew1303 pzdispenser zizanno achoczaj damonclifford graingert joegosse edx iamaris wukaiyuan andytwoods matheusrabetti geoblog esyrt chanansh andreuvall pealco nicoring mazaixing n7nx sbarman-mi9 ganjingcatherine gpfreitas ondrejtichacek salma-bouzid gsecoguy sindhu819 bfgray3 batermj zichaohuang vcgato29 awalkinclouds neonzliy tsarsguard dilemmalab mdhbh wangzijian123963 cattystats porthcawl-insurance jbinary bulatkhamitov lambdaofgod chanedwin jessicadufirst t4r7s jaeyk mysticaltech skyan fintrek 5l1v3r1 eliseygusev geangohn valiot christianjauregui d-e-f-e-a-t alexandertrg nkm-ml jsunseri himanshukhairnaraffine manohar8642 cyw-wayne blaylockbk jsh9 nrapanos delislecl quillan86 meliache shyla-kupis doslin tianlei822 bjrnfrdnnd zeddhua2077 swifilaboroka lkunc bikegeek isabella232 uchpuchmak kamelcased d1bevz kvazzart eliekawerk vbiff irinaboyarchukova athenianco arpitjain799 mikhailbatyl oxanagolodyuk juliapatsyukova

bootstrapped's Issues

Clarification on scale_test_by on bootstrap_ab

I noticed one of the parameters in bootstrap_ab is scale_test_by. The docstring reads:

scale_test_by: The ratio between test and control population
sizes. Use this if your test and control split is different from a
50/50 split. Defaults to 1.0.

However, looking at the implementation, it's using this ratio to multiply the whole dist. Is this value only needed when using stat_functions.sum? It's not clear to me why I would scale the values on my test sample when I am testing the mean on both samples.

test_ctrl_dist = compare_func(test_dist * scale_test_by, ctrl_dist)

Multi-threading does not work but easy fix

Hey,
this is a super useful package!
I needed to increase the speed of the code and set num_threads = -1 (any number other than 1 will break the code).
In bootstrap.py in line 272: "for seed in _np.random.randint(0, 232 - 1, num_threads):" will break as it exceeds the max signed int. If you change it to 231 - 1 it will work fine.
Thanks!

install freeze on building wheels for existing packages

Collecting bootstrapped
  Using cached https://files.pythonhosted.org/packages/bf/5a/9a41920ee5403625216b9c3c77880534aa7c3afee45745d4abf0cabdacf2/bootstrapped-0.0.1-py2.py3-none-any.whl
Collecting pandas==0.18.1 (from bootstrapped)
  Using cached https://files.pythonhosted.org/packages/11/09/e66eb844daba8680ddff26335d5b4fead77f60f957678243549a8dd4830d/pandas-0.18.1.tar.gz
Collecting matplotlib==1.5.3 (from bootstrapped)
  Using cached https://files.pythonhosted.org/packages/2d/de/8063c21616ac644e58f2bfd99f2e09f75870b545c2788b9aebacb080c463/matplotlib-1.5.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting numpy==1.11.1 (from bootstrapped)
  Using cached https://files.pythonhosted.org/packages/e0/4c/515d7c4ac424ff38cc919f7099bf293dd064ba9a600e1e3835b3edefdb18/numpy-1.11.1.tar.gz
Requirement already satisfied: python-dateutil>=2 in /Users/hanans/anaconda3/lib/python3.6/site-packages (from pandas==0.18.1->bootstrapped)
Requirement already satisfied: pytz>=2011k in /Users/hanans/anaconda3/lib/python3.6/site-packages (from pandas==0.18.1->bootstrapped)
Requirement already satisfied: pyparsing!=2.0.0,!=2.0.4,!=2.1.2,>=1.5.6 in /Users/hanans/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped)
Requirement already satisfied: cycler in /Users/hanans/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped)
Requirement already satisfied: six>=1.5 in /Users/hanans/anaconda3/lib/python3.6/site-packages (from python-dateutil>=2->pandas==0.18.1->bootstrapped)
Building wheels for collected packages: pandas, numpy
  Running setup.py bdist_wheel for pandas ... |

Throw warning when arrays with the wrong shape are passed

[[1], [2], [1], [2]] - throw an error
[1, 2, 1, 2] - well formatted

Submit to PyPI

Being able to run pip install bootstrapped would be great :).

np.random seed and multi-thread produces the same results

Empirical bootstrap instead of percentile

Implement emperical/pivotal bootstrap.
https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf
http://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf

Num / Denom samples can be different records

https://github.com/facebookincubator/bootstrapped/blob/master/bootstrapped/bootstrap.py#L78

Make this function return both numerator and denominator values so that the sampled indexes are aligned

Installing through pip fails

I've been trying to install from pip and have been getting the following errors:

Requirement already satisfied: bootstrapped in /Users/bschapma/anaconda3/lib/python3.6/site-packages (0.0.1)
Collecting numpy==1.11.1 (from bootstrapped)
Using cached https://files.pythonhosted.org/packages/e0/4c/515d7c4ac424ff38cc919f7099bf293dd064ba9a600e1e3835b3edefdb18/numpy-1.11.1.tar.gz
Collecting matplotlib==1.5.3 (from bootstrapped)
Using cached https://files.pythonhosted.org/packages/2d/de/8063c21616ac644e58f2bfd99f2e09f75870b545c2788b9aebacb080c463/matplotlib-1.5.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting pandas==0.18.1 (from bootstrapped)
Requirement already satisfied: pyparsing!=2.0.0,!=2.0.4,!=2.1.2,>=1.5.6 in /Users/bschapma/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped) (2.2.0)
Requirement already satisfied: cycler in /Users/bschapma/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped) (0.10.0)
Requirement already satisfied: pytz in /Users/bschapma/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped) (2018.4)
Requirement already satisfied: python-dateutil in /Users/bschapma/anaconda3/lib/python3.6/site-packages (from matplotlib==1.5.3->bootstrapped) (2.7.3)
Requirement already satisfied: six in /Users/bschapma/anaconda3/lib/python3.6/site-packages (from cycler->matplotlib==1.5.3->bootstrapped) (1.11.0)
Building wheels for collected packages: numpy
Running setup.py bdist_wheel for numpy ... error
Complete output from command /Users/bschapma/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-wheel-2l7zpfeg --python-tag cp36:
Running from numpy source directory.
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in ['/Users/bschapma/anaconda3/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE

openblas_info:
libraries openblas not found in ['', '']
Runtime library openblas was not found. Ignoring
FOUND:
libraries = ['openblas', 'openblas']
library_dirs = ['/Users/bschapma/anaconda3/lib']
language = c
define_macros = [('HAVE_CBLAS', None)]

FOUND:
  libraries = ['openblas', 'openblas']
  library_dirs = ['/Users/bschapma/anaconda3/lib']
  language = c
  define_macros = [('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2
lapack_opt_info:
openblas_lapack_info:
customize UnixCCompiler
C compiler: x86_64-apple-darwin13.4.0-clang -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9

creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders/0w
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0
compile options: '-c'
x86_64-apple-darwin13.4.0-clang: /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/source.c
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/source.c:1:12: warning: this function declaration is not a prototype [-Wstrict-prototypes]
void zungqr();
^
void
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/source.c:4:13: warning: implicit declaration of function 'zungqr_' is invalid in C99 [-Wimplicit-function-declaration]
zungqr_();
^
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/source.c:4:13: warning: this function declaration is not a prototype [-Wstrict-prototypes]
3 warnings generated.
x86_64-apple-darwin13.4.0-clang -arch x86_64 /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/source.o -L/Users/bschapma/anaconda3/lib -lopenblas -o /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmp3gi5geh0/a.out
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: in '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd', file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: in '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd', file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
NOT AVAILABLE

lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in ['/Users/bschapma/anaconda3/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE

NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
libraries tatlas,tatlas not found in /Users/bschapma/anaconda3/lib
libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
libraries tatlas,tatlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries tatlas,tatlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
NOT AVAILABLE

atlas_3_10_info:
libraries satlas,satlas not found in /Users/bschapma/anaconda3/lib
libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
libraries satlas,satlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries satlas,satlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in /Users/bschapma/anaconda3/lib
libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
NOT AVAILABLE

atlas_info:
libraries f77blas,cblas,atlas not found in /Users/bschapma/anaconda3/lib
libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_info'>
NOT AVAILABLE

FOUND:
  extra_compile_args = ['-msse3']
  extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
  define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
running bdist_wheel
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
creating build
creating build/src.macosx-10.9-x86_64-3.6
creating build/src.macosx-10.9-x86_64-3.6/numpy
creating build/src.macosx-10.9-x86_64-3.6/numpy/distutils
building library "npymath" sources
customize Gnu95FCompiler
Found executable /Users/bschapma/anaconda3/bin/gfortran
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ld
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ar
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ranlib
customize Gnu95FCompiler
customize Gnu95FCompiler using config
C compiler: x86_64-apple-darwin13.4.0-clang -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/bschapma/anaconda3/include/python3.6m -c'
x86_64-apple-darwin13.4.0-clang: _configtest.c
x86_64-apple-darwin13.4.0-clang -arch x86_64 _configtest.o -o _configtest
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py", line 386, in
setup_package()
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py", line 378, in setup_package
setup(**metadata)
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/core.py", line 169, in setup
return old_setup(**new_attr)
File "/Users/bschapma/anaconda3/lib/python3.6/site-packages/setuptools/init.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Users/bschapma/anaconda3/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 202, in run
self.run_command('build')
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build.py", line 47, in run
old_build.run(self)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 147, in run
self.build_sources()
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 158, in build_sources
self.build_library_sources(*libname_info)
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 293, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 376, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 654, in get_mathlib_info
raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

Failed building wheel for numpy
Running setup.py clean for numpy
Complete output from command /Users/bschapma/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" clean --all:
Running from numpy source directory.

setup.py clean is not supported, use one of the following instead:

- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
                    files that aren't checked into the git repo)

Add --force to your command to use it anyway if you must (unsupported).

Failed cleaning build dir for numpy
Failed to build numpy
Installing collected packages: numpy, matplotlib, pandas
Found existing installation: numpy 1.14.3
Uninstalling numpy-1.14.3:
Successfully uninstalled numpy-1.14.3
Running setup.py install for numpy ... error
Complete output from command /Users/bschapma/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-record-s382sjog/install-record.txt --single-version-externally-managed --compile:
Running from numpy source directory.

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last Numpy release on PyPi)


blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in ['/Users/bschapma/anaconda3/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['', '']
Runtime library openblas was not found. Ignoring
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/Users/bschapma/anaconda3/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/Users/bschapma/anaconda3/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2
lapack_opt_info:
openblas_lapack_info:
customize UnixCCompiler
C compiler: x86_64-apple-darwin13.4.0-clang -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9

creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders/0w
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T
creating /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1
compile options: '-c'
x86_64-apple-darwin13.4.0-clang: /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/source.c
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/source.c:1:12: warning: this function declaration is not a prototype [-Wstrict-prototypes]
void zungqr();
           ^
            void
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/source.c:4:13: warning: implicit declaration of function 'zungqr_' is invalid in C99 [-Wimplicit-function-declaration]
            zungqr_();
            ^
/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/source.c:4:13: warning: this function declaration is not a prototype [-Wstrict-prototypes]
3 warnings generated.
x86_64-apple-darwin13.4.0-clang -arch x86_64 /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/source.o -L/Users/bschapma/anaconda3/lib -lopenblas -o /var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/tmpp2606oo1/a.out
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: in '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd', file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: in '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd', file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.tbd for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
  NOT AVAILABLE

lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in ['/Users/bschapma/anaconda3/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
  libraries tatlas,tatlas not found in /Users/bschapma/anaconda3/lib
  libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
  libraries tatlas,tatlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries tatlas,tatlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
  libraries satlas,satlas not found in /Users/bschapma/anaconda3/lib
  libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
  libraries satlas,satlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries satlas,satlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /Users/bschapma/anaconda3/lib
  libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
  libraries f77blas,cblas,atlas not found in /Users/bschapma/anaconda3/lib
  libraries lapack_atlas not found in /Users/bschapma/anaconda3/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

  FOUND:
    extra_compile_args = ['-msse3']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
building library "npymath" sources
customize Gnu95FCompiler
Found executable /Users/bschapma/anaconda3/bin/gfortran
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ld
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ar
Found executable /Users/bschapma/anaconda3/bin/x86_64-apple-darwin13.4.0-ranlib
customize Gnu95FCompiler
customize Gnu95FCompiler using config
C compiler: x86_64-apple-darwin13.4.0-clang -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/Users/bschapma/anaconda3/include/python3.6m -c'
x86_64-apple-darwin13.4.0-clang: _configtest.c
x86_64-apple-darwin13.4.0-clang -arch x86_64 _configtest.o -o _configtest
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
ld: warning: ignoring file /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd, file was built for unsupported file format ( 0x2D 0x2D 0x2D 0x20 0x21 0x74 0x61 0x70 0x69 0x2D 0x74 0x62 0x64 0x2D 0x76 0x33 ) which is not the architecture being linked (x86_64): /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.tbd
ld: dynamic main executables must link with libSystem.dylib for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py", line 386, in <module>
    setup_package()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py", line 378, in setup_package
    setup(**metadata)
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/Users/bschapma/anaconda3/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/install.py", line 62, in run
    r = self.setuptools_run()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
    return distutils_install.run(self)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build.py", line 47, in run
    old_build.run(self)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/bschapma/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 147, in run
    self.build_sources()
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 158, in build_sources
    self.build_library_sources(*libname_info)
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 293, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/numpy/distutils/command/build_src.py", line 376, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 654, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

----------------------------------------

Rolling back uninstall of numpy
Command "/Users/bschapma/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-record-s382sjog/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/0w/h96n74f572zd5kcl4c91xxbr3k14tm/T/pip-install-dz7533sp/numpy/

not quite sure why lift is applied to the data generation

In this example lift is first applied to the data and then to the treatment variable. I am not sure why lift is applied to the data:
data = np.random.normal(loc=100, scale=100, size=data_size) * lift

Which bootstrap method?

Hi everyone,

I am trying to understand how this library works.

Does it use a percentile-based (PB), bias-corrected percentile-based (BC-PB) or accelerated bias-corrected percentile-based (BCa-PB) bootstrap method?

What about the pivotal method? Is it the "non-studentized pivotal (basic)” or the "basic percentile" bootstrap method?

Please, let me know.

Thanks,

Federico

Prediction intervals

An implementation of prediction intervals computation (instead of confidence intervals) would be much appreciated. Is it possible to add the functionality to the library, or would it be out of scope?

I am assuming there is some core functionality shared between the two (CIs and PIs), at least the bootstrapping itself should be the same, and then there are some differences in the calculations of the intervals.

I have found this explanation useful.

Using this utility on multi-dimensional data

Hi Folks,

I am trying to use this on X (m x n) as follows:

mean_results = bs.bootstrap(X, stat_func=bs_stats.mean)

Is there an axis parameter to say along which dimension to bootstrap X.

Permutation tests

https://onlinecourses.science.psu.edu/stat464/node/35

Memory leak

Thanks for the package! Looks really useful for A/B tests that I'm running.

I tried to run your example code
I executed the following line:
print(bs.bootstrap_ab(test, ctrl, bs_stats.mean, bs_compare.percent_change))
and it consumed a few GB of memory before having an exception of MemoryError.

I'm on Ubuntu 16.04, python 2.7, tell me any other specs you need.
Any guess what is going wrong?

Investigate int vs float speedup

'perhaps there is another speedup that can be done with int multiplication instead of float'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.