Comments (4)
Good that you mention this, it is bothering me as well 😀 note that you can already economize a bit by setting 'store_date=False' and 'copy_data=False', though this still saves a range of large-memory attributes.
I suppose an argument 'lean=True' should also drop the Y and X attributes and garbage collection memory.
Should be an easy enough addition =)
from pyfixest.
Context
Quite a lot of large objects are stored in Feols
/ Feiv
/ Fepois
objects. To avoid out-of-memory errors when working with big data sets, we want to add a function argument lean
to all three classes mentioned above and the feols()
and fepois()
APIs.
Task
Add the end of the run_all_models method of the FixestMulti
class, if lean = True
, set the following attributes to None:
- self._X
- self._Y
- self._Z
- self._cluster_df
- self._data
In code, do something as
import gc
if lean:
del self._X
del self._Y
del self._Z
del self._cluster_df
del self._data
gc.collect()
from pyfixest.
👏
from pyfixest.
Done 👍
You can now specify a lean
function argument:
%load_ext autoreload
%autoreload 2
import pyfixest as pf
data = pf.get_data()
fit = pf.feols("Y ~ X1", data = data, lean = True)
hasattr(fit, "_X")
# False
from pyfixest.
Related Issues (20)
- Wildboottest: Incorrect Initiation of R vector -> incorrect results
- Contributing Documentation does not show full code blocks HOT 2
- Contributing Documentation Formatting Issue
- Extract covariance matrix from regression object HOT 8
- F-test in pyfixest HOT 9
- narwhals = Pandas + polars + ... HOT 5
- Helping demean() Fail Gracefully - Add a'fixef_iter' argument to feols() and fepois() HOT 1
- Examples Layout Docs HOT 3
- Feiv : More diagnostic tests on the first stage regression need to be added HOT 2
- Add note on `wald_test` method to `quickstart` (or even a new notebook)? HOT 2
- Publish `pyfixest` on conda
- Add `solver` argument to `feols()`, `fepois()` APIs HOT 5
- Support for weights as an optional parameter for did2s? HOT 4
- Add benchmarks against `linearmodels` and `fastreg` HOT 8
- Add default IV Diagnostics to `pf.summary()` and `pf.etable()` HOT 1
- installing via pip broken by stargazer HOT 4
- Adding Standard Error of Prediction HOT 6
- BUG: coefplot() and iplot() show only point estimates, no confidence intervals HOT 1
- Implementing outcome plot in panelview()
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyfixest.