trainindata / deploying-machine-learning-models Goto Github PK

In the "Machine Learning Pipeline: Feature Engineering" video (Section 2) you mention that one of the assumptions of linear models is that variables should be normally distributed and this is not correct. This assumption applies to the errors and it affects to the calculation of confidence intervals.

Thanks for the course!

Error:
ImportError: cannot import name '_check_y' from 'sklearn.utils.validation' (/opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py)

--> from feature_engine.encoding import (
RareLabelEncoder,
OrdinalEncoder)

Could you please help to resolve?

ERROR tests/test_features.py::test_temporal_variable_transformer - KeyError: 'MSSubClass'
ERROR tests/test_prediction.py::test_make_prediction - KeyError: 'MSSubClass'

An update to the terms and conditions of the free version of Circle CI precludes setting docker_layer_caching: true. It will fail the tests on the deployment attempt to Heroku.

Recommend updating the circle ci config.yml code to reflect docker_layer_caching: false to allow forked repositories to step through the deployment.

Thanks for the awesome repository and course!

I am receiving the below error when I try to run the test code. Any help is greatly appreciated.

======================================================= ERRORS ========================================================
__________________________ ERROR collecting packages/regression_model/tests/test_predict.py ___________________________
packages\regression_model\tests\test_predict.py:3: in
from regression_model.predict import make_prediction
packages\regression_model\regression_model\predict.py:9: in
_price_pipe = load_pipeline(file_name=pipeline_file_name)
packages\regression_model\regression_model\processing\data_management.py:29: in load_pipeline
saved_pipeline = joblib.load(filename=file_path)
env\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py:590: in load
with open(filename, 'rb') as f:
E FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\ravit\Documents\Git\deploying-machine-learning-models\packages\regression_model\regression_model\trained_models\regression_model.pkl'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 1 error in 0.65 seconds ===============================================

Using base prefix '/usr/local'

New python executable in /home/circleci/project/venv/bin/python

Installing setuptools, pip, wheel...

done.

Looking in indexes: https://pypi.org/simple, https://****@pypi.fury.io/***********/

Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages (20.1.1)

Looking in indexes: https://pypi.org/simple, https://@pypi.fury.io/***********/, https://@pypi.fury.io/***********/

Collecting flask==1.0.2

Downloading Flask-1.0.2-py2.py3-none-any.whl (91 kB)

 |████████████████████████████████| 91 kB 7.1 MB/s eta 0:00:011

Collecting marshmallow==2.17.0

Downloading marshmallow-2.17.0-py2.py3-none-any.whl (49 kB)

 |████████████████████████████████| 49 kB 9.9 MB/s  eta 0:00:01

ERROR: Could not find a version that satisfies the requirement regression-model==0.1.0 (from -r packages/ml_api/requirements.txt (line 10)) (from versions: none)

ERROR: No matching distribution found for regression-model==0.1.0 (from -r packages/ml_api/requirements.txt (line 10))

Exited with code exit status 1
CircleCI received exit code 1

I am facing this error while executing the test script

=================================== ERRORS ====================================
______ ERROR collecting packages/regression_model/tests/test_predict.py _______
ImportError while importing test module 'D:\deploy\deploying-machine-learning-models\packages\regression_model\tests\test_predict.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
packages\regression_model\tests\test_predict.py:3: in
from regression_model.predict import make_prediction
packages\regression_model\regression_model\predict.py:9: in
_price_pipe = load_pipeline(file_name=pipeline_file_name)
packages\regression_model\regression_model\processing\data_management.py:29: in load_pipeline
saved_pipeline = joblib.load(filename=file_path)
env\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py:598: in load
obj = _unpickle(fobj, filename, mmap_mode)
env\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py:526: in _unpickle
obj = unpickler.load()
C:\Users\user\Anaconda3\lib\pickle.py:1085: in load
dispatchkey[0]
C:\Users\user\Anaconda3\lib\pickle.py:1373: in load_global
klass = self.find_class(module, name)
C:\Users\user\Anaconda3\lib\pickle.py:1423: in find_class
import(module, level=0)
E ModuleNotFoundError: No module named 'preprocessors'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 4.65 seconds ===========================

rootdir: D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package, configfile: pyproject.toml
collected 2 items

tests/test_features.py::test_temporal_variable_transformer PASSED
tests/test_prediction.py::test_make_prediction PASSED

============================================================================================================ 2 passed in 0.47s =============================================================================================================
typechecks installed: appdirs==1.4.4,atomicwrites==1.4.0,attrs==21.4.0,black==20.8b1,click==8.1.3,colorama==0.4.4,feature-engine==1.0.2,flake8==3.9.2,iniconfig==1.1.1,isort==5.8.0,joblib==1.0.1,mccabe==0.6.1,mypy==0.812,mypy-extensions==0.4.3,numpy==1.20.3,packaging==21.3,pandas==1.3.5,pathspec==0.9.0,patsy==0.5.2,pluggy==1.0.0,py==1.11.0,pycodestyle==2.7.0,pydantic==1.8.2,pyflakes==2.3.1,pyparsing==3.0.8,pytest==6.2.5,python-dateutil==2.8.2,pytz==2022.1,regex==2022.4.24,ruamel.yaml==0.16.12,ruamel.yaml.clib==0.2.6,scikit-learn==1.0.2,scipy==1.8.0,six==1.16.0,statsmodels==0.13.2,strictyaml==1.3.2,threadpoolctl==3.1.0,toml==0.10.2,typed-ast==1.4.3,typing_extensions==4.2.0
typechecks run-test-pre: PYTHONHASHSEED='220'
typechecks run-test: commands[0] | mypy regression_model
Success: no issues found in 12 source files
stylechecks installed: appdirs==1.4.4,atomicwrites==1.4.0,attrs==21.4.0,black==20.8b1,click==8.1.3,colorama==0.4.4,feature-engine==1.0.2,flake8==3.9.2,iniconfig==1.1.1,isort==5.8.0,joblib==1.0.1,mccabe==0.6.1,mypy==0.812,mypy-extensions==0.4.3,numpy==1.20.3,packaging==21.3,pandas==1.3.5,pathspec==0.9.0,patsy==0.5.2,pluggy==1.0.0,py==1.11.0,pycodestyle==2.7.0,pydantic==1.8.2,pyflakes==2.3.1,pyparsing==3.0.8,pytest==6.2.5,python-dateutil==2.8.2,pytz==2022.1,regex==2022.4.24,ruamel.yaml==0.16.12,ruamel.yaml.clib==0.2.6,scikit-learn==1.0.2,scipy==1.8.0,six==1.16.0,statsmodels==0.13.2,strictyaml==1.3.2,threadpoolctl==3.1.0,toml==0.10.2,typed-ast==1.4.3,typing_extensions==4.2.0
stylechecks run-test-pre: PYTHONHASHSEED='220'
stylechecks run-test: commands[0] | flake8 regression_model tests
lint installed: appdirs==1.4.4,atomicwrites==1.4.0,attrs==21.4.0,black==20.8b1,click==8.1.3,colorama==0.4.4,feature-engine==1.0.2,flake8==3.9.2,iniconfig==1.1.1,isort==5.8.0,joblib==1.0.1,mccabe==0.6.1,mypy==0.812,mypy-extensions==0.4.3,numpy==1.20.3,packaging==21.3,pandas==1.3.5,pathspec==0.9.0,patsy==0.5.2,pluggy==1.0.0,py==1.11.0,pycodestyle==2.7.0,pydantic==1.8.2,pyflakes==2.3.1,pyparsing==3.0.8,pytest==6.2.5,python-dateutil==2.8.2,pytz==2022.1,regex==2022.4.24,ruamel.yaml==0.16.12,ruamel.yaml.clib==0.2.6,scikit-learn==1.0.2,scipy==1.8.0,six==1.16.0,statsmodels==0.13.2,strictyaml==1.3.2,threadpoolctl==3.1.0,toml==0.10.2,typed-ast==1.4.3,typing_extensions==4.2.0
lint run-test-pre: PYTHONHASHSEED='220'
lint run-test: commands[0] | isort regression_model tests
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file regression_model\pipeline.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file regression_model\predict.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file regression_model_init_.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file regression_model\processing\data_manager.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file regression_model\processing\validation.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file tests\conftest.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file tests\test_features.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\isort\main.py:107: UserWarning: Unable to parse file tests\test_prediction.py due to 'gbk' codec can't decode byte 0x99 in position 471: illegal multibyte sequence
warn(f"Unable to parse file {file_name} due to {error}")
lint run-test: commands[1] | black regression_model tests
Traceback (most recent call last):
File "C:\Users\songyq\anaconda3\lib\runpy.py", line 194, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\songyq\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\Scripts\black.EXE_main.py", line 7, in
File "D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\black_init.py", line 6606, in patched_main
patch_click()
File "D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\black_init.py", line 6595, in patch_click
from click import _unicodefun # type: ignore
ImportError: cannot import name 'unicodefun' from 'click' (D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\lib\site-packages\click_init.py)
ERROR: InvocationError for command 'D:\Users\songyq\Documents\s_scripts\Udemy_Deployment_of_Machine_Learning_Models\section-05-production-model-package.tox\test_package\Scripts\black.EXE' regression_model tests (exited with code 1)

this can be fixed by change the "section-05-production-model-package\requirements\test_requirements.txt"
black==22.3.0
similar issue : https://stackoverflow.com/questions/71673404/importerror-cannot-import-name-unicodefun-from-click

• While running tox the mypy classification_model checks fail.
• Error:

test_package: OK ✔ in 3.99 seconds
checks: commands[0]> flake8 classification_model tests
checks: commands[1]> isort classification_model tests
Fixing C:\Users\krish\titanicSurvival\production-model-package\classification_model\pipeline.py
checks: commands[2]> black classification_model tests
checks: commands[3]> mypy classification_model
classification_model\config\core.py: note: In function "fetch_config_from_yaml":
classification_model\config\core.py:57: error: Incompatible default for argument "cfg_path" (default has type "None", argument has type "Path")  [assignment]
classification_model\config\core.py:57: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
classification_model\config\core.py:57: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
Found 1 error in 1 file (checked 12 source files)
checks: exit 1 (0.50 seconds) C:\Users\krish\titanicSurvival\production-model-package> mypy classification_model pid=28348
  test_package: OK (3.99=setup[0.06]+cmd[1.76,2.16] seconds)
  checks: FAIL code 1 (2.26=setup[0.04]+cmd[0.86,0.28,0.58,0.50] seconds)
  evaluation failed :( (6.78 seconds)

• Update the [testenv:checks] in tox.ini as following:

[testenv:checks]
envdir = {toxworkdir}/checks
deps =
	-r{toxinidir}/requirements/typing_requirements.txt
commands =
	flake8 classification_model tests
	isort classification_model tests
	black classification_model tests
	{posargs:mypy classification_model --implicit-optional}

• Note that we added --implicit-optional in the last command: posargs
• Results:

checks: commands[3]> mypy classification_model --implicit-optional
Success: no issues found in 12 source files
  test_package: OK (3.81=setup[0.09]+cmd[1.72,2.00] seconds)
  checks: OK (3.25=setup[0.02]+cmd[0.81,0.27,0.58,1.58] seconds)
  congratulations :) (7.66 seconds)

I had to add the pytest-cov module into test_requirements.text in order to see the missing lines covered by the tests. However, by running tox -e <my_env> it was not installed, the environment was not upgraded. According to the official bug, it happens since test_requirements.txt includes another requirement installation and it could be fixed in a future version (follow the thread here #93) Therefore, just in case you come across this issue, you can fix it in two manners:

1. include pip module in deps section, just like this:

[testenv:test_package]
deps =
    pip
    -rrequirements/test_requirements.txt

2. To remove -r requirements line from test_requirements.txt and specify this one as a dependency (I chose this one)

[testenv:test_package]
deps =
    -rrequirements/requirements.txt
    -rrequirements/test_requirements.txt

3. (The worst) Deleting the target environment directory directly from .tox folder and running tox command again to recreate it, but it will bring some additional costs.

I've encounter that when downloading the test and train datasets from kaggle from the fetch_kaggle_dataset.sh returns a zip file so in result the model can't be trained. I've found a solution by specifying the -f argument in the command to catch the train.csv and test.csv precisely. So in my version the script follows like this:

kaggle competitions download -c house-prices-advanced-regression-techniques -f train.csv -p packages/regression_model/regression_model/datasets/
kaggle competitions download -c house-prices-advanced-regression-techniques -f test.csv -p packages/regression_model/regression_model/datasets/

Hello,
Thanks for your course, I have a problem at lesson 11.5, could you help me?
The circle CI announce this error at the last step

#!/bin/bash -eo pipefail
heroku container:release web --app $HEROKU_APP_NAME
▸ Expected response to be
▸ successful, got 404
Exited with code 1