The Python for Scientific Computing app is an app that bundles Miniconda (a commercial binary distribution of Python) and some common Python libraries for scientific computing: numpy, scipy, pandas, scikit-learn, statsmodels.
This repo builds PSC for 3 platforms:
- Mac: https://splunkbase.splunk.com/app/2881/
- Linux 64-bit: https://splunkbase.splunk.com/app/2882/
- Windows 64-bit: https://splunkbase.splunk.com/app/2883/
- Update
packages.txt
in the repo root dir- Always specify your
python
version, keep major version consistent with the target splunk platform. e.g. Splunk 8.0.x ships with python 3.7.x, so putpython==3.7.*
in there or the specific version you need - Fix the major version, leave the minor version flexible, i.e.
pandas==0.25.*
- There's another
packages.txt
inside of each platform's folder, which act as the lock file to lockdown the specific versions of all dependencies (likepip freeze
).
- Always specify your
- Run freeze task of the build scripts for each platform and push the updated versions of platform/packages.txt to the repo.
- There are two build scripts to be used on different platforms
repack.sh
for Linux and OSXrepack.ps1
for Windows
- run
bash repack.sh freeze
- There are two build scripts to be used on different platforms
- If there's any package ADDED to existing
packages.txt
, we can runbash repack.sh analyze
to inspect the package dependency tree - Optional, check licenses, if you are redistributing this app, you need
to include the proper licenses of the package redistributed, run
bash repack.sh license
to generate a license file- Note you may need to update
license_db.csv
if you included a new package
- Note you may need to update
- Update the requirements.txt's content with the linux_x86_64/packages.txt for prodsec-review scans.
- Finally, when the platform specific
packages.txt
looks good, runbash repack.sh build
to build the Python for Scientific Computing app - Copy it to your
$SPLUNK_HOME/etc/apps
folder- Note, due to the size of this app, installing it via web installer/deployer may fail with a timeout error
You may want to work in PSC environment for development purposes. For this step, you need to have conda installed on your machine to be able to customize this PSC environment. You can install conda from https://docs.conda.io/en/latest/miniconda.html for your platform.
In order to use PSC as conda environment, you can build it in build-dev
mode. Follow the steps below:
- Run this command from the root folder:
./repack.sh build-dev
. It might take a while to build the dev folder. - When the dev folder is created, either move the build folder (
Splunk_SA_Scientific_Python_${PLATFORM}
)
under$SPLUNK_HOME/etc/apps
location to link MLTK with your custom PSC or upload the tarball from the UI. - Once PSC is installed, you can activate this PSC as a conda environment using the below command for Mac.
You can install conda from https://docs.conda.io/en/latest/miniconda.html for your platform.:
for linux,
conda activate $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_darwin_x86_64/bin/darwin_x86_64
conda activate $SPLUNK_HOME/etc/apps/Splunk_SA_Scientific_Python_linux_x86_64/bin/linux_x86_64
- Now you can customize your PSC Python environment. Make sure to install pip first using
conda install -c conda-forge pip
- You can install additional packages now using
conda install
orpip install
and once you refresh MLTK, your changes should be picked automatically. - To deactivate the PSC environment, run
conda deactivate
from the terminal.
There may be unwanted content in the build that will fail the SplunkBase App verification, e.g. hidden files. We can use Splunk Package Toolkit CLI to facilitate the packaging process.
# Get Splunk
wget http://releases.splunk.com/released_builds/8.0.5/splunk/osx/splunk-8.0.5-a1a6394cc5ae-darwin-64.tgz
tar -xf splunk-8.0.5-a1a6394cc5ae-darwin-64.tgz
# Run package command on the app folder
./splunk/bin/slim package ${APPDIR}_${PLATFORM}
If you are releasing a new PSC, you need follow these steps before running the repack scripts:
- Edit the value of
VERSION
variable at the top of the repack scripts. - Set the values of
MINICONDA_VERSION
,LINUX_MD5
, andOSX_MD5
variables to the appropriate value. Note that for OSX we use the.sh
installer and not the.pkg
installer. For the list of available miniconda versions, their respective package names, and MD5 hash, see https://repo.anaconda.com/miniconda/. - Update the README file under
package
directory with the correct version of included packages.