sitestat
sitestat tool
sitestat tool designed to catch statistics from various CMS sites. The underlying process follow these steps:
- Fetch all site names from SiteDB
- loop over specific time range, e.g. last 3m
- create dates for that range
- Use popularity API (DSStatInTImeWindow) to get summary statistics. The API returns various information about dataset usage on sites.
- Organize data in number of access bins
- For every bin collect dataset names
- Call DBS APIs to get dataset statistics via blocksummaries API.
- sum up info about file_size which will give total size used by specific site.
Here is example of sitestat tool usage
Usage of ./sitestat:
-bins string
Comma separated list of bin values, e.g. 0,1,2,3,4 for naccesses or 0,10,100 for tot cpu metrics
-blkinfo
Use block information for finding statistics, by default use dataset info
-breakdown string
Breakdown report into more details (tier, dataset)
-chunkSize int
chunkSize for processing URLs (default 100)
-dbsinfo
Use DBS to collect dataset information, default use PhEDEx
-format string
Output format type, txt or json (default "txt")
-metric string
Popularity DB metric (NACC, TOTCPU, NUSERS) (default "NACC")
-pbrdb string
Name of PBR db (see PhedexReplicaMonitoring project)
-phgroup string
Phedex group name (default "AnalysisOps")
-profile
profile code
-site string
CMS site name, use T1, T2, T3 to specify all Tier sites
-tier string
Look-up specific data-tier
-trange string
Specify time interval in YYYYMMDD format, e.g 20150101-20150201 or use short notations 1d, 1m, 1y for one day, month, year, respectively (default "1d")
-verbose int
Verbose level, support 0,1,2
Examples
In all examples below we use T2_XX_Abc as a site name.
# list site statistics for last month
sitestat -site T2_XX_Abc -trange 1m
# list site statistics for specific time range
sitestat -site T2_XX_Abc -trange 20150201-20150205
# list site statistics for last 3 months
sitestat -site T2_XX_Abc -trange 3m
# list site statistics for last month and only count AOD data-tier
sitestat -site T2_XX_Abc -trange 1m -tier AOD
# list site statistics for last month with breakdown for all data-tiers
sitestat -site T2_XX_Abc -trange 1m -breakdown tier
# list site statistics for last month with breakdown for all datasets
sitestat -site T2_XX_Abc -trange 1m -breakdown dataset
# list site statistics for last month with breakdown for all data-tiers and look for NUSERS metric
sitestat -site T2_XX_Abc -trange 1m -metric NUSERS -breakdown tier
# by default sitestat relies on PhEDEx data-service to collect
# dataset information on site, but we may use DBS instead
sitestat -site T2_XX_Abc -trange 1m -dbsinfo
# return information in json data format
sitestat -site T2_XX_Abc -trange 1m -format json
Tools
The tools directory contains useful scripts to use PhedexReplicaMonitoring which allows to obtained weighted datasets size on sites from PhEDEx DB by running pbr script from PhedexReplicaMonitoring repository.
- pbr_avg.sh script can be used to submit Spark job to calculate average size of datasets
- pbr_db.py script can be used to convert HDFS output from pbr_avg.sh and convert it into SQLiteDB. The later can be used by sitestat tool
- plot.R an R script to produce size vs bins (#accesses) plot.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent