Today our active_range is always identical to percent_full. We cannot vary these independently. Client scenarios might be interested in something like 99% full, 10% active range. In order to support this, we would most likely want to create two test files, initialize each of them, but then only run the IO generator targeting one of them.

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Currently, there's no record of errors occurring in the process executed in the background (through "bg_exec" in the recipes). This can be a problem if you run one of the targeted tests designed to check that the performance of a workload is unchanged by background activity (like write_impact or smart_check). If the bg process silently errors, the fg process will have the same performance and the drive could incorrectly pass the test.

One solution is to capture the std err from the bg process (see line 830 of Util.pm). A better option is to fail out the whole recipe if the BG process errors.

A testing scenario (where I've seen the bg process error) is running write_impact in demo mode on a slow thumb drive on win10 client. Something about this combination causes the bg process to target a drive letter that no longer exists.

Currently we default to SQLIO2, but there is more active development around DiskSpd.

Parsing and running are different enough activities that the modules should probably be split. There are a lot of examples already (LogmanParser/LogmanRunner, etc.)

Somehow we overlooked this when transitioning from SQLIO.

Add the OS version to the data collection & the data parsing. Could use "wmic os get Caption."

I was trying to run the "recipes\SIT\SIT_SSD_Sanity.rcp" recipe on a 480GB SSD and got the error message in the title when it got to 4K seq 100% read QD 128.

Event Viewer has the following:
Faulting application name: DiskSpd.exe, version: 0.0.0.0, time stamp: 0x57449527
Faulting module name: ucrtbase.DLL, version: 10.0.10586.9, time stamp: 0x5642c48d
Exception code: 0xc0000409
Fault offset: 0x00000000000698fe
Faulting process id: 0xe30
Faulting application start time: 0x01d2bc18470f6f2b
Faulting application path: C:\storscore\bin\DiskSpd.exe
Faulting module path: C:\Windows\SYSTEM32\ucrtbase.DLL
Report Id: ed649a57-280c-11e7-80d1-e41d2dece230
Faulting package full name:
Faulting package-relative application ID:

I specified a target and recipe file, and got the following with ActiveState Perl:

Error "Can't locate object method "interactive" via package "CommandLine" at C:\Users\A
dministrator.MICRONSTORAGE\Desktop\StorScore-master\StorScore.cmd line 208."

If I add --interactive or --nointeractive, I get back unknown option text like the following:
StorScore.cmd --target=.\physicaldrive1 --recipe=recipes\turkey_test.rcp --interactive returns "Unknown option: interactive"

Create a new test kind "fg_exec." This would allow you to add any command to the recipe as a step. Similar to bg_exec, but later steps block on the completion of the fg_exec.

Sometimes StorScore fails with the following message:

Could not open file "\results<drives' results folder>\logman-raw-.csv' No such file or directory.

This happens after starting a test step (ie not a purge, precondition or initialization step)

It's reasonable to want to test a sector-sized access. DiskSpd supports this, but SQLIO2 and Precondition.exe do not, as of yet.

When we create the smart.txt file we could pass --identify to smartctl.exe which would collect the result of the SATA IDENTIFY_DEVICE command. This information could be useful later on.

It's possible to coerce StorAHCI into converting FILE_FLAG_WRITE_THROUGH to FUA. This is not default, but could be enabled through the registry:

 HKLM\System\CurrentControlSet\Services\storahci\Parameters\Device\EnableFuaSupport

Hello;

I am trying to demo storscore with a plugged in usb drive with a sata disk inside of it and with a sandisk usb flash drive since I can not afford to trash my laptop's filesystem.

So preconditions fails with the following error:

Cleaning disk...
Creating new filesystem...
Creating test file...
Syncing target volume...
SmartCtl missing or broken. SMART capture disabled.
Ipmiutil missing or broken. Power measurement disabled.
Access is denied.
Precondition returned non-zero errorlevel at C:/cygwin64/home/utz_j/GIT/StorScor
e\lib/PreconditionRunner.pm line 121.

I get this same problem in both cases, any suggestions? Your help would be appreciated.

Complete capture follows:

C:\cygwin64\home\utz_j\GIT\StorScore>StorScore.cmd --target .\PHYSICALDRIVE1 r
ecipes\flush_check.rcp
Targeting HDD: SanDisk SanDisk Ultra USB Device

Loaded recipes\corners.rcp (24 tests, 24 steps)

    Run time will be >= 1.02 days after target init (1 overwrite).

    Warning!
    Detected System Center Endpoint Protection (MsMpEng.exe)
    This can delay IOs and cause bogus latency results.
    You have the following options:
        - Run on a machine without SCEP
        - Disable SCEP real-time protection
        - Exclude testfile.dat from SCEP scan.

    Warning!
    This will destroy \\.\PHYSICALDRIVE1

Do you wish to continue? [Y/N]

Cleaning disk...
Creating new filesystem...
Creating test file...
Syncing target volume...
SmartCtl missing or broken. SMART capture disabled.
Ipmiutil missing or broken. Power measurement disabled.
Access is denied.
Precondition returned non-zero errorlevel at C:/cygwin64/home/utz_j/GIT/StorScor
e\lib/PreconditionRunner.pm line 121.

C:\cygwin64\home\utz_j\GIT\StorScore>

A section in the instructions directs the user to install these, however, both links report "this download is no longer available":

The Visual Studio 2013 C++ runtime libraries for x86 & x64:
http://www.microsoft.com/en-us/download/details.aspx?id=40784

The Visual Studio 2015 C++ runtime libraries for x86 & x64:
https://www.microsoft.com/en-us/download/details.aspx?id=48145

Typical request is: what does the latency of small random reads look like when there is a simultaneous large block sequential write stream at low QD/throughput.

It's unclear how best to implement this. The load generators and preconditioner might need direct support. Another possibility is simply to arrange for everyone's CreateFile call to pass the most permissive dwShareMode, thus making it possible for StorScore to spawn multiple copies of the IO generator simultaneously targeting the same file.

Today the scores are purely based on performance. We could include WAF/DWPD or other endurance metrics in the computation, or we could create separate performance and endurance scores.

Somehow I have to run a command like

storscore.cmd -recipe=recipes\max_power.rcp –raw

but there is no such a recipe in recipes in this tool.

Any idea where I can find it, please kindly advice.

StorScore seems to have an assumption that the drive(s) under test are using a small sector size.

As an example, storscore tries to execute this command, and it fails to start threads due to error 87 (invalid parameters)
DiskSpd.exe -w100 -si -b1K -t4 -o64 -a0,2,4,6,8,10,12,14,16,18,20,22,1,3,5,7,9,11,13,15,17,19,21,23 -L -h -d300 -Z20M,"C:\StorScore\entropy\0_pct_comp.bin" D:\testfile.dat

When I investigated the scenario, the logical sector size was 4K, but Diskspeed was using a stripe size of 1K, which would not have worked. Changing the command line to use a block size and stripe size of 4K permitted the test to run.

Can anything be done to validate the parameters against the test devices logical and physical characteristics to prevent logs reporting failures?

I've been running the 4k_RR StorScore recipe on various drives and I have came across this issue:

For most drives that I have tested the test description reads "4K_Random_Reads-step-4" however for a few other drives the test description reads as "4K_Random_Reads-step-2". I selected the 4K_RR recipe each time. Is there a reason for this inconsistency?

I've been running the StorScore 4K Read and Write recipes on various drives and comparing the results to those generated by FIO. I have noticed that while the 4K Write IOPS results produced by StorScore and FIO tend to be similar to each other and both are similar to the published performance data I have for the drives I am benchmarking the 4K Read IOPS results for StorScore tend to be significantly lower than those produced by FIO or found in published data for the drives I am using. Is there any obvious reason for this discrepancy? The 4K Read recipe itself does not seem to show anything that explain the significant difference in results that I am seeing.

Today the parser is not very modular. In particular, you have to hack the source to change the scoring policy code. We could create a system akin to the DeviceDB, perhaps in a "score_policy" subdirectory, where we would ship BingPolicy.pm and AzurePolicy.pm by default. Users could create their own policies (conforming to the standard interface) by adding them to this directory.

Add another optional property to the test definition that lets the user specify additional arguments for the IO generator per-test. For example, the following test would run a 4k random read test with IOs aligned to 8k boundaries:

test(
description => "4k Random Reads",
write_percentage => 0,
access_pattern => 'random',
block_size => '4K',
queue_depth => 1,
warmup_time => 60,
run_time => 3600,
io_gen_args => '-r8k ',
);

Currently, the only way to loop through multiple values for a generator's parameter is to build a script that calls storscore with different values for --io_generator_args. This makes it difficult to name each test and manage their results as a set.

This is the so-called open-loop versus closed-loop issue.

Real world applications don't typically control for constant queue depth. Rather, they have some typical (average) throughput, which means that any "hiccups" in storage latency cause many IOs to "back up" and spike QD.

It we tested this way, it would be more realistic. The high-percentile latency outlier metrics (things like 5-nines latency) might be more representative of what a real app could expect.

First we'd have to run some kind of "probe" to determine the peak IOPS of the device, and then divide that range into discrete tests (something like 10%, 20%, etc.). The IO generator and preconditioner would need rate-limiting support as an alternative to the "-o" queue depth argument.

I've been working with some different partition sizes, and noticed the "Partition Size" and "User Capacity" columns are always equal when I target a physical drive number. This is because wmic data is collected once before all the steps run, and StorScore doesn't track the volume until the partition is created (by default on each step).

We could automatically detect errors during the test by checking the Windows Event Viewer. It's possible, for example, that some IOs will fail, retry, and eventually succeed. It would be nice to make this obvious.

This will require some thought, but I want to get the conversation started early in my thinking. I would like StorScore to support multiple targets. This is important for evaluating hardware for the datacenter, where multiple users share the same drive. It's also timely with all the recent activity on streaming and open channel drives.

For example, I should be able to verify that a streaming drive with one sequentially-written file mapped to each stream will get a very low WAF. Whereas this same workload will appear essentially random to a standard block mode drive, yielding a higher WAF.

Here are a few areas that will need attention:

• preconditioner -- the preconditioner should run for each target, and exit only when all targets have reached steady state. Preconditioner will need to take a "done" signal.
• recipe -- it will need to allow a different IO pattern for each target, and a way to signal which tests are run sequentially and which are run simultaneously.
• test setup -- We can start with having the test operator set up the targets manually, and pass the file names. Later improvements can automate this.
• parsing -- the raw data could have another column to let us view the stream's performance individually. I will need to think about how/if the scoring will change.

In precondition.cpp:
The following line numbers and variable names need to change from int to int64_t:
Line 39: outstandingIOs
Line 419: postedIOs
Line 420: completedIOs
Line 428: TOTAL_BLOCKS

Essentially turn initialize() into SNIA's workload-independent preconditioning. This means we could do something like to test "128K sequential reads @qd1 after 4K random writes":

purge();
initialize( block_size = '4K', access_pattern = 'random' ); # Always 100% write, high queue depth
test(
write_percentage = 0,
access_pattern = 'sequential',
block_size = '128K',
queue_depth = 1,
warmup_time = 60,
run_time = 3600,
purge = 0,
initialize => 0,
);

Right now it's very likely that we become CPU-limited in some cases and do not attain the requested queue depth. The solution is to multi-thread the preconditioner. The main hurdle will be to design a thread-safe version of steady_state_detector.h.

This is more of a request for information than an issue. I was wondering if someone could breakdown how StorScore preps and preconditions a drive before a test is run. Just looking at the output I can see that the drive is purged, initialized, and then preconditions. What do each of these steps consist of?

Thank you

Hi,

how can I specify output dir path for my results from a workload?

Many variations of:
"Experimental keys on scalar is now forbidden at C:\StorScore\parse_results.cmd line 1385."

Hi~

Please ask if you can give me code "max_power.rcp", because i can't search for it on website. Thanks.

BR

The parse_results.cmd is too large, and is not well-factored like StorScore.cmd. We should modularize it for better maintainability, but also to facilitate a 2-phase approach where we parse and score separately. It would be nice to parse results only once, and merely run the scoring steps multiple times when comparing different sets of devices.

Hey Mark -- I have a question about the code in lib\Recipe.pm. I'd like to add a warning based on properties of a test. I believe "warn_illegal_args" is the right place to do that. It should be called when StorScore processes the recipe ("Phase 1" in the BUILD sub), but it isn't. I've traced it down to the anonymous function on line 375. The foreach loop runs for each kind of step, but a print statement inside the anonymous function doesn't run. Can you help me understand why? I suspect the anonymous function needs to be called, but it's not clear to me where that should happen, or why it's happening for phase 2 but not for phase 1.

For example, this code:

    # Install our handler for each kind of step 
    print("warn? $recipe_warnings\n");
    foreach my $kind ( @step_kinds )
    {
        my $sym = "${package}::$kind";
        print("in foreach loop $kind\n");
        *$sym = sub
        {
            print("here\n");
            return $self->handle_step(
                $callback,
                $recipe_warnings,
                $kind,
                @_
            )
        };
    }
    print("done\n");

prints the following:
warn? 1
in foreach loop test
in foreach loop purge
in foreach loop initialize
in foreach loop precondition
in foreach loop bg_exec
in foreach loop fg_exec
in foreach loop bg_killall
in foreach loop idle
done

Hi,
Initially I started by running the 4k_RR.rcp on my SSD and I got the results in the excel sheet with all the details but later when I ran Storscore command with 4K_RW.rcp or any other recipes, then only the first sheet is filled with the data. Final and score details looks empty even after many runs.

Commands used to run:
StorScore.cmd -recipe=recipes\4k_RW.rcp --target=f:
parse_results.cmd results* results.xlsx

Can anyone guide me/tell me why I am not getting the full result sheet with all the data even after parsing?

Hi everyone,
In a recipe, how can I increase the number of queues to the device.
I see Queue Depth can be changed, is there a way to create multiple queues(threads) and have all of them pump reads/writes to the Device.
Thanks
Sowmya

I would like to define some command line options from within a recipe. For example:

$cmd_line->keep_logman_raw = 1;

test(
description => "4k Random Reads",
write_percentage => 0,
access_pattern => 'random',
block_size => '4K',
queue_depth => 1,
warmup_time => 60,
run_time => 3600,
);

Currently, the first line fails.

StorScore has the ability to test any volume, which can be handy for testing many drives in a system as a striped volume. Unfortunately, when you target a volume, StoreScore is not able to purge and wipe the history between tests.

One possible solution is to pass several drives to StorScore (eg "target=1,3,2,5"). Just as StorScore creates a volume after purges for a single drive, it could create a striped volume will the whole set of targets before each test.

It could also have a flag to let the test operator choose between these two options:

1. Stripe all the targets
2. Pass the list of targets to diskspd (so they're tested individually & simultaneously)

Note: If the operator uses the --raw flag, StorScore will need to use options 2.

If the purpose of the initialize step is to write data of full capacity sequentially, does it make sure that data is written sequentially when MDTS is smaller than 1M, the block size of initialize step? As I know, if MDTS is smaller than 1M, the device driver, stornvme.sys, divides it to several commands . In this case, does the device driver guarantee the sequential data written? If not, I suggest smaller "DEFAULT_IO_SIZE" such as 128K in precondition.h or add block size parameter in write_num_pass.pm line 78 as "$cmd .= "-b128k " to use smaller block size.

With the latest June StorScore version, I am seeing an issue on Windows 2016 server edition where the storscore Corners.rcp file is causing drive corruption messages to pop up in the eventvwr. I am using a drive that supports Secure Erase. I see this issue if the storscore partition is already created and when it runs the "secure erase/ diskpart clean" during the purge phase. wanted to check if its a known issue.

Thanks
Deepak

Right now we only "diskpart clean", which amounts to a TRIM of the whole target.

Today we get system power only via ipmitool. ACPI power meter object works on some systems to expose the same information via logman/perfmon counters.

The object is called "\Power Meter(_Total)\Power"