WINIC is a platform-independent automated microbenchmarking tool. It currently supports x86, ARM and RISC-V on Linux. WINIC can automatically determine latency and throughput values for most instructions the given CPU supports.
WINIC currently cannot measure:
- latencies on parts of memory operands e.g. base and index registers
- gather and scatter instructions
- branches, returns, system calls
- privileged instructions
For a given instruction form, WINIC either generates a kernel that allows the CPU to parallelize execution for measuring throughput, or one that forces sequential execution for measuring latency. The kernel generated is executed in a loop and the total runtime is measured. The cycles needed per executed instruction is calculated using only the clock frequency and the number of times the instruction was executed. Therefore WINIC does not need access to hardware performance counters.
WINIC uses LLVM and clang to generate and assemble benchmarks. Use setup.sh after cloning this repository to automatically download and build LLVM and WINIC. To manage multiple builds e.g. for multiple platforms in an HPC context specify --dir <buildName> to build a version of LLVM into ./llvm-build-buildName and WINIC into ./build-buildName.
To calculate the elapsed clock cycles per instruction WINIC needs the clock-frequency to be fixed e.g. by using likwid-setFrequencies. Once the frequency is fixed use WINIC as follows:
./winic [options] MODE -f <frequency> [mode-specific-options]For more consistent results it is strongly recommended to pin the application to a single core e.g. with likwid-pin.
likwid-pin -c 2 ./winic [options] MODE -f <frequency> [mode-specific-options]| Option | Description | Default |
|---|---|---|
-d,--debug |
Enable debug output | false |
-c,--cpu |
CPU model, needed if LLVM cannot detect it, or to use the INFO mode on a non-native architecture | - |
-m,--march |
Architecture, needed if LLVM cannot detect it, or to use the INFO mode on a non-native architecture | - |
-v,--version |
Show version | - |
Measure latencies or inverse throughputs.
By default WINIC measures all available instructions and generates a .yaml file with the results. Additionally a report_{MODE}_{TIMESTAMP} is generated providing additional information about how the values were obtained and warnings about unusual results. The runtime of a full run strongly depends on the architecture, here some rough estimates:
| Architecture | Runtime TP | Runtime LAT |
|---|---|---|
| x86 | 1h - 2h | 3h - 9h |
| AArch64 | 5 min | 30 min |
| RISCV | 30 min | 1h |
| Option | Description | Default |
|---|---|---|
-f,--frequency |
Frequency in GHz (required) | - |
-i,--instruction |
Measure specific instructions by LLVM instruction name. This option can take multiple names or regular expressions. For example, to measure all variants of SSE/AVX fused-multiply-add instructions use -i VFMADD.* |
- |
-o,--output |
Path to the .yaml file to save the results to. If the file already exists it will be updated (see updating a database) . If no file is specified, a timestamped one will be generated. If set to /dev/null no file will be generated. | - |
--register-init-value V |
Load V into all registers used before every benchmark run. Accepts decimal, octal with prefix 0, hexadecimal with prefix 0x, or floating point e.g. 7.0 (positive only) |
7.0 |
--immediate-value V |
Set all immediates to V. Accepts decimal, octal with prefix 0 or hexadecimal with prefix 0x |
7 |
--runs N |
Repeat each measurement N times and take the minimum runtime |
4 |
--loop-iterations N |
Number of loop iterations of the kernel. Higher values increase precision but also increase runtime. | 1000000 |
--loop-size N |
Target Number of instructions in the Loop body. Shorter bodies might be generated if there are not enough registers available | 12 |
--no-report |
Don't generate report file | false |
--output-asm |
Write the generated asm files to asm/ (clears the directory at the start of the run) |
false |
--memory |
Enable (all), disable (none) or filter (only) for instructions accessing memory |
all |
--x87-fp |
Enable, disable or filter for x87 floating point instructions. Default is none, as they are deprecated and consume a lot of time on architectures that emulate them. |
none |
--runtime-limit |
Set a limit for the time spent on an instruction in cycles. If a run takes longer than the limit the instruction is skipped | 300 |
--run-in-subprocess |
Run every benchmark in a subprocess. Always leave this enabled in production runs | true |
--keep-empty-entries |
Include instructions in the output even if they do not have any values | false |
--min-opcode |
Minimum LLVM opcode number to measure (this is mostly useful for development) | 0 |
--max-opcode |
Maximum LLVM opcode number to measure (this is mostly useful for development) | max opcode |
In manual mode, WINIC can execute arbitrary benchmark functions.
To run a function called tp from file.s and calculate the cycles per instruction assuming the loop has 12 instructions do
winic MAN -f <frequency> --path file.s --func-name tp --num-instructions 12| Option | Description | Default |
|---|---|---|
-f,--frequency |
Frequency in GHz (required) | - |
-p,--path |
Assembly file path (required) | - |
--func-name |
Function to benchmark (required) | - |
-n,--num-instructions |
Number of instructions in loop (required) | - |
--init-name |
Initialization function | - |
--runs N |
Repeat each measurement N times and take the minimum runtime |
4 |
There are always cases where WINIC doesn't produce correct data. To do a custom benchmark for an instruction, first run WINIC in TP or LAT mode with -i <LLVM_INSTRUCTION_NAME> --output-asm. This will output all .s files generated for the benchmark to asm/ and generate an assembler_out.log. The .s files can then be modified and executed using the MAN-mode.
In info mode, WINIC can provide metadata to any instruction it supports. This is useful if one wants to find the LLVM name for a specific instruction variant (find all "add" variants using -i ADD.*) and for development. The INFO mode can be used no matter the architecture WINIC is running on. To get metadata for another architecture, use the -c and -m options.
| Option | Description | Default |
|---|---|---|
-i,--instruction |
Specify the instruction name. This option can take multiple names or regular expressions. | - |
--internal |
Display the native LLVM operand types instead of WINICs abstraction | - |
By default TP and LAT mode generate a db_{TIMESTAMP}.yaml file with the results. Use -o/--output <file.yaml> to specify a custom path instead. If the file already exists the values obtained during the run will overwrite the existing ones according to the following rules:
- all new non-null values will overwrite existing values
- new null values will not overwrite existing values
- all existing values will be left unchanged if no new value was obtained
Updating the database works with single instructions as well as full TP/LAT runs. A standard workflow therefore would be to do a TP run generating a database and then a LAT run updating it.
WINIC automatically uses helper instructions to:
- break dependencies between instructions to measure throughput
- introduce dependencies between instructions to measure latency
All uses of helper instructions are logged in report_{MODE}_{TIMESTAMP}.
If an instruction would need a helper but none can be found, WINIC will fail and report "ERROR_NO_HELPER".
WINIC can only use instructions as helper if they were measured in the current run which is a problem when trying to measure single instructions.
The solution is to first do a full run and look up the dependencies of the instruction in the report, then the measurement can be reproduced by supplying all dependencies alongside the instruction using the -i <LLVM_INSTRUCTION_NAME> option.
Note that currently --output does not load the values into the internal working databases so the information read from there can not be used as helpers.
The analysis/ python module can be used to compare the results with various other sources, or to generate useful reference files which contain comprehensive information about instructions, operands, registers etc. from LLVM. For more details refer to analysis/README.md.