onnx-mlir

Logo

Representation and Reference Lowering of ONNX Models in MLIR Compiler Infrastructure

View the Project on GitHub onnx/onnx-mlir

How-Tos

Inference Using Python
Inference Using C/C++
Inference Using Java

References

ONNX Dialect
OMTensor C99 Runtime API
OMTensorList C99 Runtime API
OMTensor Java Runtime API
OMTensorList Java Runtime API
Generate ONNX Dialect
About Documentation

Development

Add an Operation
Testing Guidelines
Error Handling
Command-line Options
Instrumentation
Constant Propagation
Add an Accelerator

Tools

Tools

RunONNXModel.py
DocCheck

This project is maintained by onnx

Hosted on GitHub Pages — Theme by orderedlist

Instrumentation

Instrumentation is prototyped in onnx-mlir and can be used to debug runtime issue.

Compile for instrumentation

By default, instrumentation is turned off. You need to use following command line options to turn it on. The pass for instrumentation will be inserted in some stages by using --instrument-stage option. For example, when you specify Onnx, the instrumentation will be inserted after onnx-to-onnx conversion to get onnx-level profiling. The --instrument-ops option is an option to specify operations to be instrumented. You can use onnx.Conv for onnx Conv operations for example. Also, you can use asterisk such as onnx.* for all onnx operations, and specify two expressions with , such as onnx.Conv,onnx.Add for both Conv and Add operations. The --InstrumentBeforeOp and --InstrumentAfterOp are options to insert instrumentation before and/or after the specified operations. When you use --instrument-ops=onnx.* --InstrumentBeforeOp --InstrumentAfterOp, the instrumantation will be inserted before and after all onnx operations. For NNPA, additional stages for ZHigh and ZLow are provided. You can get profile for onnx and zhigh ops using --instrument-stage=ZHigh and --instrument-ops=onnx.*,zhigh.*, and for zlow ops using --instrument-stage=ZLow and --instrument-ops=zlow.*.

  --instrument-stage=<value>                        - Specify stage to be instrumented:
    =Onnx                                             -   Profile for onnx ops. For NNPA, profile onnx ops before lowering to zhigh.
    =ZHigh                                            -   NNPA profiling for onnx and zhigh ops.
    =ZLow                                             -   NNPA profiling for zlow ops.

  --instrument-ops=<string>                         - Specify operations operations to be instrumented:
                                                      "NONE" or "" for no instrument,
                                                      "ops1,ops2, ..." for the multiple ops.
                                                      e.g. "onnx.Conv,onnx.Add" for Conv and Add ops.
                                                      Asterisk is also available.
                                                      e.g. "onnx.*" for all onnx operations.

  Specify what instrumentation actions at runtime:
      --InstrumentBeforeOp                          - insert instrument before op,
      --InstrumentAfterOp                           - insert instrument after op,
      --InstrumentReportTime                        - instrument runtime reports time usage,
      --InstrumentReportMemory                      - instrument runtime reports memory usage.

Currently, the call of initialization, OMInstrumentInit, need to be added before you load the dynamic library. It is being considered to add it to the beginning of main_graph by compiler.

Run with instrumentation

Run the model in the same way as usual. The instrumentation library will print out the time and memory usage along at each instrumentation point. For example, a model, mymodel.onnx, is compiled with onnx-mlir --instrument-stage=Onnx --instrument-ops=onnx.* --InstrumentAfterOp --InstrumentReportMemory --InstrumentReportTime mymodel.onnx. Its runtime output is listed below:

==PERF-REPORT==, onnx.Cast, bert/encoder/Reshape__27, before, 0.000001, 1692654182.738546
==PERF-REPORT==, onnx.Cast, bert/encoder/Reshape__27, after, 0.000001, 1692654182.738547
==PERF-REPORT==, onnx.Concat, bert/encoder/Reshape__27, before, 0.000000, 1692654182.738547
==PERF-REPORT==, onnx.Concat, bert/encoder/Reshape__27, after, 0.000001, 1692654182.738548
==PERF-REPORT==, onnx.Reshape, bert/encoder/Reshape, before, 0.000001, 1692654182.738549
==PERF-REPORT==, onnx.Reshape, bert/encoder/Reshape, after, 0.000001, 1692654182.738550

The output for the time measurement is explained here.

The output for the memory measurement is explained here.

Other example for NNPA

Control instrument at runtime

By providing certain env variable at runtime, you can disable reports from instrument library.

Please note that the only way to enable instrumentation is to request it at compile time. If none of the detailed report (such as time and memory so far) is turned on at runtime, progress of instrument point will still be print out. This feature is thought to be useful as progress indicator. To fully disable any outputs requested at compile time, you must set ONNX_MLIR_NO_INSTRUMENT.

Used in gdb

The function for instrument point is called OMInstrumentPoint. Breakpoint can be set inside this function to kind of step through onnx ops.