CppConsoleDesktop

C++ Console Desktop Sample for ONNX Runtime

This sample demonstrates how to use ONNX Runtime in a C++ desktop application, focusing on:

1. Execution Provider (EP) discovery and configuration
2. Model compilation for optimized inference
3. Command-line options for flexible usage

Command-line Usage

CppConsoleDesktop.exe [options]
Options:
  --ep_policy <policy>           (Required*) Set execution provider selection policy (NPU, CPU, GPU, DEFAULT)
  --ep_name <name>               (Required*) Explicit execution provider name (mutually exclusive with --ep_policy)
  --compile                      Compile the model
  --download                     Download required packages
  --model <path>                 Path to input ONNX model (default: SqueezeNet.onnx in executable directory)
  --compiled_output <path>       Path for compiled output model (default: auto-generated with device info)
  --image_path <path>            Path to the input image (default: sample kitten image)

Exactly one of --ep_policy or --ep_name must be specified.
--use_model_catalog and --model are mutually exclusive.

Key Features

1. Execution Provider Configuration

The sample demonstrates how to discover available execution providers and configure them:

#include <win_onnxruntime_cxx_api.h>

  // Accumulate devices by ep_name
  // Passing all devices for a given EP in a single call allows the execution provider
  // to select the best configuration or combination of devices, rather than being limited
  // to a single device. This enables optimal use of available hardware if supported by the EP.
  std::unordered_map<std::string, std::vector<Ort::ConstEpDevice>> ep_device_map;
  for (const auto& device : ep_devices)
  {
      ep_device_map[device.EpName()].push_back(device);
  }
  std::wcout << L"Execution Provider Information:\n";
  std::wcout << L"-------------------------------------------------------------\n";
  std::wcout << std::left << std::setw(32) << L"Provider" << std::setw(16) << L"Vendor" << std::setw(12)
              << L"Device Type" << std::endl;
  std::wcout << L"-------------------------------------------------------------\n";
  for (const auto& [ep_name, devices] : ep_device_map)
  {
      std::cout << std::left << std::setw(32) << ep_name;
      for (const auto& device : devices)
      {
          std::cout << std::setw(16) << device.EpVendor() << std::setw(12) << ToString(device.Device().Type())
                      << std::endl;
      }
  }

// Configure and add each EP with appropriate options
for (const auto& [ep_name, devices] : ep_device_map)
{
    Ort::KeyValuePairs ep_options;
    if (ep_name == "VitisAIExecutionProvider")
    {
        // Append the Vitis AI EP
        session_options.AppendExecutionProvider_V2(env, devices, ep_options);
        std::cout << "Successfully added " << ep_name << " EP" << std::endl;
    }
    // ... other providers
}

2. Model Compilation

The sample shows how to compile an ONNX model for optimized execution. Compiled model filenames are automatically generated with device-specific identifiers to prevent collisions:

• Policy mode: SqueezeNet_ctx_PREFER_GPU.onnx
• Explicit EP: SqueezeNet_ctx_DML_GPU.onnx
• With perf mode: SqueezeNet_ctx_PREFER_NPU_MaxPerformance.onnx

Use --compiled_output to override with a custom path.

Before reusing an existing compiled model, the sample reads its execution-provider compatibility metadata and validates it against the currently selected devices. Devices are checked in single-EP groups. Explicit --ep_name and --device_type choices validate only that selected group. Policy mode follows ONNX Runtime's policy ordering and validates the preferred device and CPU fallback devices selected by the policy. Every selected group with compatibility metadata must report optimal compatibility, and the preferred group must contain matching metadata. Missing metadata is tolerated only for fallback groups that do not compile the model.

Only EP_SUPPORTED_OPTIMAL models are reused. Missing metadata, EP_NOT_APPLICABLE, EP_SUPPORTED_PREFER_RECOMPILATION, and EP_UNSUPPORTED are treated as non-optimal. With --compile, the replacement is compiled to a temporary file with EP context embedded, then atomically moved into place. This prevents the final model from referencing a temp-named sidecar. A failed compilation falls back to the original ONNX model instead of loading the stale compiled model. Without --compile, a non-optimal compiled model also falls back to the original model.

#include <win_onnxruntime_cxx_api.h>

// The helper compiles to a temporary path with EP context embedded, then uses
// MoveFileExW to atomically replace compiledModelPath after compilation
// succeeds.
OrtStatus* status = ModelManager::CompileModel(
    ortApi,
    env,
    sessionOptions,
    modelPath,
    compiledModelPath);

3. Execution Provider Selection Policy

The sample demonstrates how to set an EP selection policy to prefer specific hardware:

#include <win_onnxruntime_cxx_api.h>

// Prefer NPU if available
Ort::SessionOptions sessionOptions;
sessionOptions.SetEpSelectionPolicy(OrtExecutionProviderDevicePolicy_PREFER_NPU);