Articles EN

Running SCPI instrument tests in CI without hardware

How to test end-of-line test station code in pytest and CI against simulated SCPI instruments: a supply in current limit, a dead multimeter, a wrong instrument.

  • scpi
  • pyvisa
  • pytest
  • test-automation
  • ci

Test station software has an awkward property: it can only be tested where the instruments are. The power supply and the multimeter are on the production line or on a colleague’s bench, and they are busy. So the code that decides whether a board passes or fails is usually tested by running it once on a good board and calling it done.

That leaves the interesting cases untested: the rail that sits in current limit, the multimeter that stops answering in the middle of a shift, the instrument that was swapped for another model. This article shows how to test exactly those cases with pytest, on every commit, without a single instrument.

The station

A typical end-of-line test for a board with a 12 V and a 3.3 V rail: a two-channel power supply feeds the board, a relay multiplexer routes one rail at a time to a multimeter, and the code compares each reading with its limits. The station code talks to the instruments through PyVISA:

# station.py
"""End-of-line test for a board with a 12 V and a 3.3 V rail."""

RAILS = {101: ("12V", 12.0), 102: ("3V3", 3.3)}   # mux channel: (name, nominal)
TOLERANCE = 0.05                                   # +/- 5 %


class StationError(Exception):
    pass


def check_errors(inst) -> None:
    """Read the error queue empty; any entry means a command went wrong."""
    errors = []
    while not (reply := inst.query("SYST:ERR?")).startswith(("+0", "0")):
        errors.append(reply)
    if errors:
        raise StationError(f"instrument reported {errors}")


def check_identity(inst, model: str) -> None:
    idn = inst.query("*IDN?")
    if model not in idn:
        raise StationError(f"expected a {model}, found {idn!r}")


def power_up(psu) -> None:
    psu.query("SOUR1:VOLT 12;CURR 1;:SOUR2:VOLT 3.3;CURR 1;:OUTP1 ON;:OUTP2 ON;*OPC?")
    check_errors(psu)


def measure_rail(dmm, mux, channel: int) -> float:
    # *OPC? waits until the relay has switched: the DMM is another connection,
    # so a plain write could still be in flight when it measures.
    mux.query(f"ROUT:OPEN:ALL;:ROUT:CLOS (@{channel});*OPC?")
    return float(dmm.query("MEAS:VOLT:DC?"))


def run(psu, dmm, mux) -> dict[str, str]:
    check_identity(dmm, "DMM")
    power_up(psu)
    result = {}
    for channel, (name, nominal) in RAILS.items():
        volts = measure_rail(dmm, mux, channel)
        ok = abs(volts - nominal) <= nominal * TOLERANCE
        result[name] = f"{'PASS' if ok else 'FAIL'} {volts:.3f} V"
    return result

Nothing special, and that is the point: this is the kind of code that runs on test stations everywhere, and most of it has only ever met a good board.

Simulated instruments that behave

scpi-sim is an open-source Python package I wrote for this. It runs simulated SCPI instruments (power supply, multimeter, relay multiplexer, oscilloscope) as real network servers on a raw socket, and on request over VXI-11 and HiSLIP. PyVISA, C#, LabVIEW or nc talk to them like to real LAN instruments. Profiles make them answer like fifteen common models from Rohde & Schwarz, Keysight, Keithley, Siglent, Rigol and Tektronix.

What matters for testing is that they behave, not just answer:

  • the supply goes from constant voltage into current limit when its load draws more than the limit allows, and its output voltage drops;
  • instruments on a bench are wired together, so the multimeter measures what the supply actually delivers into the load;
  • they are strict where real instruments are strict: a malformed command ends up in the error queue, not silently accepted;
  • faults can be injected at runtime: no answer, a late answer, a dropped connection, a wrong identity, drifting readings.

The scpi-sim browser demo: a simulated four-channel power supply, a DAQ and an oscilloscope next to a SCPI console

You can try it in your browser without installing anything.

The tests

Installing scpi-sim registers a pytest plugin. scpi_bench_factory starts a bench from a dict or a YAML file, on free ports, for each test, and stops it afterwards. The bench below is the station above: two rails as resistive loads, wired to the supply and through the multiplexer to the multimeter.

# test_station.py
import pytest
import pyvisa

import station

BENCH = {
    "instruments": {
        "psu": {"type": "psu", "port": 0, "channels": 2},
        "dmm": {"type": "dmm", "port": 0},
        "mux": {"type": "mux", "port": 0, "channels": 16},
    },
    "loads": {
        "rail_12v": {"type": "resistor", "ohms": 24},   # 0.5 A at 12 V
        "rail_3v3": {"type": "resistor", "ohms": 33},   # 0.1 A at 3.3 V
    },
    "wiring": [
        "psu.ch1 -> rail_12v", "psu.ch2 -> rail_3v3",
        "mux.101 -> rail_12v", "mux.102 -> rail_3v3",
        "mux.common -> dmm.input",
    ],
}


@pytest.fixture
def bench(scpi_bench_factory):
    return scpi_bench_factory(BENCH)


@pytest.fixture
def instruments(bench):
    rm = pyvisa.ResourceManager("@py")
    insts = [rm.open_resource(bench.resource(name), read_termination="\n",
                              write_termination="\n", timeout=500)
             for name in ("psu", "dmm", "mux")]
    yield insts
    for inst in insts:
        inst.close()
    rm.close()


def test_good_board_passes(instruments):
    result = station.run(*instruments)
    assert result["12V"].startswith("PASS")
    assert result["3V3"].startswith("PASS")


def test_rail_in_current_limit_fails(instruments):
    psu, dmm, mux = instruments
    station.power_up(psu)
    psu.query("SOUR2:CURR 0.05;*OPC?")      # the 3.3 V rail needs 0.1 A
    assert psu.query("OUTP2:MODE?") == "CC"
    assert station.measure_rail(dmm, mux, 102) < 3.3 * 0.95


def test_wrong_instrument_is_refused(bench, instruments):
    bench.dmm.inject("idn", value="ACME,PSU-9000,1234,1.0")
    with pytest.raises(station.StationError, match="expected a DMM"):
        station.run(*instruments)


def test_dead_multimeter_times_out(bench, instruments):
    psu, dmm, mux = instruments
    station.power_up(psu)
    bench.dmm.inject("no_response")
    with pytest.raises(pyvisa.errors.VisaIOError):
        station.measure_rail(dmm, mux, 101)


def test_typo_in_a_command_is_caught(instruments):
    psu = instruments[0]
    psu.write("SOUR1:VOLT 12;SOUR1:CURR 1")   # path rule: needs ":" before SOUR1
    with pytest.raises(station.StationError, match="-113"):
        station.check_errors(psu)
$ pytest -q
.....                                                        [100%]
5 passed in 0.66s

The good board reads {'12V': 'PASS 12.000 V', '3V3': 'PASS 3.300 V'}. The other four tests are the ones that matter.

Three things the tests catch

A rail in current limit. Lower the current limit below what the 3.3 V rail draws, and the supply switches to constant current (OUTP2:MODE? answers CC); the voltage on the rail drops, and the station must fail the board. With a static simulator that only echoes a configured value, this test would pass for the wrong reason.

A dead or wrong instrument. no_response makes the multimeter swallow its next answer, so the station must see a timeout, not hang and not record a stale value. idn makes the instrument claim to be something else, which is what happens when somebody swaps a unit on the line. Both take one line in the test and would otherwise only show up in production.

The SCPI path rule. This one surprises people. In a compound command, everything after a ; continues at the level of the previous command. So

SOUR1:VOLT 12;SOUR1:CURR 1

is read as SOUR1:VOLT 12 followed by SOUR1:SOUR1:CURR 1, which does not exist. A real instrument answers with -113,"Undefined header" in its error queue, keeps the voltage and ignores the current limit, and carries on. The correct form is SOUR1:VOLT 12;:SOUR1:CURR 1 (or SOUR1:VOLT 12;CURR 1). The simulator is just as strict, so check_errors after configuration finds the mistake before it meets the hardware:

>>> psu.write("SOUR1:VOLT 12;SOUR1:CURR 1")
>>> psu.query("SYST:ERR?")
'-113,"Undefined header"'
>>> psu.query("SYST:ERR?")
'0,"No error"'

Two habits from this example are worth keeping on any station, simulated or not: read the error queue after configuring an instrument, and finish commands with *OPC? before measuring on another connection. Each instrument is its own connection; without *OPC?, the multimeter can measure before the relay has switched.

In CI

The pytest plugin starts the instruments itself, so CI only needs the packages. PyVISA with the pure-Python backend (pyvisa-py) needs no NI-VISA installation:

# .github/workflows/test.yml
name: test
on: [push, pull_request]
permissions:
  contents: read
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install scpi-sim pyvisa pyvisa-py pytest
      - run: pytest -q

Station code written in C# or LabVIEW can use the Docker image instead, ghcr.io/steitec/scpi-sim, which runs a bench like the one above and serves raw sockets, VXI-11 and HiSLIP; faults are injected over its control port. The README has the details.

Limits

The instruments behave plausibly, not exactly like one particular unit: noise, settling times and the quirks of a given firmware are simplified, and the profiles follow the manuals rather than tests against every real model. VXI-11 and HiSLIP are tested with PyVISA-py, not yet with NI-VISA or Keysight VISA. None of this replaces the first run on the real station. What it replaces is the hope that the station code copes with a board in current limit, a dead multimeter or a typo in a command string. That you can now test on every commit.

← All articles