Skip to content

Re-measuring the 2026-10 perf optimizations on the target

The perf-opt branch (commits 13eb0832..071e543a) attacks the hot spots found in the ARM32 profile at /home/stano/projekty/falcon/bugs/20261006/prof/perf-report.txt (33K cpu-clock samples of the opendaf nucleus, ~15 modbus units polled at 4 Hz plus panter modulators). This guide says how to confirm the diagnosis on the OLD binaries, what to measure, and what the NEW binaries should show.

What was changed, and which profile line each change targets

Change Commit Profile cost it targets
Transaction kind tag replaces dynamic_cast on the commit path perf(core): transaction kind tag dynamic_cast + type_info::operator== + strcmp ≈ 6.5% (ARM EABI compares type names by string; the C→* follower casts failed on every measurement commit)
Followers subscribe to their leader object, not the global commit signal perf(core): followers subscribe per leader object unresolved libopendaf-core cluster 0x4e494/0x4e4ac/0x4e4dc ≈ 14.5% (boost::signals2 emission loop) + pthread mutex ≈ 5.7% + command-csv-loader self 5.2% (its hidden template copies of the slot code)
Edge-triggered stream-head notify (TX_COMPLETE/LOCK forwarded again since review round 13: the edge already coalesces their notifies) perf(stack): edge-triggered stream-head notify unix_stream_sendmsg 3.3% kernel (one notify-pipe byte per enqueue, ~160/s; Dev_Poll never coalesces)
Cached scan request blocks, --locking bypass, exact-length RX copy, blocade deadline timer perf(modbus-master): cached scan requests, ... per-cycle allocations/copies on the ~60 cycles/s serial path
Panter transaction coalescing, dead METER_READOUT query, subscriber-aware scan + bug fixes fix/perf(panter): ... 5-6 transactions+notifies per internal query → 1; one wasted serial round-trip per scan period
REPORT_CHANGES_ONLY=1, PUBLISH_INTERNAL_FSM_STATES=0 (new defaults) feat(panter): ... 1 Hz unchanged-counter pushes, FSM 20→30→20 flip per query

1. Before upgrading: confirm the diagnosis on the old build

The 14.5% cluster is attributed to boost::signals2 emission hidden under -fvisibility=hidden. Confirm with the unstripped old library (take it from the matching -dbg package or the build tree):

addr2line -f -C -e libopendaf-core.so.2.13.3 0x4e494 0x4e4ac 0x4e4dc

Expected: frames inside boost::signals2::detail::signal_impl::operator() / slot iteration instantiated for core::Event<void(core::Transaction*)>. If these resolve somewhere else entirely, stop and re-profile before trusting the expectations below.

Baseline the notify-pipe traffic (writes of 1 byte to the reactor notify fd) and overall CPU on the running old daemon:

pidof dafmand                       # -> PID
strace -f -c -e trace=write,sendmsg -p $PID & sleep 60; kill %1
pidstat -p $PID 10 6                # %CPU baseline

Record: writes/s on the notify fd (expect in the order of ~160/s with the falcon config), total %CPU, and keep the perf invocation used for the original report so the after-run is comparable:

perf record -F 997 -a -g -- sleep 60
perf report --sort dso,symbol > perf-report-before.txt

2. Upgrade

Build perf-opt as RelWithDebInfo (or keep the unstripped libs around — otherwise the after-profile is as unreadable as the before one). No configuration changes are required: the new panter parameters default to the optimized behavior, LOCKING defaults to 0 which now also short-circuits the in-stack lock messages.

To restore any legacy stream while validating:

  • REPORT_CHANGES_ONLY=0 — counters and permanent-error republished every tick as before, 1 Hz execution-queue timer back.
  • PUBLISH_INTERNAL_FSM_STATES=1 — FSM state published on every transition including the internal 20→30→20 flips.
  • LOCKING=1 — full lock message flow (only meaningful with a locking serial-datalink setup).

Known behaviour, panter: each request/response cycle now reaches the repository as ONE transaction, and a transaction commits all or nothing. A single misconfigured measurement (datatype not matching its value, e.g. a stackUmask that skips the scaler) therefore discards the whole cycle's updates - counters, FSM state, permanent error, ID data - not only its own. With REPORT_CHANGES_ONLY=1 those values come back only on their next change. The log shows both sides; grep for:

Measurement(<name>): Datatype <x>, can't update with <y>!
MeasurementUpdateTransaction rolled back: N of M updates invalid, all M discarded!

Fix the named measurement's datatype or stackUmask.

Behavioral notes for whoever validates SCADA-side: - Follower updates now always commit AFTER their leader's transaction (previously the order depended on subscription order). - AC_MV_ID_INFO arrives once per interrogated node (it was duplicated). - Panter stored-data values now pass raw→EU processing and deduplication (previously committed directly, bypassing the stack). - Delayed panter commands execute punctually instead of up to 1 s late. - The periodic METER_READOUT query (FC3 @ 0x65) is gone — its response was never parsed.

3. After: what the same measurements should show

Repeat the perf record, strace and pidstat runs with identical durations and load, then compare:

Metric Before Expected after
libopendaf-core unresolved cluster (0x4e494 area) ≈ 14.5% collapses to low single digits (emission loop now runs per-leader, not per-commit × per-follower)
dynamic_cast/type_info/strcmp triplet ≈ 6.5% essentially gone from the commit path
pthread_mutex_lock/unlock ≈ 5.7% roughly halved (far fewer signal slots per commit)
command-csv-loader self time ≈ 5.2% ≈ 0 (loaders now call core factories; no local template copies — verify with nm -D that the loader .so only imports createCommandFollower)
notify-fd writes/s ~160/s tens/s (edge-triggered; one wakeup per burst)
panter transactions per internal query 5–6 1
overall dafmand %CPU baseline down roughly a quarter to a third

If a line does not move as expected, diff the per-DSO view (perf diff perf-before.data perf-after.data) before looking for new causes — the usual culprit is comparing builds with different visibility/strip settings rather than a regression.