I'm building a video server and keep hitting a hard freeze I can't pin down. I'd appreciate ideas from anyone who knows Arc (DG2) on Linux, VT-d, or older Xeon platforms.
Hardware:
ASUS Z10PE-D8 WS, BIOS 4301 (latest)
2× Xeon E5-2696 v4 (Broadwell-EP, 22 cores each), microcode 0xb000040
2× Sparkle Arc A310 4 GB (4×HDMI "OMNI View", 8086:56a6 rev 05) - both on CPU 2's slots, PCIe Gen3 x8 and each card has its own PCIe switch in front of the GPU (8086:4fa1), No Resizable BAR on this board (256 MiB BAR)
6 × Kingston valueram 16GB ddr4 2Rx4 PC4-2133P-RA0-11 ECC RDIMM; memtest clean, EDAC 0
Other PCIE: Intel E810 NIC and a AMD FPGA card
Cooler master 1000w Elite Gold (BRAND NEW)
Software
Ubuntu 24.04, HWE kernel 7.0 (7.0.0-34 and 7.0.0-42 both tested)
i915 driver (xe is loaded but not bound)
Firmware: GuC 70.53.0, HuC 7.10.16, DMC 2.08, GSC DG02_2.2357
intel-media-driver (iHD) 24.1.0, later 25.2.4; FFmpeg with VA-API
VT-d on (translated default domain, lazy flush-queue invalidation), interrupt remapping on (x2APIC)
Workload/operation
6 live 1080p50 encodes split across both cards: H.264, HEVC 8/10-bit, AV1 and VP9. Also tested with 8 (4 per card). Each card draws only 14–18 W, at 56–75 °C.
Issue:
The whole server basically stops at once; every network interface goes silent within 0.2 s.
No panic and no reboot, even with kernel.panic=30 and hardlockup_panic=1. My NMI status never reports.
netconsole receives nothing.
After a reset there are no MCE, BERT, EDAC or AER records, with WHEA on or off.
Its not like it turns off, the server still stays powered, fans running; the Q-Code display shows the same AA POST code so nothings different from normal operation.
The board's CATT (CATERR) LED is on and ERR_CPU is off in every freeze where we looked (10 of them). I'm thinking its something to do with the CPU IERR, which explain why the OS logs nothing? I'm very confused.
Tests:
22 freezes in about total of 113 hours of GPU-encode load. The mean is about 5 h; the range is 24 seconds to 14 hours.
I've done the same load with the encodes moved to the CPU and both GPUs idle: ive done a run for 8 hours and one 24 h run and no freeze issues with them...
The GPU encodes alone (no capture cards, no DPDK) still freeze, so im pretty sure its the A310 encodes
Tried, and it still froze:
Kernel 7.0.0-34 → 7.0.0-42, iHD 24 → 25, newer HuC
i915.enable_dc=0
Every OS power saving off: ASPM, runtime PM, D3cold, C-states deeper than C1, SATA LPM, EEE, EPB
Core C6 off, uncore clock capped, turbo off.
Assumptions went to temperature, cooled my room as much as I could the GPU was at 56 °C and still froze.
A310 links forced to Gen2: 8 h clean, then froze in the repeat
intremap=off (interrupt remapping only)
WHEA off
Encode threads and buffers moved to the GPUs' NUMA node
Could test, but haven't (I've done so much testing so I've just tried to post here if anyone has had similar issues and what was the fix):
iommu=pt (GPUs untranslated), or iommu.strict=1 (no lazy flush queue)
The xe driver instead of i915
GSC firmware 2359 (how do you update it on Linux?)
A power limit on the cards through i915's hwmon
A different PSU, or one GPU per CPU socket
My guess:
Both GPUs sit behind the same VT-d unit: the one for CPU 2's root ports, which also covers the E810. My top theory is that two GPUs' constant map/unmap traffic floods that unit with page lookups and invalidations and wedges CPU 2's I/O hub. That would give an IERR with no PCIe error logged.
Or maybe:
cross-socket posted-write credit loss (Broadwell-EP erratum BDF83);
GPU power-state transitions on a lightly loaded card while DMA is in flight.
I've had endless hours and considering just returning the csrds, not sure if return is possible with Overclockers UK ... Alternatively I'll have to have a chat with the manufacturers to get some type of support/repair
Just wondering if anyone else had these issues?
CATERR/IERR hard locks with Arc (DG2) on Broadwell-EP or other older Xeons?
Are there known issues with DG2 under VT-d translation, or with two DG2 cards behind one IOMMU unit?
Is xe on DG2 mature enough for VA-API encodes (HuC/BRC) to be worth testing?
No BMC is fitted (currently shipping from china, but not exactly sure it will be able to identify my exact issue).
Is there any way to capture CPU state at an IERR, such as reading the MCA banks after a warm reset before the BIOS clears them?
What would you try first?
I can post lspci -vvv, dmesg or the kernel config if that helps.
Tldr: I'm stuck, just like my server after any GPU encodes.