Python as the glue between lab instruments and firmware, where do you draw the line?

1

Hi all, long time reader, first real post.

I work in a small research group where we build custom rigs (mostly optical and sensor stuff) and Python has slowly become the layer that holds everything together. NumPy for the math, a bit of SciPy for signal work, and pyserial or python-can for talking to the boards. That part is smooth.

Where it gets messy is the handoff to firmware development. Half our team writes C on bare metal microcontrollers, the other half lives in Jupyter. Every project eventually hits the same wall: who owns the protocol between the two sides, and how much logic belongs on the MCU versus in the Python layer?

A few honest observations from the last year:

The more we pushed into the firmware, the harder it got for the scientists to iterate. Small experiment tweaks turned into flashing sessions.

The more we pulled into Python, the more we ended up rewriting the same buffering, timing and CRC code across projects.

Async python helped a lot for streaming data at 10 to 50 kHz, but past that we started dropping samples on anything but a very quiet machine.

So I’m curious how others here handle it, especially people doing scientific work rather than product work:

  • Do you keep the MCU dumb and let Python drive everything, or push as much as possible down to firmware and treat Python as a thin client?
  • For those doing serious firmware development alongside your Python analysis code, are you using anything like MicroPython or CircuitPython for the boards, or sticking with C and just improving the serial protocol?
  • Any love for tools like pyocd, or has anyone had a good experience wrapping their firmware build into a Python package so pip install also flashes the board? I’ve seen it done, never seen it done cleanly.
  • How are you testing the Python side without the hardware present? We fake the serial port with a small simulator but it drifts from reality fast.

Not looking for one right answer. Genuinely want to hear what has worked and what has quietly failed for people who live in this hybrid Python plus firmware development world.

Thanks for reading, happy to share our setup in more detail if useful.

2

Hi @hadiaali, and welcome to the forum.

If you are dealing with any platform with near-realtime requirements, you will likely benefit from Python being an interface on top of a low-level library. There are many ways to implement that: some modern options include nanobind (for C) and PyO3 for rust.

I am quite curious about your testing challenge. In the past, we’ve used tools like pytest-vcr to record responses from a web server, which we can then use in our test suite (and easily refresh from time to time). Perhaps a similar approach can work with hardware. Why can the tests not be run against real hardware?

I’m sorry, I don’t have much firmware experience, but others on this forum do and will hopefully provide you with a more detailed response. I should also note that, while there is some overlap, some of the firmware community may be more present on other forums (although I do not know where!).