Operating Systems and Embedded Systems usually don't come with source code or binaries that one can review.

How can a vulnerability researcher look for flaws in a system (Architecture, Protocols, etc.) without any access to source code?

Obviously, black box research will not produce good enough results.

  • Imagine an OS is like a human body. Just think about how germs/toxic get into your body.
    – mootmoot
    Commented Jun 14, 2019 at 8:29
  • I'm not sure I understood what you meant
    – joe
    Commented Jun 14, 2019 at 8:47
  • Germs don't need to know how your body works, it just enters your body through wounds, foods, etc. One doesn't need to know how the OS is built by constantly poking and punching any potential weak point.
    – mootmoot
    Commented Jun 14, 2019 at 8:51
  • 1
    "Obviously, Black box research will not produce good enough results." - citation needed
    – schroeder
    Commented Jun 14, 2019 at 8:59
  • Learn reverse engineering.
    – forest
    Commented Jun 14, 2019 at 9:20

1 Answer 1


Your question has many false assumptions, and these are the reason for your confusion.

An OS doesn't come with source code or binaries.

Yes, it does. Open-Source Operating Systems like Linux come with source code, which is regularly looked at for possible vulnerabilities.

Closed-Source Operating Systems like Windows still come in binary form. If you install your operating system on a hard drive, this hard drive then contains all the code necessary to run the operating system. Furthermore, these binaries come in a specific architecture for the CPUs they are designed to run on. As such, anyone can look at the instructions that the Windows Kernel executes, as long as they have a copy of the operating system.

Embedded Systems don't come with source code or binaries.

Embedded systems are a bit different than "regular" computers, and require specialized hardware to analyze. For example, to analyze a "custom" chip, researchers could dissolve the chip bit-by-bit and thus reverse engineer how the chip works.

How can a researcher understand the underlying system?

Reverse engineering is a simple process: You observe the system, using knowledge you acquired analyzing similar systems, and see if you can draw conclusions from that. For example, when looking at communication between two chips, I can see that communication between two chips is always calling first, and the other chip always answer. The "caller" always sends data in a fixed width, while the "responder" sometimes sends data that's 8 bit long, sometimes 16, 32 or 64 bit long. Sometimes the answers are much longer.

Can you guess what the responder is? It's most likely some sort of storage.

As more and more of the communication is being observed, the "protocol" that is used to communicate can be deciphered. It doesn't need to be complete, but it could be observed that the communication is always in this form:

| 2 Bytes | 4 Bytes | 2 Bytes | 8 Bytes | Variable Bytes | 0x00FF00FF |
| Unknown | Unknown | Unknown | Length  | Payload        | EoT        |

While you might not know yet what exactly these bytes mean, you can look at correlations between them. For example, you notice that the 8 bytes denote the length of the payload, followed by the payload, followed by the constant 0x00FF00FF.

This process keeps going, until you can be reasonably sure that the "calls" and "responses" you see all make sense. You can then attempt to insert your own "calls" and see if the responses are what you expect them to be.

Obviously, black box research will not produce good enough results.

The universe didn't come with a conveniently formatted man page, but we observed how things acted, drew hypotheses from that, made experiments and confirmed those hypotheses.

This allowed us to understand classical mechanics, thermodynamics, fluid mechanics, acoustics, optics and more. None of these things came with documentation, and they were essentially the same as a black box to us.

Reverse Engineering is the same thing. You observe, you hypothesize, you experiment, you succeed.

  • Most "custom" chips are just variations of ARM, MIPS or, in extreme cases, ARC. It's generally not necessary to do a complete chip-off analysis when the chip with high likelihood uses a well-known and public ISA.
    – forest
    Commented Jun 14, 2019 at 10:33
  • @forest True, but I had something smaller in mind, like a custom DAC or some Op-Amp in a Tube Screamer, rather than a SoC.
    – user163495
    Commented Jun 14, 2019 at 10:38
  • Ah I see. So something you couldn't just attach a JTAG probe to.
    – forest
    Commented Jun 14, 2019 at 10:39
  • @forest Yeah, my mind was really on something like a guitar pedal with some digital chip that does God-knows-what. My intention was to show that you can analyze what a chip does by looking at it. It's probably the most economic solution to see if it's similar enough to something pre-existing.
    – user163495
    Commented Jun 14, 2019 at 10:41
  • Thank you very much! but I still have to ask, some vulnerabilities for example are long chain of very specific low level bugs that creates the final big picture. its practically impossible to find them out of the blue. You can try, but sometimes you just must to get some technical information, those kind of things that happen inside the system, and not visible outside. thank you again!
    – joe
    Commented Jun 14, 2019 at 10:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .