A null byte is a byte with the value zero, i.e. 0x00 in hex.
There have been security vulnerability related to null bytes. These occur because C uses null bytes as a string terminator. Other languages (Java, PHP, etc.) don't have a string terminator; they store the length of every string separately.
Now, consider a Java web application that accepts file uploads. Perhaps we want to let users upload
.jpg files, but nothing else. In fact, if a user can upload a
.jsp file this will be a serious security vulnerability.
What a hacker might try is to upload
hack.jsp<NUL>.jpg Lets think about how this will be processed. First, Java will look at the file name, see it ends in
.jpg and allow the upload. It then calls the operating system library, which is written in C. C sees the
<NUL> character as the string terminator, so it saves the file as
Many languages fix this by explicitly disallowing bytes in file names. I know Python and PHP do this. However, if your language does not do this for you, you must do it yourself. More information - OWASP: Null-Byte Injection
I don't know how exactly "serialised instances" is related to this, but I think this gives you some idea what's going on.