IDS is not about either penetration testing or malware analysis. An IDS inspects traffic (typically) to look for Indicators of Compromise (IoC).
An IDS, then, is a consumer of whatever the IoCs are, and developing the IoCs is a function of malware and malicious traffic analysis.
So, to develop IoCs, you might want or need to know quite a lot of about both penetration testing (can help with malicious traffic analysis) and malware analysis.
What it sounds like is that you want to develop IoCs for IOT, and then deploy those IoCs in some kind of analysis platform (what you are calling an IDS).
If you want to inspect traffic for previously unknown IoCs and determine IoCs on the fly without any previously defined IoCs, then that's a completely different prospect. I have done this, but you need quite extensive experience in a number of areas. Choosing one over another will restrict what you can detect.