I need to secure the communication between an application (runs on any PC and has been designed for Windows) and an external device (runs on UNIX). The application is running on Windows while the device is Unix-based. I can directly influence the application, but I cannot influence the device (I have to communicate with the guy that is developing the firmware for the devices). Both sides must be on the same LAN in order to communicate with each other, but the traffic between those must be secured. There are several raw TCP-Connections as well FTP. The communication is bidirectional. There is also a directed UDP broadcast but this only happens sometimes and the sent data is not that sensitive so there is no need for protection here yet. A device can be connected to multiple hosts running the application and the application can be connected to multiple devices. I also need to make sure that no ones gets unauthorized access to the devices or manipulates data packets between the application and the devices. This means I need authorisation and integrity checks. My goal is to simply make the TCP and FTP communications inside the LAN secure. It might also be possible that users will need remote access to the devices from outside the LAN but that is a different story. For now I only want to secure the traffic that is happening inside the LAN.
So far so good. I have done some research on what security options are available. My problem appears to be pretty simple yet it seems that there are a lot of different solutions out there. Two of them appear to be suiting my problem perfectly.
TLS/SSL SSH
As far as I know, SSH operates on the application layer whereas TLS/SSL is implemented at above the TCP layer. Does this mean that I basically need an external application that sets up a protected SSH tunnel for data communication? Do I need multiple tunnels for every connection that I am using (raw TCP, FTP)? If going for SSH, I will certainly replace FTP with SFTP. This reduces the number of open ports as far as I know since SFTP is only using one port for both control and data transfer. However, I heard that it is not possible with SFTP to limit the bandwith of the server. We need to decrease the speed at which the FTP server is sending files because the device has limited resources and the traffic might block other data. This is possible with FTP
On the other hand, there is TLS/SSL. I do not need third party programs here, but I need to take care of TLS/SSL in the application I am using itself. This means the application must be aware of encryption, as far as I know. It allows for encryption, authentification and can garantuee data integrity for the communication. Thats nice, but is it possible here to tunnel everything through one connection? (again, I have several TCP connections and FTP transfer). I could replace the FTP with FTPS, but I heard that FTPS (and FTP in general) seems to have problems with NAT or firewalls. In the future, it might be possible that someone needs to have access to the FTP-servers on the devices. However, as I mentioned before, if that happens then we will need a central access-server anyways so that shoiuld be no problem.
Ok, sorry for all the text that I am bothering you with. I am just unsure what solution fits my problem best. My goals are simply to secure the data traffic and minimize the ports that need to be open on the devices or the host which runs the application.
What can you suggest? SSH or TLS/SSL? Or maybe something completely different (like VPN)?