In light of the recent FREAK vulnerability I am trying to get a hold on what ciphers are being used by inbound connections. I have FTP tracing on which is giving me which ciphers are being negotiated between external clients and the system I am using. However, I would like to know what actually negotiates the SSL cipher? Is it negotiated by the client software such as FileZilla, etc or is it negotiated by the OS? Something else?
In SSL/TLS the client suggests but the server chooses. The client sends a list of supported cipher suites, and the server selects one in that list. The list sent by the client is ordered by preference, but nothing forces the server to really honour the preferences of the client.
On both the client and server side, as @schroeder explains, the application software really does the job, but it may keep its configuration in a centralized emplacement, especially if the actual protocol implementation is provided by the OS. In general, such centralization is a lot more common in the Windows world than in the Linux ecosystem.
The FileZilla client uses GnuTLS for its SSL/TLS implementation; as far as I know, as a library, it does not use any centralized configuration for client-side protocol details, so whatever list of supported cipher suites sent by FileZilla will be decided upon by the FileZilla application code itself -- or, more probably, they just use the default settings of GnuTLS.
Cipher suite negotiation happens in the TLS handshake as the first step. The applications/services themselves determine what cipher suites they support and both ends agree on what suites to use.
The question of "does the OS negotiate the cipher suite" is a little confused in Windows by the fact that many of the service settings (IIS, for example) are stored in the registry, which can make it look like the OS is the one doing the negotiation. But, the idea is still that it is the service (w3svc) that does the negotiating, and its settings are stored in the OS registry.