If you have OpenSSL then use SSL/TLS: the protocol which is simplest to implement is the one which is already implemented. SSL just offers a raw tunnel, so you would have to include some kind of management protocol which organizes the actual "file transfer", e.g. HTTP. HTTP within SSL is known as HTTPS: that's standard, so the hard work of specifying the protocol is already done.
Without OpenSSL, I would still personally recommend SSL/TLS for the following reason: I have done it (SSL client and server for embedded systems, each fits in less than 20 kB of code). So I know it is doable. Also, there are other SSL/TLS libraries targeted at small systems, e.g. CyaSSL or axTLS.
Reimplementing the SSH protocol should not be awfully harder than reimplementing SSL, but the SSH specification is bigger (it begins at RFC 4251, which describes the architecture, and the actual protocol is spread over several subsequent RFC; on the other hand, SSL/TLS fits in the single RFC 5246). That's obviously a not-very-objective way of choosing a protocol, but it is not totally uncalled for either.
For SSL/TLS, I suggest the following:
Use TLS 1.2 and do not bother with previous versions (you control the client and the server, so you can enforce a specific version). TLS 1.2 requires the implementation of only one hash function, while previous versions required both MD5 and SHA-1.
Use a single cipher suite (e.g. TLS_RSA_WITH_AES_128_CBC_SHA256
). This simplifies things quite a bit and allows you to hardcode things like block size or MAC length.
Use RSA key exchange and arrange for your system to be the "client": this allows it to use RSA in encryption mode only, which is notably simpler to implement than decryption. Also, the client needs not store an asymmetric private key, which is good since secure key storage is difficult.
Do not try to decode certificates. In normal SSL/TLS, the server sends its public key to the client as part of an X.509 certificate chain; the client is supposed to validate the chain with regards to a hard-coded public key. Just cut the middle-man, and hardcode the server public key in the client (X.509 certificate chain validation can be done -- even in 6 kB of code -- but it is utterly tricky).
Either way, without a crypto library on hand, you will have to reimplement a few basic algorithms, and that's known not to be easy (especially if you want to avoid side-channel attacks, which are always a concern with embedded systems). You will also need access to a cryptographically secure source of random number: on a pre-boot system, this may prove difficult. But it is extremely important for security.