The handshake error is most likely to occur when the server/keystore config is non-compliant with a upgraded PayPal SSL encryption algorithms.
IPN simulator will connect againest a PayPal Sandbox environment, which has been upgraded with a Versign G5 root cert / SHA-2 encryption algorithms (Note: This is an industry standard as supposed to be everywhere in the 2016). Once your IPN host failed to meet a compliance, the SSL connnection will error out with a handshake problems.
Simply check the server by running a connection script against a PayPal sandbox endpoint like below:
openssl s_client -connect api-3t.sandbox.paypal.com:443 -showcerts -CApath /etc/ssl/certs/
If you were not getting the
Verify return code: 0 (ok), you may have to check the cert/SHA-2 compatibility.