fix(pty): use OpenSSL crypto backend on Windows for curve25519 KEX

[srstack]

Problem

SSH sessions from a Windows build of otty fail during the handshake against modern SSH servers (OpenSSH >= 10) with:

ssh error: [Session(-5)] Unable to exchange encryption keys

Root cause

The server-side log (OpenSSH 10.0, Debian 13) shows what a Windows build of libssh2 actually offers:

Unable to negotiate with <ip>: no matching key exchange method found.
Their offer: diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,
diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,
diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1 [preauth]

The offer is missing curve25519-sha256 and ecdh-sha2-nistp*. Reason: in libssh2-sys's build.rs, Windows targets default to the WinCNG crypto backend unless the openssl-on-win32 feature is enabled. WinCNG only supports finite-field DH key exchange. OpenSSH 10 servers by default offer only mlkem768x25519, sntrup761x25519, curve25519 and ecdh-sha2-nistp* — zero intersection with WinCNG's offer, so the KEX negotiation fails.

This affects any Rust project using ssh2 with default features on Windows connecting to OpenSSH >= 10 servers, not just otty.

Fix

Enable ssh2's openssl-on-win32 feature. The crate already forces a vendored OpenSSL build (see the comment above the openssl dependency), so libssh2 now compiles against the same vendored OpenSSL on Windows and offers curve25519-sha256 / ecdh-sha2-nistp* again. On unix this feature is a no-op (the OpenSSL branch is already taken there).

Verification

• nm on the Windows libssh2.a after the change: contains _libssh2_curve25519_new / _libssh2_curve25519_gen_k and 70+ EVP_* symbol references (previously none — WinCNG build)
• End-to-end on a Windows host: otty now connects to an OpenSSH 10.0 server and reaches the shell (previously Session(-5) at handshake)
• cargo clippy -p otty-pty --all-targets --all-features -- -D warnings, cargo test -p otty-pty --all-features, cargo deny check all green

[Copilot]

Pull request overview

This PR fixes SSH handshake failures on Windows by ensuring the ssh2/libssh2-sys build uses the OpenSSL crypto backend (rather than the WinCNG backend), restoring support for modern key exchange algorithms required by OpenSSH >= 10 defaults.

Changes:

• Enable ssh2’s openssl-on-win32 feature for Windows builds.
• Add inline documentation explaining why this feature is required (OpenSSH >= 10 KEX compatibility).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[srstack]

Good catch, thank you — you're right that WinCNG/BCrypt itself supports ECDH (NIST P-256); it's the libssh2 WinCNG backend that doesn't implement ECDH/curve25519 KEX. I've rephrased the comment in a61b77b to say exactly that.

[kemokempo]

hm, what do you mean when you tell windows build? otty does not have the windows support right now

[srstack]

Fair question! otty's crates (including the SSH backend in otty-pty) already compile and run on Windows — the codebase is largely platform-neutral (iced/wgpu UI, mio runtime), and otty-pty/src/ssh.rs was already cross-platform. I've been running otty on a Windows host to operate Linux machines over SSH, and this KEX failure was the first hard blocker: with the default features, ssh2 builds libssh2 with the WinCNG crypto backend on Windows, whose KEX offer can't intersect with any reasonably modern sshd (OpenSSH >= 10 defaults), so every SSH connection fails at handshake with Session(-5).

This one-line feature flag is self-contained and doesn't change anything on unix, so I figured it was worth upstreaming independently rather than as part of a larger Windows PR. For context, there's a broader windows-support branch on my fork (SSH quick-launch flow, config paths, CI windows runner, another mio/libssh2 fix at #79) that I intend to propose once these small pieces land.

[kemokempo]

Fair question! otty's crates (including the SSH backend in otty-pty) already compile and run on Windows — the codebase is largely platform-neutral (iced/wgpu UI, mio runtime), and otty-pty/src/ssh.rs was already cross-platform. I've been running otty on a Windows host to operate Linux machines over SSH, and this KEX failure was the first hard blocker: with the default features, ssh2 builds libssh2 with the WinCNG crypto backend on Windows, whose KEX offer can't intersect with any reasonably modern sshd (OpenSSH >= 10 defaults), so every SSH connection fails at handshake with Session(-5).

This one-line feature flag is self-contained and doesn't change anything on unix, so I figured it was worth upstreaming independently rather than as part of a larger Windows PR. For context, there's a broader windows-support branch on my fork (SSH quick-launch flow, config paths, CI windows runner, another mio/libssh2 fix at #79) that I intend to propose once these small pieces land.

Okay, now I get it, thx! Let me take some time to test it on my Windows VM.