https://bugzilla.redhat.com/show_bug.cgi?id=2062202
--- Comment #21 from Hubert Kario hkario@redhat.com --- CVE-2022-0778 is a bug that causes OpenSSL to enter an infinite loop in the BN_mod_sqrt() function.
The problem is caused by using the BN_mod_sqrt() function with the p parameter that's not a prime number (something that was documented since the function was added to OpenSSL: https://github.com/openssl/openssl/blame/22b52164aaed31d6e93dbd2d397ace04136... as incorrect).
The way to cause OpenSSL to call BN_mod_sqrt() with p that's not a prime is to provide OpenSSL with custom elliptic curve parameters and a point in compressed form.
All the elliptic curves we support for ECDSA and ECDH operations are defined by the following equation: y^2 = x^3 + ax + b mod p. The x and y are the coordinates of a point, which in Elliptic Curve Cryptography (ECC) is the public key. the a, b, and p are the parameters of the curve.
The secp224r1, secp256k1, secp384r1, secp521r1 and prime256v1 curves specify values for a, b, and p. For interoperability there's also a need to specify a so-called base point G. That is just one specific point (pair of x and y values) that satisfies the curve equation. The base point is used as a sort-of "starting position" for calculations in ECDSA and ECDH.
All of those values (a, b, p, G) are static for all operations on those curves (signatures, verification, and key exchange) irrespective of the public and private key values. Since they are static and shared between all implementations, protocols generally just specify a generic short identifier of the curve instead of sending all the parameters explicitly. Some protocols, like ECDHE in TLS 1.3, don't allow sending the parameters explicitly, some, like X.509, do support both specifying the curve as an identifier (so-called "named curve") and as explicit parameters, some, like ECDHE in TLS 1.2 do support both identifiers and explicit parameters, but there are no implementations that support explicit parameters.
Note that if you know the value of x, a, b and p, you can calculate the possible values of y by calculating the square root of (x^3 + ax + b) mod p. That property is used to create so-called "compressed" representation of the point, where the point is specified only as the x coordinate and the sign of y (reminder: equation y^2 = 4 has two solutions: y = 2 and y = -2).
So to perform the attack OpenSSL needs to receive explicit elliptic curve parameters with p that's not a prime, accept them, and then attempt to recover the y value of a compressed representation of a point.
The versions of OpenSSL included in Red Hat Enterprise Linux have been modified to support only specific curve parameters: prime256v1, secp384r1, etc. That check, if the provided parameters match one of those well-known curves, is performed before attempting to decode the G point or read any public point (public key). Since all the curves that OpenSSL on RHEL does support use p that's a prime number, the attack is not possible against OpenSSL as distributed in Red Hat Enterprise Linux.