The global/default password hashing algorithm is stored in passwordStorageScheme (cn=congi),
# ldapsearch -x -D cn=directory\ manager -w xxxx -LLL -b "cn=config" -s base passwordStorageScheme
dn: cn=config
passwordStorageScheme: SSHA
To find the hashing algorithm used on an existing user entry:
# ldapsearch -LLL -x -D cn=Directory\ Manager -w xxxx -b <base_dn> uid=luser1 userPassword
dn: uid=luser1,dc=example,dc=com
userPassword:: e1NTSEF9czNPcjAyWWhYV3laSXJCUk9tSnhYU2RnbmJyc1hFTU9BaDFxT3c9PQ==
ldapsearch encodes 'userPassword' attribute by default, decode it and check the {first portion} to get the algorithm used.
# echo e1NTSEF9czNPcjAyWWhYV3laSXJCUk9tSnhYU2RnbmJyc1hFTU9BaDFxT3c9PQ==|base64 -d
{SSHA}s3Or02YhXWyZIrBROmJxXSdgnbrsXEMOAh1qOw==
^^^^^