Last week I had a really strange problem with one of my TYPO3 websites. A colleague came to me and said he was unable to login to the TYPO3 backend. I tried to login with my own TYPO3 backend account and experienced the same problem.
First I thought the TYPO3 site could have been hacked, so I took a look into the be_users and sys_log tables of the TYPO3 installation but could not see anything suspicious. Also the webservers logfiles did'nt show any errors. I then logged into the TYPO3 install tool and created a new admin backend user for the TYPO3 site. With this user, I should be able to login to the TYPO3 backend... well, at least I thought so. But also with the new admin account I was'nt able to login, so something was really wrong with the TYPO3 website.
Again I looked into the be_users table and saw, that the newly admin account had a normal MD5 hash in the password field. The site used rsaauth and saltedpassword and normally both extensions should be able to distinguish between passwords with MD5 hashes and password with salted hashes, but obviously not on this TYPO3 website. I therefore removed both rsaauth and saltedpasswords, resetted the loginSecurityLevel to "normal" mode in the TYPO3 install tool and was finally able to login to the TYPO3 backend with my new admin user.
So the login problems must have something to do with saltedpasswords and rsaauth. First I checked the path, where rsaauth writes its temporary data, but could not see anything special there. Next I wanted to check the database table, where rsaauth stores the private keys and finally came to the solution of this problem.
The database table tx_rsaauth_keys was marked as crashed and could not be repaired automatically. So the rsaauth extension was unable to write its private keys to the database and therefore the rsaauth login never succeeded. After I reparied the crashed table manually and reenabled rssauth and saltedpasswords, everything worked fine again.