Monday, April 1, 2019

Extbase $query->statement() - What can possibly go wrong?

Last week I had to resolve a problem in a 3rd party Extension, where an Extbase Plugin returned unexpected results when used multiple times on the same page. The problem showed up in the frontend, where the plugin listed some products by a given category. When the plugin was present one time on a page, the output was as following (simplified):

Output of plugin 1
Product 1 for Category 1
Product 2 for Category 1
Product 3 for Category 1

When the plugin was present two times on a page, the output was as following (simplified):

Output of plugin 1 with Category 1 as selection criteria
Product 1 for Category 1 (uid 1)
Product 2 for Category 1 (uid 2)
Product 3 for Category 1 (uid 3)

Output of plugin 2 with Category 2 as selection criteria
Product 1 for Category 2 (uid 10)
Product 2 for Category 1 (uid 2) <-- Whoops!
Product 3 for Category 2 (uid 11)

Somehow, the output of plugin 2 contained a result, that did not belong to the result set. As written, the examples above are simplified. The output on the production website showed hundreds of products, and just some of them were wrong.

In order to debug the problem, I had a look at the Extbase Repository for the Products Domain model and found this (again simplified).

class ProductRepository extends \TYPO3\CMS\Extbase\Persistence\Repository
     * @param $categoryUid
     * @return array|\TYPO3\CMS\Extbase\Persistence\QueryResultInterface
    public function findByCategory($categoryUid)
        $query = $this->createQuery();
        $query->statement('SELECT * FROM tx_products_domain_model_product_' . $categoryUid);
        return $query->execute();

OK... so there are several individual tables for products by category. They all have the same structure and the only difference is, that they have a different name (post-fixed with the category uid) and hold different data. There is also a SQL injection vulnerability, but that has nothing to do with the main problem.

What goes wrong here?

In order to explain, why plugin 2 returns an object, that obviously belongs to plugin 1, you have to know the internals of an Extbase repository, the Extbase QueryResult object and the DataMapper.

Extbase determines the Domain Model based on the Classname. This is done in the constructor of the repository like shown below:

public function __construct(\TYPO3\CMS\Extbase\Object\ObjectManagerInterface $objectManager)
    $this->objectManager = $objectManager;
    $this->objectType = ClassNamingUtility::translateRepositoryNameToModelName($this->getRepositoryClassName());

So when the findByCategory function uses the createQuery() function, the query is initialized to create a query for the object type the Repository determined (in this case Product).

When the query is executed using $query-execute(), it returns an object of the type \TYPO3\CMS\Extbase\Persistence\Generic\QueryResult and here we come closer to the explanation of the problem. The QueryResult object has the following function:

protected function initialize()
    if (!is_array($this->queryResult)) {
        $this->queryResult = $this->dataMapper->map(

This function uses the result from the persistenceManager (raw data from the database with language/workspace overlay) and uses the TYPO3 DataMapper to  create an array with Objects of the given type (Product). The DataMapper does this row by row using the following function mapSingleRow($className, array $row)

And here is the final explanation for the behavior of the 2 plugins on the same page.

protected function mapSingleRow($className, array $row)
    if ($this->persistenceSession->hasIdentifier($row['uid'], $className)) {
        $object = $this->persistenceSession->getObjectByIdentifier($row['uid'], $className);
    } else {
        $object = $this->createEmptyObject($className);
        $this->persistenceSession->registerObject($object, $row['uid']);
        $this->thawProperties($object, $row);
    return $object;

For performance reasons, the DataMapper caches all objects it creates based on their UID. Since the repository in this TYPO3 extension uses different tables (with own UIDs) for data storage, it may happen, that the DataMapper already processed an object with the given UID (but from a different table) and therefore will return a cached version of an object.

So when the output for plugin 1 was created, the DataMapper did create a cached Product object for UID 2 and when the output for plugin 2 was created, the DataMapper returned the cached version of the Product object with UID 2.

So always keep in mind, that an Extbase repository will return objects of exactly one type and that the datasource must always contain unique uids.

Monday, March 18, 2019

Extending Extbase domain models and controllers using XCLASS

In TYPO3 9.5 LTS it has been deprecated (see notice) to extend Extbase classes using TypoScript config.tx_extbase.objects and plugin.tx_%plugin%.objects. In order to migrate existing extensions, which extends another TYPO3 extension, you should now use XLASSes.

For my TYPO3 Extension sf_event_mgt I also provide a small demo extension, which shows how to extend domain models and controllers of the main extension. The previous version using config.tx_extbase.objects can be found here. I migrated this demo extension to use XCLASSes instead.

The code below shows, how two models and one controller are extended using XLASS

// XCLASS event
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Event::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Event::class

// Register extended domain class

// XCLASS registration
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Registration::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Registration::class

// Register extended registration class

// XCLASS EventController
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Controller\EventController::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Controller\EventController::class

For domain models, the important part is the registerImplementation() call, since this instructs Extbase to use the extended domain model when an object is processed by the property mapper.

Note, that there are some limitations using XCLASS, so it is highly recommended to read the official documentation.

Tuesday, July 10, 2018

TYPO3 usage statistics for july 2018

TR;DR - I analyzed 48.146.633 websites for TYPO3 usage - the results with aggregated charts can be found here.

I'm proud to introduce my latest side project called t3versions. It is a Python (Django) web application to identify, if a website is running TYPO3 and if so, which major version is being used. As a result, TYPO3 websites are saved to the database in order to create an aggregated overview with several charts.

Beside the live check in the frontend part, the application also contains a task queue, which can analyze multiple websites in the background an on multiple servers. I imported a list with 48.146.633 domains of european websites to the task queue and after 21 days, all websites were analyzed.

As a result, t3versions did find 292.629 websites using TYPO3. Those websites are located on servers with 61.269 individual IP Addresses. The archived results including 6 charts (Version overview, Supported version, TLDs, Webservers, Countries and SSL) can be found here.

I will reschedule the background check on a regular basis. If someone can provide me a list with e.g. Asian or US domains, feel free to contact me.

Sunday, April 15, 2018

TYPO3 MySQL database import fails with "Index column size too large. The maximum column size is 767 bytes"

I recently migrated a TYPO3 7.6 Website to TYPO3 8.7 and while importing the migrated TYPO3 database on the production server, the import failed with the following MySQL error:

ERROR 1709 (HY000) at line 2060: Index column size too large. The maximum column size is 767 bytes.

The error occurred for the import of the TYPO3 table sys_refindex. After some research and local debugging I found out, that I locally was using MySQL 5.7 and the production server was using MySQL 5.6, but settings in regard to innodb_large_prefix and innodb_file_format were equal.

In order to import the dump from my MySQL 5.7 server to the production MySQL 5.6 Server, I executed the following SQL query before creating the MySQL dump:  


After setting the ROW_FORMAT to DYNAMIC, the database dump from MySQL 5.7 could finally be imported without errors on the MySQL 5.6 production server.

Wednesday, April 11, 2018

TYPO3 extension sf_event_mgt version 3.0 released

Today I finally released the new version 3.0 of my TYPO3 extension sf_event_mgt - Event management and registration. The new version comes with tons of new features, bugfixes and improvements and also contains 2 breaking changes, so make sure to read the release notes.

Thanks to everyone, who contributed to the extension over the last few months. Also a special thanks to Alex Kellner for his extension powermail, from which I adapted ideas and some code for the registration fields feature.

New features 

Below follows some of the new features of sf_event_mgt 3.0.

Registration fields

Im order to make the extension more easy to use for non-programmers, I added the possibility for editors add additional registration fields to the default registration form on event basis.

Registration fields
Creating of registration fields works basically as in powermail. The user can add registrations fields in a new tab as shown in the screenshot above and choose one of the following field types: input, textarea, radio and checkbox.  When a participant registers to an event, all filled out registration fields are saved to the registration record in the TYPO3 backend.

allowLanguageSynchronization for TYPO3 8.7

The TCA settings allowLanguageSynchronization has been introduced in TYPO3 8.6 and is the successor for "l10n_mode=mergeIfNotBlank". Note, that sf_event_mgt did not use "l10n_mode=mergeIfNotBlank", so the language synchronization feature is only available in TYPO3 8.7


Finally, I also added signals in various actions, so it is now easily possible to modify/extend variables given to the desired views.

Monday, February 5, 2018

TYPO3 - How to render a Fluid standalone view multiple times in different languages

Back in 2015, I wrote 2 blogposts (first and second) about rendering a localized Fluid standalone view in a scheduler task (commandController). The main problem was to render a Fluid standalone view multiple times within the same request but with different languages. Back then, my solution was to create an own ViewHelper and a modified version of the TYPO3 LocalizationUtility which were responsible for handling the localization changes during the rendering request.

Meanwhile, I got feedback from readers of my blog pointing me to a more simple solution for the problem.

The main problem with changing the TYPO3 backend language during one request is the language cache, which is only initialized once for the current language. So when you switch the backend language multiple times, the cached language files for the previous language will still be used.

The solution is to unset the language cache for the extension you are rendering your Fluid standalone view from. In order to do so, you have to extend the TYPO3 LocalizationUtility with a new function as shown below.

class LocalizationUtility extends \TYPO3\CMS\Extbase\Utility\LocalizationUtility
     * Resets the language cache for the given extension key
     * @param string $extensionName
    public static function resetLocalizationCache($extensionName)

The example code below shows, how to use the resetLocalizationCache method before rendering a Fluid standalone view in a given language.

 * Renders a Fluid StandaloneView respecting the given language
 * @param string $language The language (e.g. de, dk or se)
 * @return string
public function renderStandaloneView($language = '')
    // Set the extensionKey
    $extensionKey = GeneralUtility::underscoredToUpperCamelCase('standaloneview');

    if ($language !== '') {
        // Temporary set Language of current BE user to given language
        $GLOBALS['BE_USER']->uc['lang'] = $language;

    /** @var \TYPO3\CMS\Fluid\View\StandaloneView $view */
    $view = $this->objectManager->get(StandaloneView::class);
    $template = GeneralUtility::getFileAbsFileName(

    // Set Extension name, so localizations for extension get respected

    return $view->render();

So with a small extension of the LocalizationUtility it is now easily possible to render a Fluid standalone view in a language of choice and it is also possible to switch the language within the same request (e.g. sending out localized e-mails to multiple recipients).

I updated the Demo-Extension, which contains a backend module and a command controller for demonstration purposes.

I would like to thank Johannes Rebhan for giving me the hint about the localization cache and also Ulrich Fischer for showing me a different approach, which requires more code, but lead to the same result.

Thursday, January 25, 2018

SSH reports "Too many Authentication Failures" on first connect

Today I wanted to connect to a new clients SSH server and received a "Too many Authentication Failures" message just on the first connect to the host. After a short break and some Google research, I found the very simple reason for the message.

Since I have several SSH keys in my .ssh/ directory, SSH tries to use each of it to connect to the SSH server. So when the SSH server has a very low "MaxAuthTries" setting configured, then the SSH connection may fail before password authentication is offered.

In order to connect a SSH servers with a low "MaxAuthTries" setting, you can use the following command:

ssh -o PubkeyAuthentication=no [email protected]

After using the "PubkeyAuthentication=no" option, I could login to the host and add a SSH public key to the .ssh/authorized_keys file.