Friday, November 22, 2019

TYPO3 Extension "Plain FAQ" released

Today I released the first public version of my latest TYPO3 Extension called "Plain FAQ". The name already says it - the extension is a very simple solution to manage Frequently Asked Questions in TYPO3. Below follow some of the facts about the Extension:
  • Compatible with TYPO3 8.7 and 9.5
  • Based on Extbase and Fluid
  • Covered with unit and functional tests
  • Easy usage for editors
  • Uses TYPO3 system categories to structure FAQs by category
  • Field for media and files
  • Possibility to add related FAQs
  • Configurable template layouts for the views
  • Automatic cache cleanup when a FAQ article has been updated in backend
  • Signal slots to extend the extension with own functionality
  • Symfony Console commands to migrate from ext:irfaq
The extension is available on TER and packagist.

Migration from "Modern FAQ (irfaq)"

If you currently use the TYPO3 Extension "Modern FAQ (irfaq)" you may have noticed, that the extension is not compatible to TYPO3 9.5 (last Extension-Update in March 2018)  and the architecture is quite old (AbstractPlugin, Marker Based Templates and TypoScript configuration for wraps). 

For users where "Modern FAQ (irfaq)" is a blocker for an upcoming TYPO3 9.5 Update, it is possible to migrate to "Plain FAQ" using the Symfony Console Commands included in "Plain FAQ". The migration is as easy as the usage of the extension:

1. Migrate existing Categories to sys_category
2. Migrate existing FAQs to "plain_faq" records
3. Migrate existing Plugins including Plugin settings

The migration may not cover all possible scenarios (e.g. Ratings, Question asked by, irfaq Plugin settings set by TypoScript), but is for sure a good starting point in order to migrate existing records. I guess, for most websites the included migration will suite without further work on migrated data.

You can find details about the migration process in the Extension Manual.

Thanks for sponsoring


I would like to thank Julius-Maximilians-Universit├Ąt W├╝rzburg for sponsoring the initial development of the TYPO3 extension. Thanks for supporting TYPO3 and open source software!







Monday, September 2, 2019

How to fix TYPO3 error "There is no column with name 't3ver_oid' on table"

Recently the following error message showed up in a project I was updating to TYPO3 9.5:

There is no column with name 't3ver_oid' on table 'tx_news_domain_model_news'.

When you see this message in a TYPO3 project, you should of course first check, if the field is really available in the mentioned table and second, you should check, if extension dependencies are correct.

When extending an Extbase extension (in this case ext:news), you must ensure, that the extending extension is loaded after the extension you extend. In order to do so, you must add a dependency to the extension you extend in your ext_emconf.php like shown below (example when extending ext:news):


'constraints' => [
    'depends' => [
        'news' => '7.3.0-7.3.99'
    ],
],

After adding the dependency, make sure to regenerate PackageStates.php ("Dump Autoload Information" in install tool or "composer dumpautoload"  for composer projects)

Sunday, August 25, 2019

How to disable the nginx TYPO3 cache set by ext:nginx_cache in development context

When you run TYPO3 on nginx webservers, you can use the nginx FastCGI cache to really increase the performance of your website. Basically, the nginx FastCGI cache stores pages rendered by TYPO3 in the webservers memory. So once a page is cached in the nginx FastCGI cache, it will be delivered directly from the webservers memory which is really fast.

When using nginx FastCGI cache, the TYPO3 extension nginx_cache can be used to ensure that the nginx FastCGI cache is purged, when TYPO3 content changes. Also the extension ensures, that pages are removed from the cache, when the TYPO3 page lifetime expires.

However, when you do not need the nginx FastCGI cache while developing locally (e.g. no cache configured or even a different webserver), the clear cache function of TYPO3 results in an error message.


In the TYPO3 log you can find messages like shown below:


Core: Exception handler (WEB): Uncaught TYPO3 Exception: #500: Server error: `PURGE https://www.domain.tld/*` resulted in a `500 Internal Server Error`

The message states, that the PURGE request to the nginx FastCGI cache failed, simply because the PURGE operation is not allowed or configured.

Since the extension nginx_cache has no possibility to disable it functionality, you can remove it locally. But if you have the file PackageStates.php in version control, uninstalling the extension can be error prone, since one by accident may commit the PackageStates.php without ext:nginx_cache installed resulting in the cache to be disabled on the production system.

In order to disable the extensions functionality locally (or in your development environment), you should add the following to the ext_localconf.php file of your sitepackage:

// Disable nginx cache for development context
if (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()->isDevelopment()) {
    unset($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['nginx_cache']);
    unset($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/cache/frontend/class.t3lib_cache_frontend_variablefrontend.php']['set']['nginx_cache']);
    unset($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['tslib/class.tslib_fe.php']['pageLoadedFromCache']['nginx_cache']);
}

This will unset the cache and all hooks set by the extension if TYPO3 runs in development context. Note, that adding the snippet to AdditionalConfiguration.php will not work, since the ext_localconf.php of ext:nginx_cache is processed after AdditionalConfiguration.php.

Finally, you have to ensure, that the new code is processed after the ext_localconf.php of ext:nginx_cache is processed. In order to do so, you have must add a dependency to ext:nginx_cache the ext_emconf.php of your sitepackage as shown below:

    'constraints' => [
        'depends' => [
            'nginx_cache' => '2.1.0-9.9.99',
        ],
        'conflicts' => [],
    ]

I should be noted, that the shown technique can also be used to unset/change a lot of other settings which are made by installed extensions.

Monday, April 1, 2019

Extbase $query->statement() - What can possibly go wrong?

Last week I had to resolve a problem in a 3rd party Extension, where an Extbase Plugin returned unexpected results when used multiple times on the same page. The problem showed up in the frontend, where the plugin listed some products by a given category. When the plugin was present one time on a page, the output was as following (simplified):

Output of plugin 1
Product 1 for Category 1
Product 2 for Category 1
Product 3 for Category 1

When the plugin was present two times on a page, the output was as following (simplified):

Output of plugin 1 with Category 1 as selection criteria
Product 1 for Category 1 (uid 1)
Product 2 for Category 1 (uid 2)
Product 3 for Category 1 (uid 3)

Output of plugin 2 with Category 2 as selection criteria
Product 1 for Category 2 (uid 10)
Product 2 for Category 1 (uid 2) <-- Whoops!
Product 3 for Category 2 (uid 11)

Somehow, the output of plugin 2 contained a result, that did not belong to the result set. As written, the examples above are simplified. The output on the production website showed hundreds of products, and just some of them were wrong.

In order to debug the problem, I had a look at the Extbase Repository for the Products Domain model and found this (again simplified).


class ProductRepository extends \TYPO3\CMS\Extbase\Persistence\Repository
{
    /**
     * @param $categoryUid
     * @return array|\TYPO3\CMS\Extbase\Persistence\QueryResultInterface
     */
    public function findByCategory($categoryUid)
    {
        $query = $this->createQuery();
        $query->statement('SELECT * FROM tx_products_domain_model_product_' . $categoryUid);
        return $query->execute();
    }
}

OK... so there are several individual tables for products by category. They all have the same structure and the only difference is, that they have a different name (post-fixed with the category uid) and hold different data. There is also a SQL injection vulnerability, but that has nothing to do with the main problem.

What goes wrong here?


In order to explain, why plugin 2 returns an object, that obviously belongs to plugin 1, you have to know the internals of an Extbase repository, the Extbase QueryResult object and the DataMapper.

Extbase determines the Domain Model based on the Classname. This is done in the constructor of the repository like shown below:

public function __construct(\TYPO3\CMS\Extbase\Object\ObjectManagerInterface $objectManager)
{
    $this->objectManager = $objectManager;
    $this->objectType = ClassNamingUtility::translateRepositoryNameToModelName($this->getRepositoryClassName());
}

So when the findByCategory function uses the createQuery() function, the query is initialized to create a query for the object type the Repository determined (in this case Product).

When the query is executed using $query-execute(), it returns an object of the type \TYPO3\CMS\Extbase\Persistence\Generic\QueryResult and here we come closer to the explanation of the problem. The QueryResult object has the following function:

protected function initialize()
{
    if (!is_array($this->queryResult)) {
        $this->queryResult = $this->dataMapper->map(
            $this->query->getType(),
            $this->persistenceManager->getObjectDataByQuery($this->query)
        );
    }
}

This function uses the result from the persistenceManager (raw data from the database with language/workspace overlay) and uses the TYPO3 DataMapper to  create an array with Objects of the given type (Product). The DataMapper does this row by row using the following function mapSingleRow($className, array $row)

And here is the final explanation for the behavior of the 2 plugins on the same page.

protected function mapSingleRow($className, array $row)
{
    if ($this->persistenceSession->hasIdentifier($row['uid'], $className)) {
        $object = $this->persistenceSession->getObjectByIdentifier($row['uid'], $className);
    } else {
        $object = $this->createEmptyObject($className);
        $this->persistenceSession->registerObject($object, $row['uid']);
        $this->thawProperties($object, $row);
        $this->emitAfterMappingSingleRow($object);
        $object->_memorizeCleanState();
        $this->persistenceSession->registerReconstitutedEntity($object);
    }
    return $object;
}

For performance reasons, the DataMapper caches all objects it creates based on their UID. Since the repository in this TYPO3 extension uses different tables (with own UIDs) for data storage, it may happen, that the DataMapper already processed an object with the given UID (but from a different table) and therefore will return a cached version of an object.

So when the output for plugin 1 was created, the DataMapper did create a cached Product object for UID 2 and when the output for plugin 2 was created, the DataMapper returned the cached version of the Product object with UID 2.

So always keep in mind, that an Extbase repository will return objects of exactly one type and that the datasource must always contain unique uids.



Monday, March 18, 2019

Extending Extbase domain models and controllers using XCLASS

In TYPO3 9.5 LTS it has been deprecated (see notice) to extend Extbase classes using TypoScript config.tx_extbase.objects and plugin.tx_%plugin%.objects. In order to migrate existing extensions, which extends another TYPO3 extension, you should now use XLASSes.

For my TYPO3 Extension sf_event_mgt I also provide a small demo extension, which shows how to extend domain models and controllers of the main extension. The previous version using config.tx_extbase.objects can be found here. I migrated this demo extension to use XCLASSes instead.

The code below shows, how two models and one controller are extended using XLASS


// XCLASS event
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Event::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Event::class
];

// Register extended domain class
GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\Container\Container::class)
    ->registerImplementation(
        \DERHANSEN\SfEventMgt\Domain\Model\Event::class,
        \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Event::class
    );

// XCLASS registration
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Registration::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Registration::class
];

// Register extended registration class
GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\Container\Container::class)
    ->registerImplementation(
        \DERHANSEN\SfEventMgt\Domain\Model\Registration::class,
        \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Registration::class
    );

// XCLASS EventController
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Controller\EventController::class] = [
    'className' => \DERHANSEN\SfEventMgtExtendDemo\Controller\EventController::class
];


For domain models, the important part is the registerImplementation() call, since this instructs Extbase to use the extended domain model when an object is processed by the property mapper.

Note, that there are some limitations using XCLASS, so it is highly recommended to read the official documentation.

Tuesday, July 10, 2018

TYPO3 usage statistics for july 2018

TR;DR - I analyzed 48.146.633 websites for TYPO3 usage - the results with aggregated charts can be found here.

I'm proud to introduce my latest side project called t3versions. It is a Python (Django) web application to identify, if a website is running TYPO3 and if so, which major version is being used. As a result, TYPO3 websites are saved to the database in order to create an aggregated overview with several charts.

Beside the live check in the frontend part, the application also contains a task queue, which can analyze multiple websites in the background an on multiple servers. I imported a list with 48.146.633 domains of european websites to the task queue and after 21 days, all websites were analyzed.

As a result, t3versions did find 292.629 websites using TYPO3. Those websites are located on servers with 61.269 individual IP Addresses. The archived results including 6 charts (Version overview, Supported version, TLDs, Webservers, Countries and SSL) can be found here.

I will reschedule the background check on a regular basis. If someone can provide me a list with e.g. Asian or US domains, feel free to contact me.

Sunday, April 15, 2018

TYPO3 MySQL database import fails with "Index column size too large. The maximum column size is 767 bytes"

I recently migrated a TYPO3 7.6 Website to TYPO3 8.7 and while importing the migrated TYPO3 database on the production server, the import failed with the following MySQL error:

ERROR 1709 (HY000) at line 2060: Index column size too large. The maximum column size is 767 bytes.

The error occurred for the import of the TYPO3 table sys_refindex. After some research and local debugging I found out, that I locally was using MySQL 5.7 and the production server was using MySQL 5.6, but settings in regard to innodb_large_prefix and innodb_file_format were equal.

In order to import the dump from my MySQL 5.7 server to the production MySQL 5.6 Server, I executed the following SQL query before creating the MySQL dump:  

ALTER TABLE sys_refindex ROW_FORMAT=DYNAMIC;

After setting the ROW_FORMAT to DYNAMIC, the database dump from MySQL 5.7 could finally be imported without errors on the MySQL 5.6 production server.