Sunday, August 15, 2021

"Unterminated nested statement!" using TYPO3 rector

TYPO3 rector is a really helpful application when it comes to TYPO3 major updates. It helps you to identify and refactor TYPO3 deprecations in custom extensions and can save hours of manual refactoring. I use TYPO3 rector quite a lot and stumbled across the following error recently.

"Unterminated nested statement!"







This message is not really helpful, so I digged deeper into the problem. The "Parser.php" throwing the exception is located in "helmich/typo3-typoscript-parser" package, so I first thought that there was a problem with the TypoScript in the desired extension, but after checking ever line manually, I could not find any error. 

It came out, that the extension I wanted to process by TYPO3 rector had a "node_modules" folder, which contained a lot of Typescript (not TypoScript) files. Those files where obviously parsed by the TypoScript parser resulting in the shown error message. After removing (excluding should also work) the "node_modules" folder, everything worked as expected.

If you like rector and/or TYPO3 rector, please consider to support the authors.
 

Friday, June 4, 2021

How to use constructor dependency injection in a XCLASSed TYPO3 class

Some time ago in needed to extend an Extbase controller in TYPO3 10.4 which used dependency injection through constructor injection. So I used XCLASS to extend the original controller and added an own constructor which added an additional dependency, but this obviously did not work out properly, since the constructor was always called with the amount of arguments from the original class.

Later I created this issue on TYPO3 forge in order to find out if this is a bug/missing feature or if I missed something in my code. In order to demonstrate the problem, I created this small demo extension which basically just extended a TYPO3 core class using XCLASS and just days later, a solution for the issue was provided.

The solution is pretty simple and you just have to ensure to add a reference to the extended class in the Services.yaml file of the extending extension.

Example:


  TYPO3\CMS\Belog\Controller\BackendLogController: '@Derhansen\XclassDi\Controller\ExtendedBackendLogController'

The complete Services.yaml file can be found here.

Thanks a lot to Lukas Niestroj, who pointed out the solution to the problem.


Monday, March 22, 2021

How to migrate switchableControllerActions in a TYPO3 Extbase extension to single plugins

TL;DR - I created this TYPO3 update wizard which migrates plugins and Extbase plugin settings for each former switchable controller actions configuration entry.

Since switchableControllerActions in Extbase plugins have been deprecated in TYPO3 10.4 and will be removed in either TYPO3 11 but most likely 12, I decided to remove switchableControllerActions in my TYPO3 Extbase extensions already with the upcoming versions that will be compatible with TYPO3 11.

In this blogpost I will show, how extension authors can add a smooth migration path to their existing extensions by adding an update wizard which migrates all existing plugin settings, so users do not have to change plugin settings manually. 

As a starting point lets have a look at my TYPO3 extension Plain FAQ, which is a very simple Extbase extension with one plugin, that has 3 switchableControllerActions.

  • Faq->list;Faq->detail
  • Faq->list
  • Faq->detail
For all 3 switchableControllerActions, I created 3 individual plugins (Pilistdetail, Pilist, Pidetail) which handle the action(s) of each switchable controller action from the list above. 

For each new plugin, I added an individual FlexForm file which holds the available settings for the plugin. This can be done by duplicating the old FlexForm (Pi1 in this case) and removing those settings, which are not available in the new plugin. Also display conditions based switchableControllerActions must be removed.

Finally I created a new item group for the Plugins of the extension, so all plugins are grouped as shown on the screenshot below.


This is basically all work that needs to be done on code side in order split the old plugin to the new plugins.

Migration of existing plugins


To be able to migrate all existing plugins and settings to the new plugins, I created a custom upgrade wizard that takes care of all required tasks. Those tasks are as following:
  • Determine, which tt_content record need to be updated
  • Analyse existing Plugin (field: list_type) and switchableControllerActions in FlexForm (field: pi_flexform)
  • Remove non-existing settings and switchableControllerAction from FlexForm by comparing settings with new FlexForm structure of target plugin
  • Update tt_content record with new Plugin and FlexForm
As a result, a SwitchableControllerActionsPluginUpdater has been added to the extension. It takes care of all mentioned tasks and has a configuration array which contains required settings (source plugin, target plugin and switchableControllerActions) for the migration.

private const MIGRATION_SETTINGS = [
    [
        'sourceListType' => 'plainfaq_pi1',
        'switchableControllerActions' => 'Faq->list;Faq->detail',
        'targetListType' => 'plainfaq_pilistdetail'
    ],
    [
        'sourceListType' => 'plainfaq_pi1',
        'switchableControllerActions' => 'Faq->list',
        'targetListType' => 'plainfaq_pilist'
    ],
    [
        'sourceListType' => 'plainfaq_pi1',
        'switchableControllerActions' => 'Faq->detail',
        'targetListType' => 'plainfaq_pidetail'
    ],
];
So basically, one configuration entry has to be added for each switchable controller action setting of the old plugin. The wizard determines the new FlexForm settings using configured TCA, removes all non-existing settings (which is important, since TYPO3 will pass every setting available in pi_flexform to Extbase controllers and Fluid templates) and changes the "old" Plugin to the new one.

The update wizard can possibly also be used in other Extbase extensions, since the MIGRATION_SETTINGS are the only configuration options that need to be changed.

The required changes for the complete removal of switchableControllerActions is available in this commit.

Tuesday, December 29, 2020

Replace functionality of TYPO3 extension mw_keywordlist with a custom sitemap content element

One of the big pain points when it comes to TYPO3 major updates are extensions. If the extension stack contains unmaintained / unsupported extensions, updating can really be hard, since existing functionality needs to be replaced and existing data needs to be migrated.

I recently had this problem on a website, where the TYPO3 extension mw_keywordlist (A-Z Keyword List) was used. The project was running on TYPO3 8.7 and was about to be updated to TYPO3 10.4, but an updated version of the extension was not available, so it was a major blocker in the project.

The extension creates a sitemap generated from keywords in the "pages.keywords" field and renders this sitemap in alphabetical order grouped by keyword. So basically it creates just another type of sitemap, which TYPO3 already has other content elements for. In the project team we decided to replace the extension with a custom sitemap content element, which uses a custom dataProcessor to group pages by the configured keywords.

In the sitepackage of the project the following files were added:

  • TCA override for tt_content, so the new content element (CType) "menu_keywordlist" is registered
  • PageTS Config to add the new content element to the content element wizard
  • TypoScript to configure rendering of the new sitemap content element
  • A custom DataProcessor to group pages by keyword
  • A Fluid template to define how the markup is generated
All required files are shown in this GitLab code snippet

After the new sitemap content element has been configured and tested, all existing instances of the mw_keywordlist content element were replaced with the new custom sitemap element. This was done using the following SQL query:


UPDATE `tt_content`
SET `CType` = 'menu_keywordlist'
WHERE `CType` = 'mw_keywordlist_pi1' OR `CType` = 'menu_mw_keywordlist_pi1';

After the existing content elements were replaced, the extension mw_keywordlist could be removed. The new solution was added to the website when it was still running on TYPO3 8.7, since the code is compatible with TYPO3 8.7, 9.5 and 10.4

Thanks to the University of W├╝rzburg for allowing me to share the code of this solution.

Thursday, November 19, 2020

Unexpected sorting behavior after update from MariaDB 10.1 to 10.3

TL;DR The sorting behavior changed from MariaDB 10.1 to 10.2 due to a bug in MariaDB 10.1 

After updating from Ubuntu 18.04 LTS to 20.4 LTS a previously working a PHP application which contains a data export suddenly did not return the expected result any more. I debugged this scenario by comparing the database query results in the data export and obviously, something in the sorting changed from MariaDB 10.1 to MariaDB 10.3

In order to fully reproduce the problem, I created a really simple use case as shown in the SQL dump below.


CREATE TABLE `test` (
  `a` int(11) NOT NULL AUTO_INCREMENT,
  `b` varchar(255) NOT NULL,
  `c` text NOT NULL,
  `d` varchar(255) NOT NULL,
  PRIMARY KEY (`a`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;

INSERT INTO `test` VALUES (1,'A','\r\n','CRLF'),(2,'A','',''),(3,'A','',''),(4,'A','\r\n','CRLF'),(5,'A','','');

So we have a really simple table structure with some data. The only special thing is, that 2 values in column "c" contain a carriage return and line feed (CRLF). Since this is not printed when selecting data, I also added column d which contains the value "CRLF" for rows, where column c is a CRLF.

So now I select some data. 

SELECT * FROM test;


+---+---+----+------+
| a | b | c  | d    |
+---+---+----+------+
| 1 | A |    | CRLF |
| 2 | A |    |      |
| 3 | A |    |      |
| 4 | A |    | CRLF |
| 5 | A |    |      |
+---+---+----+------+

This result is as I would expect it. Now sorting comes into the game...

Ubuntu 18.04 with MariaDB 10.1.47

SELECT * FROM test ORDER BY b ASC, c ASC, a ASC;


+---+---+----+------+
| a | b | c  | d    |
+---+---+----+------+
| 2 | A |    |      |
| 3 | A |    |      |
| 5 | A |    |      |
| 1 | A |    | CRLF |
| 4 | A |    | CRLF |
+---+---+----+------+

OK, so the sorting of column c puts the CRLF values at the end for MariaDB 10.1. Now I try the same on another system.

Ubuntu 20.04 with MariaDB 10.3.25

SELECT * FROM test ORDER BY b ASC, c ASC, a ASC;


+---+---+----+------+
| a | b | c  | d    |
+---+---+----+------+
| 1 | A |    | CRLF |
| 4 | A |    | CRLF |
| 2 | A |    |      |
| 3 | A |    |      |
| 5 | A |    |      |
+---+---+----+------+

As you notice, the sorting for column c is now reversed...

I did not find a setting in MariaDB 10.3 to switch back to the sorting as it was in MariaDB 10.1. I could also reproduce the same behavior on MySQL 8.0. So... bug or feature - who knows? I think the described scenario can be considered as an edge case, but if you somehow depend on, that sorting for a column with CRLF values is exactly the same, this can hit you really hard.

I created an issue in the MariaDB bug tracker. I'm curious if this is supposed behavior or not.

Update 23.11.2020: It has been confirmed, that the sorting behavior is as expected in MariaDB 10.2+ and that it was wrong in 10.1


Monday, November 16, 2020

How to extend existing FlexForm select options of a TYPO3 plugin using Page TSconfig

Sometimes existing options of a TYPO3 plugin may not fully suite the project requirements. As an example, I refer to my TYPO3 extension "Event Management and Registration" (sf_event_mgt). The extension allows to select the ordering of records by a specific field in the FlexForm plugin options as shown on the screenshot below.


The 3 options shown are configured in the Flexform options for the field "settings.orderField".

In a project it was required to order by a custom field which was not part of the main extension. So I added the custom field named "slot" to the extension using an extending extension for sf_event_mgt.

In order to allow the new field as sorting field, the field "slot" needs to be added to the allowed ordering fields using TypoScript (note, this step is only specific to the extension sf_event_mgt).


plugin.tx_sfeventmgt {
  settings {
    orderFieldAllowed = uid,title,startdate,enddate,slot
  }
}

Finally the FlexForm of the plugin needs to be extended, so the new field appears in the "Sort by" select field. In order to do so, the following Page TSconfig has been added:


TCEFORM.tt_content.pi_flexform.sfeventmgt_pievent.sDEF.settings\.orderField {
  addItems.slot = slot
  altLabels.slot = Slot
}

You might notice the backslash before the dot in "settings\.orderField". This is required to escape the dot of the fieldname "settings.orderField", since Page TSconfig also uses dots to separate between configuration options/levels.

After adding the Page TSconfig, the plugin now shows the new field.

Pretty cool and not a single line of PHP code required :-) 

Reference: TYPO3 TCEFORM

Friday, June 26, 2020

Testing email delivery of a TYPO3 extension with Codeception, MailHog and GitHub Actions

Some weeks ago I published my blogpost about how to create a GitHub Actions build pipeline for a TYPO3 Extension that executes Unit-, Functional- and Acceptance tests. The extension tested in that blogpost was only a simple demo extension and for me this was a preparation to finally migrate the CI pipeline for my TYPO3 extension sf_event_mgt to GitHub Actions.

The extension comes with lots of unit and functional tests, which are automatically executed on each commit. One missing piece in the puzzle was the automatic execution of my Acceptance Tests, which are based on Codeception and additionally require MailHog in order to test if emails are sent by the extension and if the email content is as expected.

The concept of testing emails in Acceptance Tests using Codeception is really simple. You have to add the composer package ericmartel/codeception-email-mailhog to your dev dependencies and then you are ready to test emails as shown in the abstract of one of my tests below:

$I->fetchEmails();
$I->haveUnreadEmails();
$I->haveNumberOfUnreadEmails(2);
$I->openNextUnreadEmail();
$I->seeInOpenedEmailSubject('New unconfirmed registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('[email protected]');
$I->openNextUnreadEmail();
$I->seeInOpenedEmailSubject('Your registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('johndo[email protected]');

It is also possible to check the email body for various content like I do in other parts of my testsuite.

GitHub Actions supports docker based service containers and MailHog is also available as docker container, so in order to execute my Acceptance testsuite I added MailHog as service container to my CI setup as shown below:

jobs:
  build:
    runs-on: ubuntu-18.04
    services:
      mailhog:
        image: mailhog/mailhog
        ports:
          - 1025:1025
          - 8025:8025

Having the MailHog container in place, the execution of the Acceptance Tests works like a charm. 

Since the Acceptance Tests also cover tests of a plugin that is only accessible by logged in frontend users, the TYPO3 website for Acceptance Tests includes a special configured page with ext:felogin for this scenario. It turned out, that those tests failed on GitHub actions, since Argon2i was not available on the testing runner for whatever reasons. In order to resolve this problem, I configured the TYPO3 website to use BcryptPasswordHash instead of Argon2i which is ok for me, since strong password hashes are not really required in this scenario.

The GitHub actions YAML file is currently available in the development branch of my extension.

The CI results including a downloadable Codeception HTML report for all acceptance tests is available for each build as shown in this example: https://github.com/derhansen/sf_event_mgt/actions/runs/142799855