Friday, June 26, 2020

Testing email delivery of a TYPO3 extension with Codeception, MailHog and GitHub Actions

Some weeks ago I published my blogpost about how to create a GitHub Actions build pipeline for a TYPO3 Extension that executes Unit-, Functional- and Acceptance tests. The extension tested in that blogpost was only a simple demo extension and for me this was a preparation to finally migrate the CI pipeline for my TYPO3 extension sf_event_mgt to GitHub Actions.

The extension comes with lots of unit and functional tests, which are automatically executed on each commit. One missing piece in the puzzle was the automatic execution of my Acceptance Tests, which are based on Codeception and additionally require MailHog in order to test if emails are sent by the extension and if the email content is as expected.

The concept of testing emails in Acceptance Tests using Codeception is really simple. You have to add the composer package ericmartel/codeception-email-mailhog to your dev dependencies and then you are ready to test emails as shown in the abstract of one of my tests below:

$I->seeInOpenedEmailSubject('New unconfirmed registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('[email protected]');
$I->seeInOpenedEmailSubject('Your registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('[email protected]');

It is also possible to check the email body for various content like I do in other parts of my testsuite.

GitHub Actions supports docker based service containers and MailHog is also available as docker container, so in order to execute my Acceptance testsuite I added MailHog as service container to my CI setup as shown below:

    runs-on: ubuntu-18.04
        image: mailhog/mailhog
          - 1025:1025
          - 8025:8025

Having the MailHog container in place, the execution of the Acceptance Tests works like a charm. 

Since the Acceptance Tests also cover tests of a plugin that is only accessible by logged in frontend users, the TYPO3 website for Acceptance Tests includes a special configured page with ext:felogin for this scenario. It turned out, that those tests failed on GitHub actions, since Argon2i was not available on the testing runner for whatever reasons. In order to resolve this problem, I configured the TYPO3 website to use BcryptPasswordHash instead of Argon2i which is ok for me, since strong password hashes are not really required in this scenario.

The GitHub actions YAML file is currently available in the development branch of my extension.

The CI results including a downloadable Codeception HTML report for all acceptance tests is available for each build as shown in this example:

Tuesday, May 5, 2020

Unit-, Functional- and Acceptance-Tests for a TYPO3 Extension with GitHub Actions

Back in 2017 at TYPO3 Camp Munich I held a talk about Unit-, Functional- and Acceptance-Tests for a TYPO3 Extension with GitLab CI. I never really used that setup for my Open Source Extensions, since they all are hosted on GitHub. But since november 2019 GitHub Actions are available, so I finally took some time to migrate my GitLab CI Build Pipeline to GitHub Actions. The results of this migration process is available on GitHub and summarized in this blogpost.

To keep things simple, I created a little demo Extension for TYPO3 to make the setup as easy and understandable as possible.

All in all, the results are very satisfying and the build process is really fast without the requirement to use additional docker images (e.g. MySQL or Selenium Standalone). GitHub has really done a great job by providing preconfigured hosted runners with lots of useful tools ­čĹŹ

The GitHub Repository with all sources and the GitHub Actions workflow is available at

During creation of the setup, I ran into some issues, that took me some time to figure out. All issues are easy to resolve and I summarized them in the "Good to know"-section at the end of this article.

TYPO3 demo extension "gha_demo"

The repository includes a very simple TYPO3 extension that basically does nothing special. It has a simple domain model with just one field and a simple plugin that shows all records of the domain model. The extension has the following tests

  • Unit Tests for the domain model
  • Functional Tests for the repository
  • Acceptance Tests (based on codeception) for the plugin output

Before I created the GitHub Actions workflow, I ensured that all tests execute properly in my local development environment.

GitHub-hosted virtual environments

GitHub hosted runners are preconfigured runners that already contain a lot of available software (e.g. composer, PHP in various versions, Google Chrome, Selenium) that can be used to test an application. No need to puzzle around with building or pulling docker images that contain requirements and no waste of build time to install required packages.

For the gha_demo TYPO3 extension I use the Ubuntu 18.04 LTS runner only without any other docker images.

Workflow YAML file

It is very easy to enable GitHub Actions for a repository. You create a directory called .github/workflows and add a YAML file with your workflow configuration that must follow the Workflow Syntax for GitHub Actions.

The workflow YAML file I created for this article is (hopefully) self explaining:

The workflow uses 3 GitHub actions. The first action "actions/checkout" just checks out the GitHub repository.

The second action "actions/cache" ensures, that Composer cache files are shared for builds. You just have to configure a unique key for the cache and I choose to use the hash of the composer.json as a key, so every time dependencies change, the cache is rebuilt. To ensure, that the cache is working you should see "loading from cache" in the output of the composer command.

What helps when you want to debug your workflow is to save build artifacts. For this I use the third action "actions/upload-artifact" which uploads the log of the PHP server and the Codeception output if the build failed.

All other steps in the workflow are based on commands that are executed on the runner (e.g. start MySQL Server, Update Composer, ...).

You may note, that the workflow contains 2 "sleep" commands. Both are required so previous commands have enough time to finish execution (start PHP Server and start Selenium).

Another thing you may note is, that I added many TYPO3 packages to the require-dev section of my composer.json file. This is not a requirement and can be moved to an additional build step (e.g. composer require typo3/cms-filelist ....).

Acceptance Tests with Codeception

In order to execute the Codeception Acceptance Tests, it is required to setup a fully working TYPO3 website including a preconfigured database dump with e.g. pages and records to test. For the Acceptance Tests I included the following files/folders in Tests/Acceptance/_data

  • config/ 
    TYPO3 Sites configuration
  • typo3conf/LocalConfiguration.php
    Preconfigured LocalConfiguration PHP that matches the environment and settings (e.g. DB Credentials) for GitHub Actions
  • typo3.sql
    Full Database dump of my local test TYPO3 website

To separate between Acceptance Test environments (local and GitHub) there are configuration settings for both in Tests/Acceptance/_env

At this point I would like to thank Daniel Siepmann for sharing his GitLab CI configuration about Acceptance Tests. I adapted some parts of his examples to my current setup.

Update 29.05.2020: If you want to test an extension against multiple TYPO3 versions, you can use a build matrix as shown in this example from Sebastian Fischer.

Good to know

#1 - Composer dependencies are not cached during builds

Update 15.05.2020: Not required any more, since the issue is fixed.

Due to a misconfiguration in the Ubuntu 18.04 runner (that has already been fixed), the .composer directory is owned by root:root with 775 rights. This makes it impossible for the runner user to write into that directory. To fix this, make sure to remove the the directory recursive as shown below in a build step before composer is executed.

- name: Delete .composer directory
  run: |
    sudo rm -rf ~/.composer

#2 - PHP server with "php -S" is obviously not starting

I used "php -S -t .Build/public/ &> php.log.txt &" to start a PHP server that serves my application for Acceptance Tests. Somehow the acceptance tests step was not able to connect to the given port and always showed "Failed to connect to x.x.x.x port 8888: Connection refused”

To solve this issue, I forced the workflow to stop for 2 seconds (just added "sleep 2;" right after the PHP -S line) so PHP has enough time to server the application.

#3 - MySQL credentials not accepted / MySQL "Connection refused"

Setting up a build step that uses MySQL I ran into problems connecting to the MySQL server that comes with the default Ubuntu 18.04 runner. The solution to this problem was really simple, since you just have to start the MySQL service.

- name: Start MySQL
  run: sudo /etc/init.d/mysql start

The default credentials for the MySQL are root:root

Wednesday, April 29, 2020

How to add a replacement for the removed TCA Option "setToDefaultOnCopy" in TYPO3 10.4

The TYPO3 TCA Option "setToDefaultOnCopy" has been removed in TYPO3 10 in order to reduce the amount of checks in DataHandler and the amount of available options in TCA. The documentation says, that "This option was only there for resetting some `sys_action` values to default, which can easily be achieved by a hook if needed. If an extension author uses this setting,
this should be achieved with proper DataHandler hooks."

I use this option in one of my extensions. Basically, I have one "main" record, that has one Inline field with various "subrecords". Those "subrecords" are user generated and should not be copied, when the main record is copied, so I had to find out which DataHandler hooks should be used to get the removed functionality back for the TYPO3 10 compatible version of my extension.

After some hours with several breakpoints in the TYPO3 DataHandler I came to the conclusion, that this may not be as "easy" as described, since there is no Hook, where you can unset certain field values during the copy (or localize) process. And if there was, then another problem would have shown up, since relation fields are processed different (basically the relation is resolved using TCA) on copy or translation commands in DataHandler.

Knowing the last about TCA however makes it possible to hook into the process. At a very early stage in DataHandler, I use processCmdmap_preProcess to set the TCA type for affected fields to "none" as shown below:

public function processCmdmap_preProcess($command, $table, $id, $value, $pObj, $pasteUpdate)
    if (in_array($command, ['copy', 'localize']) && $table === 'tx_extension_table') {
        $GLOBALS['TCA']['tx_extension_table']['columns']['fieldname1']['config']['type'] = 'none';
        $GLOBALS['TCA']['tx_extension_table']['columns']['fieldname2']['config']['type'] = 'none';

With this configuration in TCA, the affected fields are completely ignored by the copy/localize command in DataHandler. It is now just important to change the field types back after the command is finished in processCmdmap_postProcess hook as shown below:

public function processCmdmap_postProcess($command, $table, $id, $value, $pObj, $pasteUpdate, $pasteDatamap)
    if (in_array($command, ['copy', 'localize']) && $table === 'tx_extension_table') {
        $GLOBALS['TCA']['tx_extension_table']['columns']['fieldname1']['config']['type'] = 'text';
        $GLOBALS['TCA']['tx_extension_table']['columns']['fieldname2']['config']['type'] = 'inline';

Hard to say, if this is a good approach to get the functionality back. It feels not really right to change existing TCA at runtime as shown, but at least, I could not find any downsides in the solution and it works fine for me.

Thursday, April 2, 2020

How to limit the TYPO3 category tree to a subset of categories for extension records

In many TYPO3 projects I've been working in, the TYPO3 category system is used to structure content by one or multiple categories. A typical category tree I often see is build up as shown in the example below:

Full TYPO3 category tree

This is a very plain way to create a category tree and the structure in the example is limited to 3 independent main categories (Events, Products, Staff). 

Quite often, the shown example category tree is used system wide in TYPO3 and all main categories are shown for all record types. This can be confusing for editors, since when you for example want to assign categories for e.g. event records, why should one see and be able to select the categories "Products" and "Staff" including all subcategories?

Fortunately TYPO3 can be configured to limit the category tree for tables to a given root category. As an example, I limit the shown categories for event records to only "Event" categories. I assume that the category "Events" has the sys_category UID 1.

PageTS example

TCEFORM.tx_sfeventmgt_domain_model_event.category.config.treeConfig.rootUid = 1

In PageTS such configuration options can be set for any record as long as the following configuration path is met: TCEFORM.[tableName].[fieldName].[propertyName]

The fieldName is usually "categories" or "category", but this can also be different depending on how categories are implemented in 3rd party extensions.

The PageTS setting can also be set in TCA as shown below.

TCA example

$GLOBALS['TCA']['tx_sfeventmgt_domain_model_event']['columns']['category']['config']['treeConfig']['rootUid'] = 1;

As a result, the category tree for event records is now limited to the category "Events" and all subcategories.

TYPO3 category tree limited to a subcategories of one main category

I think this is way more clear for an editor than it was before. In general, this can be configured for every table in the TYPO3 TCA (e.g. pages, files, extensions, ...)

The configuration only allows to define one UID as root for the category tree. If more flexibility is needed to limiting the category tree, then TCEFORM.[tableName].[fieldName].[config][foreign_table_where] may be the place to add own custom conditions.

Wednesday, February 19, 2020

Apache rewrite rule to replace %20-%20 with a dash (#) in URLs

Some old(?) versions of Microsoft Excel replace a dash (#) in an URL with "%20-%20". The following example shows, how Excel transforms URLs:

Original URL:

URL when clicked in Excel:

This may lead to unexpected behavior on webserver application level e.g. when routing can not be resolved successfully and the request will results in an 404 error.

The probably best way would be to fix this behavior "somehow" in Excel, but this does not always seem to be possible as described in this stackoverflow question.

In order work around this problem for a certain application on a webserver, I added a simple redirect which replaces the "%20-%20" with a "#" using the following .htaccess rewrite rule:

RewriteRule ^(.*)\ \-\ (.*)$ /$1#$2 [NE,L,R=301]

This is for sure not a general solution for the problem, put works perfectly when you only have to fix incoming links for a given application.

Friday, November 22, 2019

TYPO3 Extension "Plain FAQ" released

Today I released the first public version of my latest TYPO3 Extension called "Plain FAQ". The name already says it - the extension is a very simple solution to manage Frequently Asked Questions in TYPO3. Below follow some of the facts about the Extension:
  • Compatible with TYPO3 8.7 and 9.5
  • Based on Extbase and Fluid
  • Covered with unit and functional tests
  • Easy usage for editors
  • Uses TYPO3 system categories to structure FAQs by category
  • Field for media and files
  • Possibility to add related FAQs
  • Configurable template layouts for the views
  • Automatic cache cleanup when a FAQ article has been updated in backend
  • Signal slots to extend the extension with own functionality
  • Symfony Console commands to migrate from ext:irfaq
The extension is available on TER and packagist.

Migration from "Modern FAQ (irfaq)"

If you currently use the TYPO3 Extension "Modern FAQ (irfaq)" you may have noticed, that the extension is not compatible to TYPO3 9.5 (last Extension-Update in March 2018)  and the architecture is quite old (AbstractPlugin, Marker Based Templates and TypoScript configuration for wraps). 

For users where "Modern FAQ (irfaq)" is a blocker for an upcoming TYPO3 9.5 Update, it is possible to migrate to "Plain FAQ" using the Symfony Console Commands included in "Plain FAQ". The migration is as easy as the usage of the extension:

1. Migrate existing Categories to sys_category
2. Migrate existing FAQs to "plain_faq" records
3. Migrate existing Plugins including Plugin settings

The migration may not cover all possible scenarios (e.g. Ratings, Question asked by, irfaq Plugin settings set by TypoScript), but is for sure a good starting point in order to migrate existing records. I guess, for most websites the included migration will suite without further work on migrated data.

You can find details about the migration process in the Extension Manual.

Thanks for sponsoring

I would like to thank Julius-Maximilians-Universit├Ąt W├╝rzburg for sponsoring the initial development of the TYPO3 extension. Thanks for supporting TYPO3 and open source software!

Monday, September 2, 2019

How to fix TYPO3 error "There is no column with name 't3ver_oid' on table"

Recently the following error message showed up in a project I was updating to TYPO3 9.5:

There is no column with name 't3ver_oid' on table 'tx_news_domain_model_news'.

When you see this message in a TYPO3 project, you should of course first check, if the field is really available in the mentioned table and second, you should check, if extension dependencies are correct.

When extending an Extbase extension (in this case ext:news), you must ensure, that the extending extension is loaded after the extension you extend. In order to do so, you must add a dependency to the extension you extend in your ext_emconf.php like shown below (example when extending ext:news):

'constraints' => [
    'depends' => [
        'news' => '7.3.0-7.3.99'

After adding the dependency, make sure to regenerate PackageStates.php ("Dump Autoload Information" in install tool or "composer dumpautoload"  for composer projects)