How to fix wrong sorting of inline records in a TYPO3 workspace preview

TL;DR: Records fetched with the database-query data processor for an inline (IRRE) field are always queried from the live workspace, so the orderBy restriction is evaluated on live data and a changed sorting of records in a workspace is not reflected in the workspace preview. Since the workspace overlay however replaces the sorting field value of each record, the wrong record order can be fixed with a small custom data processor that re-sorts the overlaid records.

The problem

In one of my TYPO3 projects I have a tt_content content element with an inline (IRRE) field to records of a custom table. The records are passed to Fluid as a variable using the database-query data processor with orderBy = sorting. When an editor changes the sorting of the inline records in a workspace (e.g. by drag & drop in the inline element), the workspace preview still shows the records in the sorting of the live workspace.

Analysis

The TYPO3 documentation for the database-query data processor contains the following note:

When using the DatabaseQueryProcessor, you may encounter issues with language and/or versioning overlays,
that currently can not be resolved.

This limitation is described in detail in a comment on TYPO3 forge issue #85284. The DatabaseQueryProcessor calls ContentObjectRenderer->getRecords(), which executes the query and applies the workspace and language overlays afterwards for each record.

The query itself always fetches live records, so all query restrictions like where, max and also orderBy are evaluated by the database on live data. After the query is done, PageRepository->versionOL() replaces each row with its workspace version, but the result set is neither filtered nor sorted again. This is why restrictions like where can lead to wrong results in workspaces (see the forge issue) and why my inline records are returned in the live sorting order.

There is however one important detail, which makes it possible to create a workaround for the sorting issue: since versionOL() replaces all field values of the row with the workspace version, each overlaid record contains the changed sorting field value from the workspace. Only the order of the result set is still the live one, so the records just have to be re-sorted on PHP level.

The solution

To fix the issue in the project, I created a custom data processor, which is executed after the database-query data processor and re-sorts the processed records by the sorting field, when a workspace preview is active:

<?php

namespace Derhansen\Sitepackage\DataProcessing;

use TYPO3\CMS\Core\Context\Context;
use TYPO3\CMS\Frontend\ContentObject\ContentDataProcessor;
use TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer;
use TYPO3\CMS\Frontend\ContentObject\DataProcessorInterface;

/**
 * DataProcessor to fix sorting of inline records in a workspace preview for the given variable name
 */
readonly class WorkspaceInlineRecordSortingProcessor implements DataProcessorInterface
{
    public function __construct(
        protected ContentDataProcessor $contentDataProcessor,
        protected Context $context
    ) {}

    public function process(
        ContentObjectRenderer $cObj,
        array $contentObjectConfiguration,
        array $processorConfiguration,
        array $processedData
    ): array {
        if ((int)$this->context->getPropertyFromAspect('workspace', 'id') <= 0) {
            return $processedData;
        }

        $variableName = $processorConfiguration['variableName'] ?? 'records';
        $sortedRecords = [];
        foreach ($processedData[$variableName] as $record) {
            $sorting = $record['data']['sorting'];
            $sortedRecords[$sorting] = $record;
        }

        ksort($sortedRecords);
        $processedData[$variableName] = array_values($sortedRecords);
        return $processedData;
    }
}

The data processor is registered with an identifier in Services.yaml:

Derhansen\Sitepackage\DataProcessing\WorkspaceInlineRecordSortingProcessor:
  tags:
    - name: 'data.processor'
      identifier: 'workspace-inline-record-sorting'

Finally, the data processor is added in TypoScript after the database-query data processor:

dataProcessing {
    10 = database-query
    10 {
        table = my_custom_table
        pidInList.field = uid
        where = deleted=0 AND hidden=0 AND foo = 1
        as = my-records
        orderBy = sorting
    }
    20 = workspace-inline-record-sorting
    20.variableName = my-records
}

For live pages, the data processor does nothing, since the workspace check exits early. Note that the data processor expects unique values in the sorting field of the processed records, since the sorting value is used as array key. The shown data processor only fixes the record sorting, but the same technique can also be adapted to perform possible fixes for other restrictions.

Why the sorting change is not a “move” in the workspace

While debugging the issue, I noticed that the versioned records of the inline field have t3ver_state = 0 and not t3ver_state = 4 (move pointer), although the records were “moved” by drag & drop in the inline element.

The reason is, that moving a record (e.g. a content element in the page module) is processed through DataHandler::process_cmdmap() → moveRecord(), which is versioning aware and creates a move pointer with t3ver_state = 4 in a workspace. Reordering records in an inline field does however not emit a move command at all. FormEngine simply resubmits the inline field of the parent record with the child records in the new order, and the DataHandler inline handling (checkValue_inline_processDBdata) writes the updated sorting values as a normal field update through process_datamap(). From the DataHandler perspective nothing was “moved” - a field value was simply changed, so a regular offline version with t3ver_state = 0 (or 1 for new records) is created.

This is also relevant if you build custom workspace preview or diff logic, which checks for t3ver_state = 4 to detect moved records - inline record reordering will not be catched by this check.

Learnings

The database-query data processor always queries live records - workspace and language overlays are applied after the query, so where, orderBy, max and begin are evaluated on live data.
The workspace overlay replaces all field values of a record, so the overlaid sorting value can be used to re-sort records on PHP level.
Reordering records in an inline field is a normal field update (sorting in datamap) and not a move command, so no move pointer (t3ver_state = 4) is created in workspaces.