Doctrine Core Team Meetup, ORM 2.19.8, 2.20.0, 3.3.0 Releases, DBAL 4.2.0 13 Oct 2024 8:00 PM (5 months ago)

Last week, we meet with the Doctrine ORM and MongoDB Core Teams for 3 days in Bonn, Germany. A few releases followed immediately from that, including some deprecation reversals and a discussion of our upcoming roadmap.

We want to thank all our sponsors on OpenCollective and GitHub for contributing towards making this in-person team meetup possible. If you are not a sponsor of Doctrine already, please consider becoming one.

Undeprecation of PARTIAL Object Hydration

Starting with ORM 3.3.0 you can use SELECT PARTIAL DQL syntax again and with ORM 2.20 you will not get a deprecation message for using that anymore. If your application uses partial objects with DQL, you can migrate from 2.20 to 3.3.0 now without having to change their use. Partial objects have exactly the same behavior (and downsides) in version 3 and 2.

Why the change? It was a long time plan to remove partial objects and their hydration, because of the many edge cases they produced. This plan was in effect and shipped for 3.0 - 3.2, where the syntax and feature was completely removed. In discussion with our users we found a lot of use-cases and ideas that are powerful with partial objects, but were firm on our goal to remove partial objects.

This all changes with PHP 8.4 and its lazy object feature. With this feature we will be able to implement partial objects transparently to PHP. That means it will not be necessary to know if an object is a proxy, a partial object, or a full object. Whenever a property that is not available is accessed, Doctrine can load it.

We hope to add this behavior to ORM 3.4 in the next months.

Support for PHP 8.4 and Property Hooks

Doctrine supports PHP 8.4 starting with ORM 2.20.0 and 3.3.0, DBAL 4 and 3.

There are caveats though. You cannot use property hooks in entities yet. This is because we need to rework internally how we read and write property values (ReflectionProperty::setValue() vs setRawValue() and ReflectionProperty::getValue() and getRawValue()). For now, if you try to create a property hook on an entity, Doctrine will throw an exception. We plan to address this with an upcoming ORM 3.4 release, hopefully before PHP 8.4 is released itself, otherwise shortly after.

For MongoDB ODM, PHP 8.4 support depends on a to-be-completed migration from ProxyManager to symfony/var-exporter.

Lazy Objects and PHP 8.4 requirement in ORM 4

The lazy objects RFC in PHP 8.4 changes everything for the better in Doctrine ORM internally. This is why we decided that ORM 4 will be mainly a decision about exclusively using lazy objects and therefore will require PHP 8.4.

Support for ENUM Database types

DBAL 4.2 now supports a new Enum type that is mainly useful for introspection of database schemas that contain enums. All values of the enum are parsed out of the type and are available to the schema abstraction layer.

Although we do not recommend the enum type in MySQL/MariaDB due to its quirky implementation details, you can also use this type to directly map columns to enums in the database via the ORM as of 3.3.0:

class Subscription
{
    #[Column(type: "enum", options: ['values' => ['future', 'active', 'cancelled', 'expired']))]
    public string $state = 'future';
}

You can also map Enums directly to MySQL enums and let Doctrine auto-detect all the configuration:

class Subscription
{
    #[Column(type: Types::ENUM)]
    public State $state = State::FUTURE;
}
enum State : string
{
    case FUTURE = 'future';
    case ACTIVE = 'active';
    case CANCELLED = 'cancelled';
    case EXPIRED = 'expired';
}

We have also discussed at length how we can make types parameterized, which will further improve Doctrine Schema introspection and comparisons with non-default types.

Query Cache and Pagination Variables

If you have used setFirstResult() and setMaxResults() with DQL queries then up until 2.20.x of Doctrine, each combination of first result and max results lead to their own DQL query parsing cache entry. This could easily balloon the query cache size out of control.

Starting with 2.20, the DQL parser is now running in a two-step process, where the first step generates the cacheable result and the second step amends the cached result with the LIMIT query part (or other database equivalent).

This is a bigger internal change, and we hope that we thought of all the edge cases, but it might be possible that especially in combination with the Paginator abstraction and collection fetch joins, there are some cases where this change could lead to breaks on upgrading to 2.20 from 2.19.

DQL: Nested DTOs and Named Arguments

Starting with ORM 3.3 you can now create nested DTOs with the NEW syntax and furthermore, use a short named arguments syntax to populate the constructor of a DTO. This feature was contributed by GitHub user eltharin over the last few months and builds upon previous work.

Migrations: Fixing a decade old schema comparison bug with PostgreSQL

When you used Doctrine Migrations with the ORM the down migration included a statement to drop the public schema for the better part of a decade.

This bug has finally been fixed and you won't see this drop schema statement in newly created migrations for PostgreSQL anymore.

Psalm and PHPStan going forward

After a long discussion we have decided to only use one static analysis tool and Doctrine projects will use PHPStan going forward. For now Psalm checks will be removed from repositories over the next weeks.

This is mainly because PHPStan has outpaced Psalm in depth and quality in the last few years and it feels unlikely that Psalm can point to a problem that PHPStan did not detect before. If in the future Psalm catches up to PHPStan we may reconsider adding it.

The skeletons are now out of the closet - So long, Skeleton-Mapper 15 Aug 2024 8:00 PM (7 months ago)

While Doctrine ORM and DBAL have a main focus in our daily development, a deeper look into their dependencies show, that Doctrine has much more projects at hand than just the database-related ones. Some projects even weren't created for ORM or DBAL, just like our Skeleton-Mapper.

The Skeleton-Mapper project won't be maintained anymore because of its lack of usage nowadays and is now an archived repository. The Doctrine Skeleton-Mapper was an object mapper where you are responsible for implementing the object mapping of the persistence operations. This means you write plain old PHP code for the data repositories, object repositories, object hydrators and object persisters. A lot of freedom but also a lot of work for a developer, including its maintenance for us.

Some projects grow and others become obsolete after some time, which are 9 years in the case of our Skeleton-Mapper. We also want to express our gratitude to the contributors and maintainers who kept this project alive for so long. Thank you.

Doctrine Core Team Meetup 2024 in Bonn, Germany 8 Jul 2024 8:00 PM (9 months ago)

We are organizing another Doctrine Core Team Meetup in Bonn, Germany from Tuesday, 8.10.2024 to Thursday, 10.10.2024 at the offices of one of our sponsors Tideways GmbH.

The goal is to get the current team together, discuss and work on Doctrine DBAL, ORM and ODM.

We thank all of our GitHub and OpenCollective sponsors for making this possible.

Doctrine ORM 3 and DBAL 4 Released 2 Feb 2024 8:00 PM (last year)

We are thrilled to announce the release of Doctrine ORM 3.0 and DBAL 4.0. These releases are the culmination of over a decade of hard work across dozens of contributors and the Doctrine maintainers.

What's New

A Slimmer, More Efficient ORM: The new Doctrine ORM 3.0 comes in at 326KB, down from 400KB in ORM 2.18.0. This reduction not only makes the ORM lighter but also signals our efforts to streamline and optimize every aspect of our library and focus our maintenance efforts on the core functionality of an ORM and less on tooling and helpers that are only useful by a small number of our users.

Enhanced Code Quality and Coverage: With ORM 3.0, we've pushed our code coverage from 84% to 89%. For DBAL 4.0, we've pushed our code coverage from 86% to 94%. This improvement underscores our commitment to reliability and the stability of the Doctrine ecosystem, ensuring that your applications run smoothly.

Leaner Dependencies: In Doctrine ORM 3.0, we have finally eliminated dependencies on doctrine/cache and doctrine/common. This change reduces complexity and improves maintainability of Doctrine as we now depend on PSR-6: Caching Interface for our caching responsibilities. Implementing a PSR means we are more interoperable with other frameworks and easier to use by a broader amount of users.

A Growing Community: The Doctrine project now boasts 1029 contributors across all its projects. This vibrant community is the backbone of Doctrine, providing valuable insights, feedback, and contributions that drive the project forward.

Upgrading

We understand that upgrading to a new major version can be difficult. The best way to upgrade is to first upgrade to the latest Doctrine ORM 2.x and DBAL 3.x version and address any deprecation warnings that are reported. You can read more about how Doctrine handles deprecations here. Once you have addressed all of the deprecations, you should have a clear path to upgrade.

In addition to that, we've maintained comprehensive documentation about every change, deprecation and BC break to facilitate a smooth transition to ORM 3.0 and DBAL 4.0.

The Future of Doctrine ORM 2

We plan to maintain Doctrine ORM 2 for at least the next 2 years by providing bug and security fixes. We may also add or deprecate things in 2.x to improve the existing forward-compatbility layer to make the transition to ORM 3 smoother.

Looking Forward

ORM 3 and DBAL 4 are a big step forward towards modernizing the API of our libraries, increasing safety with the use of scalar types in the code base, better error handling and generally cleaning up the code to make it easier to maintain. We look forward to continuing work on Doctrine and focusing on being the most stable and reliable PHP database persistence related libraries available.

Archiving Unmaintained Packages 29 Jan 2024 8:00 PM (last year)

After long consideration, we have decided to archive a number of repositories that have not seen any activity in a while. This affects the CouchDB and OrientDB ODMs and their respective libraries, as well as the KeyValueStore project. The following repositories and composer packages are affected:

The composer packages will remain available and installable, but we will not be making any bug fixes or security fixes in the affected libraries. If you or your business depends on one of these libraries, please fork them and maintain them yourself in the future.

ORM 3.0 Beta 1, DBAL 4 RC 1 and future plans 10 Oct 2023 8:00 PM (last year)

We have released the first beta of the long awaited Doctrine ORM 3 and a release candidate of DBAL 4.

The target audience for these releases are framework integration and extension library authors. ORM 3 is not yet production ready and the APIs may change.

Our goal is to release ORM 3.0 as soon as possible and to gather feedback from greenfield project authors first.

This beta release is the result of a lot of work by many contributors, especially Grégoire, Alexander, Claudio and Matthias on ORM, Sergei and Alexander on DBAL. To iron out the final details, we met in Düsseldorf for a Doctrine Core Team meeting, generously funded by our sponsors through OpenCollective and GitHub. We also welcomed Matthias as the latest member of the Doctrine Core Team.

Continued ORM 2 support and forward compatibility

We will maintain the latest branch of the 2 line in ORM for at least another 2 years, possibly longer, to give you enough time to upgrade and us more time to learn from upgrader feedback and improve forward compatibility.

This means that we will be making ORM 2.x work with newer versions of PHP, fixing security bugs, and introducing layers and features that help with forward compatibility in the upgrade path to ORM 3.

Current users of ORM 2 should note that there is no urgency right now to update to ORM 3, as we are still working on replacement APIs and forward compatibility, and do not intend to ship them all with ORM 3.0, but with later versions.

ORM 2 users can already prepare for 3 by addressing deprecations

But there is already work to be done as an ORM 2 user: to help you find all the places where things may be deprecated or changing behaviour, we have created the doctrine/deprecations library and integrated it heavily into DBAL, ORM and other components.

It allows the use of deprecated behaviour to be logged at runtime with low overhead, automatic suppression of the same deprecation occurring multiple times, and a way to ignore selected deprecations for the time being. Each deprecation message always links to a GitHub issue with more details.

Many deprecated features have no replacement, such as Mapping Exporters, Generate Mapping from Database, Named Queries.

For some of the deprecations in ORM, we are still planning replacement APIs, especially:

There is currently no way to limit the number of entities that the flush operation considers changed. Flush will currently always calculate change sets on all entities that are not read-only.
As a replacement for removing PARTIAL object hydration, we are looking at making embeddable objects lazy, perhaps improving nesting of the new DTO expression in DQL. We are also looking to introduce subselect or batch loading for collections for more efficient multi-level hydration.

These will be released in 2.x as forward compatible APIs so that you can switch to using them before upgrading to ORM 3.

Doctrine ORM Team Meetup in Bonn, Germany 20 Aug 2023 8:00 PM (last year)

We are organizing a Doctrine ORM Core Team Meetup in Düsseldorf, Germany from Monday, 9.10.2023 to Wednesday, 11.10.2023 at the offices of one of our primary sponsors Tideways GmbH.

The goal is to get the current team together, discuss and work on the missing pieces for the long-awaited Doctrine ORM 3.0 release that is planned for later this year.

From annotations to attributes 3 Nov 2022 8:00 PM (2 years ago)

Last month, we migrated the tests of the ORM from annotations to attributes. Let us look back on what lead to this moment.

Annotations

Let's go 22 years back in time. In October 2000, Ulf Wendel introduces phpdoc comments at the PHP-Kongress. These comments follow a structure that allows to produce API documentation from them. They are inspired by javadoc.

In 2002, Alex Buckley, a Specification lead for the Java language publishes JSR-175, thus proposing to add user-defined annotations to the language, allowing to tag language elements with extra information. 2 years later, it gets approved and Java 1.5, also known as Java 5 is released, with support for annotations.

4 more years elapse and in 2006, Jano Suchal publishes Addendum, a PHP library that adds support for using "Docblock/JavaDoc" as annotations, meaning that contrary to what is done in Java, Addendum annotations are contained inside phpdoc comments, like this:

/** @test */
function test_it_throws_on_invalid_argument(): void
{}

That is because they are implemented in userland, without requiring a change in PHP itself.

Doctrine ORM 2.0 is not released yet at that point, but the library is used to build an annotation driver in Doctrine 2 in early 2009. At that time, Doctrine was a project in a single repository, with Common, DBAL and ORM as top-level namespaces. Addendum is replaced 6 months later, with a new namespace under Common called Annotations.

In the summer of 2010, Guilherme Blanco and Pierrick Charron submit an RFC to add annotations support to PHP, but it gets declined. The RFC already mentions the need for annotations in PHPUnit, Symfony, Zend Framework, FLOW3 and of course, Doctrine.

Late 2010, Doctrine 2 is tagged, and the single repository is split into 3 repositories.

Finally, in 2013, the namespace above is isolated in its own repository, and doctrine/annotations 1.0.0 is tagged.

Today, the package is widely used in the PHP ecosystem and has a little short of 300 million downloads on Packagist, and is depended on by over 2 thousand packages, including major frameworks and tools. It is fair to say annotations have proven valuable to many users.

Attributes

The RFC mentioned above is only one among many. As mentioned before, annotations were implemented as phpdoc comments, which has several drawbacks:

The comments are necessary to run the code, and need to be kept in the opcode cache.
They are obtained at runtime, by using the reflection API, and because of that, can only be detected as invalid at runtime.
They are not well supported by IDEs if at all.
They clutter comments, which were originally intended for humans.
They can be confused with phpdoc, which are something else entirely.

In March 2020, Benjamin Eberlei resurrects Dmitry Stogov's attributes RFC and submits the seventh RFC on this topic, introducing attributes to PHP.

A few rounds of RFCs about syntax later, PHP 8.0 is released, with a notable feature missing: nested attributes. PHP 8.0 attributes use a syntax that is forward-compatible with them though, and finally, with PHP 8.1, nested attributes are supported.

Migrating from one to the other

Since attributes are much better than annotations, with doctrine/orm 3.0, annotations will no longer be supported, which means applications using them as a way to map entities to tables need to migrate towards attributes (or another driver). As maintainers of that library, even we needed to migrate: most of the test suite of doctrine/orm used annotations.

Enter Rector. Rector is a standalone tool that is invaluable when it comes to performing such migrations: it is able to understand PHP code and apply so-called Rectors to it. It is extensible, so it is possible to define such Rectors in order to address upgrades for anything, including Doctrine.

What's more, it comes with batteries included: when you install rector/rector, what you get is code from rector/rector-src and its official extensions, among which you will find rector/rector-doctrine. That's right, there is already an entire extension dedicated to Doctrine.

Rules are grouped together in sets, and the set that interests us here is Rector\Doctrine\Set\DoctrineSetList::ANNOTATIONS_TO_ATTRIBUTES.

To migrate doctrine/orm's test suite to annotations, here is how we proceeded:

Install Rector: composer require --dev rector/rector.

Create a file called rector.php at the root of the library with the following contents:

    <?php

    declare(strict_types=1);

    use Rector\Config\RectorConfig;
    use Rector\Doctrine\Set\DoctrineSetList;

    return function (RectorConfig $rectorConfig): void {
        $rectorConfig->paths([
            __DIR__ . '/tests',
        ]);
        $rectorConfig->sets([
            DoctrineSetList::ANNOTATIONS_TO_ATTRIBUTES,
        ]);
    };

Run vendor/bin/rector, which obeys the above configuration.
Uninstall Rector: composer remove rector/rector && rm rector.php
Run vendor/bin/phpcbf to make the migrated codebase compliant with our coding standard.

Or at least, it was the plan, because some annotations were not perfectly migrated. All in all, I found only 2 bugs, which looks great given how so many edge cases should appear in our test suite.

I went on and reported those 2 bugs, and this is where the experience went from great to stellar: the issue template leads to a playground, much like the one you can find for other tools such as Psalm or PHPStan.

This one comes with 2 buttons: "Create an issue", which pre-fills the Github issue with useful information, and "Create a test", that lets you create a test in the right directory (and also, the right repository, which is rectorphp/rector-src, and not rectorphp/rector).

If you want to add a new test for the bug you reported, you should let the official tutorial walk you through that, it is very well written.

Anyway, now that these 2 bugs are fixed and you know how to migrate, plan that migration, and let us know how it goes! Happy Rectoring!

New Release: Doctrine DBAL 3.4.0 5 Aug 2022 8:00 PM (2 years ago)

Doctrine is proud to announce the release of Doctrine DBAL 3.4.0. Below is a summary of the most noteworthy changes in the new release:

Database schema introspection optimization (#5268)

Older DBAL versions, in order to introspect database schema, performed a set of queries for each table individually. This caused noticeable performance issues on some platforms like Oracle which seemingly rebuild their internal views for each such query.

As of this release, the entire schema is introspected in a fixed number of queries. The more tables the schema contains, the more noticeable this optimization should be.

It was impossible to make these optimizations while using the schema introspection platform methods (e.g. getListTableColumnsSQL()). As a result, although these methods are kept in the codebase for backward compatibility, the DBAL itself no longer uses them. The SQL queries used for schema introspection are no longer considered part of the public DBAL API.

Support for foreign key constraints on SQLite (#5427)

Although SQLite has supported foreign key constraints since its earliest versions, their support in the DBAL was quite limited. One of the reasons for that was that managing foreign key constraints in SQLite is quite different from the rest of the supported platforms.

For example, when a foreign key constraint is declared, platforms like MySQL require that the referenced table must already exist. To support creating tables with mutually referencing constraints, the DBAL would create the tables first and create the constraints via ALTER TABLE … ADD FOREIGN KEY ….

This approach doesn't work with SQLite since it doesn't allow adding constraints to an existing table. Fortunately, it doesn't require the referenced table to exist at the time of creating the foreign key either.

The new DBAL release introduces a new API for building CREATE TABLE and DROP TABLE statements for multiple tables which could be tailored to the requirements of a given platform. The AbstractPlatform::supportsForeignKeys() method is now deprecated since the DBAL supports foreign key constraints on all supported platforms.

Support for TEXT/BLOB default values on MariaDB (#5332)

The platform layer in the DBAL is organized in the way that the code implementing the support for MySQL is also used to support MariaDB. As a result, even though MariaDB may support certain features the DBAL doesn't support them because they are not supported by MySQL. One of such features is the default values for TEXT and BLOB columns.

As of the new release, the default TEXT and BLOB values are supported on MariaDB but are still unsupported on MySQL, even though MySQL supports them as of release 8.0.13.

Support for result caching in QueryBuilder (#5539)

The recently added enableResultCache() method of the QueryBuilder class allows specifying the query cache profile to be used for performing the queries built by the builder.

PHP 7.4 or newer is required (#5459)

The DBAL no longer supports PHP 7.3 since its support by the community ended last year. The codebase now actively uses such features of PHP 7.4 as covariant return types and typed properties.

Deprecations

In light of the DBAL 4 release planned for later this year, the 3.4.0 release introduces over 30 deprecations which, as usual, focus on cleaning up obsolete features and making the API more robust and clearer from the static analysis standpoint.

To learn more about upgrading your application, see the upgrade notes. You can find the full list of changes in the release milestone.

On Doctrine Sponsoring with the help of OpenCollective 23 Mar 2022 8:00 PM (3 years ago)

To simplify our own administrative burden, we have decided to move over the funds to OpenCollective, primarily motivated by the success and positive experience of the new PHP Foundation.

We have started raising money for the Doctrine Project in 2019 through Patreon and Github Sponsors to support the project. It was planned to organize core team get-togethers/hackathons, but due to the pandemic we haven't been able to do this. In addition the legal and tax implications of raising money also caused us some headaches.

The move to OpenCollective will allow us to delegate much of the administrative work to them for a small percentage of the raised capital. The fee is a much smaller amount than the taxes that we had to pay on the raised money previously, so it is a win-win for us.

We want to assure our sponsors that we still plan to make use of the funds to further the Doctrine project and we are actively looking to increase our funding for these goals:

Regularly organize hackathons for Doctrine Core team contributors, including 3-4 days of accommodation, food and travel for roughly 10-15 people.
Infrastructure, servers and hosting.
Marketing material such as stickers, t-shirts and other small things.
If the budget increases significantly we might be able to pay someone part- or full-time to do maintenance work such as responding to and processing issues, prepare for new PHP releases and general cleanups of the codebase.

As such we hope to convince you to sponsor Doctrine through either our Github Sponsors or OpenCollective directly.

Doctrine View RSS