When iterating on features can break a Drupal site in development

It's generally safe to use Features to maintain and deploy the configuration of Drupal sites, but I'd heard rumours from other developers that, especially during a rapid development process, some phase of featurization could lead to database errors. I'd not seen that for ages until a few days ago, when re-enabling features led to PDO exceptions. Here's what I did, what I didn't do, and how to mitigate this problem in future.

The process I followed: getting a new live database mid-featurizing

I had been working on a feature locally, part of which included adding new fields to a content type. All the additions had been exported into the feature and checked into version control. The features dashboard was showing green "Default" lights all the way.

At that point, I needed to see how my feature would look when it was finally moved to production after approval. To do that—I don't have shell access to the server—I downloaded a database using Backup/Migrate and unzipped it into MySQL at the command prompt. So far, so good; at which point I enabled my new features, saw they were listed as "Overridden" and reverted them.

This was when all hell broke loose:

$ drush feature-revert-all
The following modules will be reverted: foo_something, foo_another, foo_yetanother
Do you really want to continue? (y/n): y
Do you really want to revert context? (y/n): y
Reverted context.                                                                  [ok]
Do you really want to revert field? (y/n): y
Reverted field.                                                          [ok]
Do you really want to revert views_view? (y/n): y
Reverted views_view.                                                     [ok]
Do you really want to revert field? (y/n): y
exception 'DatabaseSchemaObjectExistsException' with message 'Table      [error]
field_data_field_test already exists.' in
/var/www/drupal7-foo/web/includes/database/schema.inc:657
Stack trace:
#0 /var/www/drupal7-foo/web/includes/database/database.inc(2720):
DatabaseSchema->createTable('field_data_fiel...', Array)
...

(Output edited for confidentiality.) So what exactly had happened here? More importantly, how was I to prevent it happening on production?

What broke feature reversion and how to prevent it

Here's the order in which important events occurred on my local build.

1. Drupal began with a database similar to production.
2. At least one field was created, called field_test in the above example.
3. This field, among other things, was featurized into code.
4. A new database dump was brought down from production, and unzipped directly into MySQL's command prompt (via drush sql-cli.)
5. All features were reverted, during which a PDO exception was observed owing to a table already, unexpectedly existing.

The key here is step 4, where I unzipped the database dump straight into MySQL. Doing so does successfully drop and recreate tables that production knew about at the point of creating the zipfile. However, it doesn't touch—for better or worse—tables which don't exist on production. They're left lying around, untracked by Drupal.

The upshot is that I ended up with a database where fields didn't exist in Drupal's field configuration, but their corresponding tables did, left behind by step 2. When the feature was enabled, the field configuration was changed to add the new field, and only then was an attempt made to create the table... which already existed, hence the PDO exception.

This is straightforwardly—if annoyingly—solved for local development, by ensuring the database is DROPped and reCREATEed before piping the new zipfile in to fill it up. This drops all tables and leaves a clean slate for production to be faithfully replicated in.

Will PDO exceptions happen on production?

What about production, which is far more crucial than a development environment? Well, field creation through features has generally been safe for me on production. However, it's at least theoretically possible that this could cause problems.

The only situation I can think of in which PDO exceptions might happen on production are when tables already exist that might replicate the above problem. In theory, once you've deleted a field instance, its per-entity data should also be deleted; in practice, this might not happen until cron, and even then until several cron runs, once all the field data is purged. It's not clear to me, but it's something to be aware of if anyone has created fields on production in the past. You can check your production database matches what's expected of it by Drupal using the schema module, although it's not clear to me how that works with deleted yet unpurged fields.

Ultimately, if you really don't feel you can trust your production environment to feature-revert safely, you just shouldn't do it (especially when features contain field creations.) Instead, use the diff module to see what parts of the feature are in need of reversion, and then create the fields manually and accordingly. Once you're left with a few fragments of overridden configuration—maybe field weights, or boolean flags you've forgotten to set—it's probably OK to revert by that point. Probably.

Oh, and always back up production before making changes to it, obviously. That goes without saying. Right?