I’m still working on this. Apologies for yet incomplete bits.
I used Jeff Geerling’s Drupal-VM to create both a local development Drupal server and a live production server very much along the lines of his tutorial which he wrote for Drupal MidCamp in March 2017. So you might like to go there if I have omitted a detail you want to follow.
We use Drupal-VM to keep our Development, Staging, and Live sites’ environments exactly in sync.
The operating system is at exactly the same level
The required software components are exactly the same
The configuration of both operating system and software components are the same.
The following describes our use of Drupal-VM using the
branch of our GitHub repo iainhouston/bradford-abbas.uk
The major section Important Drupal-VM configuration files below gives further information about our oparticular use of Drupal-VM so that you can see how I’ve used Drupal-VM to build and maintain our website here at bradford-abbas.uk.
Applicability to other UK Parish Councils
Another Parish Council could fork our GitHub repo and follow the steps below to buid their own website and manage their own Meetings, Agendas, Minutes and other Documents; distribute News Articles; summons Councillors to attend Mettings and so on.
There is very little, like website names, that are peculiar to Bradford Abbas Parish Council. Everything else that follows could be used to build a Parish Council website that conforms to the Transparency Code for Smaller Authorities which came into law in 2014.
Drupal-VM uses only free, Open Source software. The inescapable costs of our website are:
the small monthly cost of the Amazon EC2 web server;
the annual renewal of our domain name;
the annual renewal of our SSL Certificate;
the annual renewal of our data protection (ICO) certificate which we’d need even if we didn’t have a website.
Typical use cases
There are three typical use cases that use steps described in one or more of the major sections below.
Most typically, deploying updates to the live server
Deploying updates to the live server after having applied a security update to a Drupal core or contributed module, or an update we’ve made to our Drupal theme:
Set up the local development environment. We only need to do this once on a Mac where we haven’t been doing our website’s development work before.
Do some development and testing.
Deploy the updated and tested system to the live server.
Less typically, switch live servers.
This is one that I have done recently to upgrade the live server to the a more recent Linux distro and a later PHP verdsion so as to accomodate a more recent Drupal version.
Set up a new live prodction server
Clone the live server to the local development server
Do some development
Deploy the updated and tested system to the new live server.
Theme development: css and html changes
As part of either of the “Do some development” steps above: use the theme’s development toolchain to create a new git-tagged version of
Set up the local development environment
Clone this GitHub repo to
Add the following alias to your
alias cdbadev="cd ~/bradford-abbas.uk && source ./scripts/badev/dev_aliases.sh"
$ source ~/.zshrc(or
updateLiveCode - Code and Config to Live site cloneLive2Dev - Clone Live Database and Files to Dev site safecex - Safe export of Dev site's configuration endev - Enable development modules in Dev site
We have written several convenience shell commands to do much of the day-to-day heavy lifting for us. As above, we are reminded of four of them when we switch to our
cdbadevalso assigns several key exports and aliases.
Install the libraries and a working Drupal
composer installwill download all the libraries. These include the Symfony PHP libraries, upon which Drupal is built; Drupal’s core and contributed modules; other non-PHP libraries like our own Ansible rôles and tasks in
iainhouston/drupal-vm, and our own Drupal theme
composer installalso places the appropriate Drupal contributed and core modules in the appropriate directories in the
webdirectory which it creates for us (please ensure you don’t start out with
composer installwill not create the Drupal site.)
You can reassure yourself that the installation has been successful and will shortly, after we have fired up the development VM, respond to
httprequests by the presence of
~/bradford-abbas/web/autoload.phpand of the presence of
web/autoload.phpso that running Drupal modules have access to the libraries in
vendor/composer/autoload_real.phpwhich can then load any of the PHP classes etc. it discovers in the
webmaster’s access to the Live Server
Satisfy yourself that the
Hostsettings in your
~/.ssh/configsettings match the host names in
scripts/badev/dev_aliases.shand that you have the correct key pair
IdentityFileprovided when the AWS EC2 Server was set up. Do this by
Host wpbapc ForwardX11 no User webmaster Hostname bradford-abbas.uk PreferredAuthentications publickey IdentityFile ~/.ssh/BAPC-2.pem
export LIVE_SSH_ALIAS="wpbapc"and several scripts in
This access is needed in the next step.
Fire up the development VM
vagrant upwill also provision the VM if it does not previously exist, otherwise Vagrant will just boot it up and manage the IP Addresses in
/etc/hostsand the NFS shared directories in
vagrant sshand accept the authenticity of the newly-created host / vagrant server. (If you don’t do this, the next step will fail.)
exitthe vagrant server to return to your Mac.
Ensure that the drush aliases for the live and development servers in
Sync the Live Drupal to the Development Drupal
`cloneLive2Dev` clones the live database and the static files (uploaded images, PDFs etc.) to the development Drupal in the VM. Drush will not copy/clone between to remote servers. We (the host controller Mac) are dumping the live SQL into a newly-named file in `vm/saved_sql/live` for later use if required, and importing it into `@badev`, the development Drupal. For this reason, too, `cloneLive2Dev` `rsync`s the static files from `wpbapc`'s access in the live server to the controlling host's `/web/sites/default/files/` rather than using `drush core:rsync`. NFS makes these files immediately available to the development server's VM.
- Make changes to the site’s Theme
We have just (June 2020) changed from our own bespoke theme to a si=ubtheme of `drupal/olivero` which is `web/themes/contrib/pellucid_olivero` We are aiming not to use any styling toolchain (gulp, postcss etc.) to keep things really simple with just straight-ahead CSS.
Deploy the updated and tested system to the new live server.
The static data files (
sites/default/files/) and the database on the server are not affected by the first step.
Pushing updated stuff to the live site
updateLiveCode is shorthand for:
DRUPALVM_ENV=prod \ ansible-playbook vendor/iainhouston/drupal-vm/provisioning/playbook.yml \ --inventory-file=vm/inventory \ --tags=drupal \ --extra-vars="config_dir=$(pwd)/vm" \ --skip-tags=test_only \ --become --ask-become-pass --ask-vault-pass
This doesn’t re-provision the live server, it does just those playbook tasks - those with
tags: ['drupal'] - required to update the following:
- Drupal core and contributed modules (via
- Our SSL key and certificate
- Drupal configuration YAML from most recent local
drush @badev cex(and thus
safecex). But note that it doesn't run a
drush @balive cim
So you run
updateLiveCode to deploy to the live server after any of these have been updated and tested on the local development site.
Provisioning the development site
I do my development on a Mac but Jeff describes here how its done on a Windows 10 machine.
Our local environment (at the time of writing is shown by using our helper alias
PHP 7.4.0 (cli) (built: Nov 29 2019 16:18:44) ( NTS ) Copyright (c) The PHP Group Zend Engine v3.4.0, Copyright (c) Zend Technologies with Zend OPcache v7.4.0, Copyright (c), by Zend Technologies Composer version 1.9.1 2019-11-01 17:20:17 Vagrant 2.2.6 VirtualBox 6.0.14r133895 ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19] ansible 2.9.1 ... NodeJS Version v13.2.0 npm Version 6.13.1 vagrant-auto_network (1.0.3, global) - Version Constraint: > 0 vagrant-hostsupdater (18.104.22.168, global) - Version Constraint: > 0 vagrant-vbguest (0.21.0, global) - Version Constraint: 0.21 Developer Edition of Mozilla Firefox 72.0b
Errors ahead?: My experience over several years of using Drupal-VM shows that unexplained provisioning errors can often disappear after you are sure you have upgraded to the latest of each of
Key environment variable
If you think you’re provisioning a live server rather than a development one, or vice versa, ensure that the
DRUPALVM_ENVenvironment variable is correctly set by issuing vagrant commands in the form:
DRUPALVM_ENV=vagrant vagrant upand
DRUPALVM_ENV=vagrant vagrant provision. Keep and eye on
echo $DRUPALVM_ENV: that caught me out.
drushright has taken a lot of my bandwidth over various releases. I now take the approach of using a single, locally installed
drushthat is aliased to the
vendordirectory on the host machine; uses
<project root>/drush/siteslocally for this website’s aliases, and, because NFS has the devlopment server accessing exacly the same
drushbinary for execution in both host and guest machines - i.e. in MacOS host and Linux guest VM - we know we’re at exactly the same
drushrelesase: host (a.k.a.controller); dev server; and live server.
Managing the Production Server’s secrets file
This is where the passwords for Drupal admin, drupal db; and mysql root on the Live Production Server are encoded. (The development server uses plain text values from
ansible-vault create vm/prodn_secrets.yml ansible-vault view vm/prodn_secrets.yml ansible-vault edit vm/prodn_secrets.yml
Encrypting SSL key and certificate.
ansible-vault encrypt vm/certs/SSL.crt
The Live Production Server
Initialising the AWS EC2 live server
These steps are required before we can run Ansible to automatically provision the live server with all the software we require.
Store the one-time-generated security key pair associated the AWS EC2 Server in
On EC2 server: Install Python (and editor).
Ansible needs this to work its provisioning magic. (Install
emacsor your favourite editor). Python most probably will have been installed as part of the Ubuntu Linux distro.
The following assumes that we have the AWS EC2 server account’s key pair in
~/.ssh/BAPC-2.pemon the Mac
ssh firstname.lastname@example.org -i ~/.ssh/BAPC-2.pem # from local sudo apt install emacs25-nox python # on remote
rootlogin on EC2 server.
Log in using the user automatically created when AWS launched the EC2 server (
ssh email@example.com -i ~/.ssh/BAPC-2.pem # from local sudo emacs /root/.ssh/authorized_keys # on remote
Remove the preamble
remove the preamble before the string
On local control machine: Create
vendor/iainhouston/drupal-vm/examples/prod/bootstrap/vars.ymlper the tutorial creating a new admin account (
webmaster) on the server with the password recorded in
Vault PWhere on the Mac.
On local control machine: run the ‘init’ playbook.
ansible-playbook -i vm/inventory vendor/iainhouston/drupal-vm/examples/prod/bootstrap/init.yml -e "ansible_ssh_user=root"
We should now have created
webmasterand be able to
ssh firstname.lastname@example.org thence
sudothings using the password recorded in
Vault PWhere on the Mac.
On EC2 server: revert the preamble
revert the preamble before the string
/root/.ssh/authorized_keysto prevent anyone logging into
Provisioning the Production server
Additional provisioning rôles
There are several Ansible tasks that are peculiar to our setup and are not already provided by Drupal-VM’s roles and tasks:
# setup DKIM; Place SSL cert etc. pre_provision_tasks_dir: "/pre_provision_tasks/*" post_provision_tasks_dir: "/post_provision_tasks/*"
Run the provisioning playbook
DRUPALVM_ENV=prod ansible-playbook \ vendor/iainhouston/drupal-vm/provisioning/playbook.yml \ --become --ask-become-pass \ --ask-vault-pass --inventory-file=vm/inventory \ --extra-vars="config_dir=$(pwd)/vm" \ --skip-tags=test_only
Note that we don’t use
--tags=drupal, because, in this case, we require all the Provisioning tasks to be run.
Ad hoc backups of critical datasets
On the Live Server (
/usr/local/bin/drush -r /var/www/drupal/web sql:dump \ --result-file=../bradford-abbas.uk.sql /usr/bin/s3cmd sync \ /var/www/drupal/web/sites/default/files/ \ s3://bradford-abbas.uk.files --exclude-from=/var/www/drupal/web/../.s3ignore /usr/bin/s3cmd put \ /var/www/drupal/bradford-abbas.uk.sql \ s3://bradford-abbas.uk.db
Important Drupal-VM configuration files
These are in the following directories:
config/syncis where the Drupal configuration
.ymlfiles are kept under
gitversion control. After you have run
updateLiveCodeyou then run
drush @balive cimto import the new configuration into the live site.
New configuration settings can occur both as a result of changes you make to Drupal Content Types etc., and changes to updated / newly installed Drupal Corre and Contributed modules.
When you run
config/syncis replaced by configuration
.ymlfiles exported from the development Drupal’s current database into the
drush/sitescontains the drush alias definitions for
badevWhere the shortcut / convenience commands (see above) are defined.
composerinitialisation script that will install a basic Drupal site. Very useful. Make sure your
webdirectory doesn’t exists, though, before you run
When a new SSL key / certificate pair are downloaded (via LCN, our registrar) they are encrypted here with
As expected, Ansible tasks that extend the functionality of
drupal-vmfor our needs. For example: installing and configuring DKIM’s digital ‘signing’ of emails
jinja2templates for files to be customised by Ansible before boing copied into a server during provisioning.
This is Drupal’s docroot. Make sure it doesn’t exist before you run
composer installas will be the case first thing after having cloned this repo.
Note that the Project root is
.../bradford-abbas.ukwhere you cloied this repo.
Updated by all the
composer require ...and
composer update ...we’ve done over the lifetime of the project. It is used both to establish our development and live production Drupal systems and to deploy updates through
require-devsection we have packages that are not required on the live production server (e.g.
Note that this loads the
vendor/iainhouston/drupal-vmbut also sets some important local
ENVironmen=t variables including some Ansible argumments that you might have overlooked
This is where
composerinstalls all the required libraries including the Symfony PHP modules etc. for Drupal, and other non-PHP libraries - including
These directories are created by Ansible in the project folder - i.e. a level above the Drupal docroot (where the
index.phpsits). Ansible does this in two circumstances.
On the devlopment server as a side-effect of provisioning with
On the live production server as a side-effect of provisioning with
ansible-playbook(See Run the provisioning playbook)
These two directories are purely for use by a running Drupal system