Jekyll2023-01-07T00:28:56+00:00https://iainhouston.com/feed.xmlIain Houston’s Web LogJust a few posts for public viewingIain HoustonAnecdotes of CICS Development2023-01-06T00:00:00+00:002023-01-06T00:00:00+00:00https://iainhouston.com/CICS_Restructure_Anecdotes<p>This article was made in response for a “call for history” in 2019 in anticipation of the 50th anniverary of IBM’s program product CICS. The call asked particularly for memories of the CICS Restructure project.</p>
<h1 id="anecdotes-of-cics-development">Anecdotes of CICS Development</h1>
<p>I joined the CICS Development team as a freelance programmer in 1975 / 1976 and joined IBM as a permanent member of staff in 1976. My time in CICS Development formed a vey important part of my life. I think about it a lot and am glad to have the opportunity to recall some memories.</p>
<p>Firstly:</p>
<h2 id="z-and--cics-restructure">Z and CICS Restructure</h2>
<p>CICS Restructure was a landmark event in the history of CICS, following the 1977 Exec/ Command-level Application Programming Interface in COBOL etc.which made CICS available to thousands of programmers around the world. These events definitively put Hursley’s stamp on the “inherited” software.</p>
<p>It was decided in 1983 to invest resources in a major restructuring of the internals of CICS to clarify internal interfaces and to provide the basis for future development of CICS. And to help the Restructure team make sense of the structure they wanted, and to ensure the new modules fitted in with the old modules untouched by Restructure, they used the Z notation to specify, in a precise mathematical notation, the behaviour of CICS’s restructured parts. The very substantial benefits of this decision are summarised in this publication from 1991:</p>
<p><em>I.S.C.Houston and S.King. CICS project report: Experiences and results from the use of Z in IBM.
In S. Prehn and W. J. Toetenel, editors, VDM’91: Formal Software Development Methods, volume 551 of Lecture Notes in Computer Science, pages 588-596. Springer-Verlag, 1991</em></p>
<p>But Restructure was not the only thing going on at the time. I was not part of the Restructure team itself, but I participated in many of the specification and code inspections as did other programmers from teams across CICS Development. In this way, expertise in how the restructured modules should work was combined with wider expertise on how the specifications should be implemented.</p>
<p>You can see an example of this cross-pollination between concurrent CICS product offerings when you consider that CICS Restructure was developed for CICS/ESA V3.1 for the MVS operating system platform, being developed alongside CICS/VS running with a tiny “working set” memory footprint - merely 10’s of Kilobytes! - destined for the IBM DOS OS platform (?DOS-ENTRY (program number 5736-XX6) for DOS/360 machines).</p>
<p>Both CICS/ESA Restructure and CICS/VS required a Storage Control Program (DFHSCP), and the performance constraints on the CCS/VS version of DFHSCP were paramount from the start. The more complex functionality - multiple storage pools, for example - of the restructured CICS/ESA DFHSCP was specified formally in Z but the performance characteristics were not initially a prominent part of the specification. In code inspections David Page and Iain Houston provided their CICS/VS performance-tested DFHSCP as proof of concept for some aspects the eventual performant CICS/ESA DFHSCP, namely the avoidance of unnecessary page faults.</p>
<p>The previous example brings out a significant point for those interested in the software development process we were deploying at that time. For CICS Restructure, the “waterfall” from the formal, mathematical, Z specification to code - “waterfall” being the most appropriate process here - was essentially an informal step. Peter Lupton worked with Carroll Morgan of the Programming Research Group on program verification methods; in the event the Restructure team employed one or more levels of specification-to-design “refinement” in Dijkstra’s language of guarded commands, but in most cases there was no formal relationship between the documents produced at different levels of abstraction, and so very little proof work was done to establish that the code was a correct implementation of the Z specification, although the preconditions of operations were recorded. Nevertheless, even though an ideal formal program refinement process was not possible, the benefits of being “as formal as practically possible” resulted in tremendous savings in testing costs and maintenance costs once the product had become available to customers at the end of June 1990. This was due to everybody involved knowing precisely and unambiguously how each restructured module was supposed to behave.</p>
<h2 id="queens-award-for-technological-achievement">Queen’s Award for Technological Achievement</h2>
<p>In collaboration with the Oxford University Computing Laboratory, under the leadership of Tony Hoare, this work with Z won, jointly, the Queen’s Award for Technological Achievement.</p>
<h2 id="queens-award-commemoration-events">Queens Award Commemoration Events</h2>
<h3 id="queens-award-ibm">Queens Award (IBM)</h3>
<p><img src="/assets/images/Queens_Award_Phyllis_Byrne.jpg" alt="Queens Award (IBM) (#1)" /></p>
<h3 id="queens-award-oxford-university-computing-lab">Queens Award (Oxford University Computing Lab)</h3>
<p><img src="/assets/images/Queens_Award_Computing_Lab.jpg" alt="Queens Award (Computing Lab) (#2)" /></p>Iain HoustonThis article was made in response for a “call for history” in 2019 in anticipation of the 50th anniverary of IBM’s program product CICS. The call asked particularly for memories of the CICS Restructure project.The Significance of CICS2023-01-06T00:00:00+00:002023-01-06T00:00:00+00:00https://iainhouston.com/CICS_Significance<p>I hope IBM will forgive me for reproducing the <em>CICS: Securing Online Transactions</em> article below which they published in 2009 and which is as true now, I expect, as it was then. I was motivated to post this after having met up with my newly rediscovered good friend and first line manager of the RDO project in CICS 1.7, Chris Nix.</p>
<p>So this post is partly “the significance of CICS to me” and partly “the significance of CICS to the world” and partly “the significance of CICS to IBM”</p>
<h1 id="cics-17-rdo">CICS 1.7 (RDO)</h1>
<p>I worked on CICS from 1976 for several years. At some time in 1980? I proposed an addition to the system that allowed users to add, remove, update and delete “resources” such as files, application programs, queues etc. <em>dynamically</em>, in other words without having to re-assemble definition tables and recycle the system which was the norm for systems like CICS and IMS at that time. So we developed a CICS <em>transaction program</em> we named CEDA for <em>Dynamic Allocation of resoures</em> which you might just make out in the screenshot below. We called the project RDO for <em>Resource Definition Online</em>.</p>
<p>I wrote several thousand lines of PL/S to implement the CEDA transaction. The specification and design was written using an IBM-proprietary set-theoretic specification notation (less formal than Z or VDM) and the design implemented the basic set-theoretic “classes” of the specification, and as there were no PL/S libraries avilable for this kind of thing we borrowed from the published Modula-2 Library to which I had contributed. CICS 1.7 was the first bit of CICS to be released that was written mostly in a high level language. Previously all the CICS programs were written in (the wonderful, and anything but Basic) IBM /370 Basic Assembler Language. Later releases contained <em>CICS Restructure</em> code in PL/S.</p>
<p>CICS RDO was developed by Chris Nix’s <em>Systems</em> Development team whose members also included:</p>
<ul>
<li>Ken Davies (who did the difficult bits of RDO’s interprocess syncronisation if I recall)</li>
<li>Brian Harder</li>
<li>John Stainforth</li>
<li>David Bowdler</li>
<li>Val Harris</li>
<li>Ros Reader</li>
<li>Alan Hollingshead (student)</li>
</ul>
<h1 id="ibm-article-reproduced">IBM Article reproduced</h1>
<h2 id="cics-securing-online-transactions">CICS: Securing Online Transactions</h2>
<p>If asked to list the world’s bestselling software, it is unlikely that many people would be able to name IBM ® CICS ® , the Customer Information Control System. In fact, this revolutionary product ranks as one of the top 35 technologies that shaped the industry, according to Computerworld magazine. It may well be IBM’s best kept secret.</p>
<p><img src="/assets/images/CICS_screenshot.jpg" alt="CICS Screenshot (#1)" /></p>
<h2 id="at-work-billions-of-times-every-day">At work billions of times every day</h2>
<p>CICS applications are necessary for all kinds of businesses. Many financial institutions conduct hundreds of millions of transactions every day using CICS; providing banking, investment, securities brokerage and other services, not to mention ATMs communicating with banks. CICS is at work when delivery drivers or warehouse employees scan a package and communicate tracking information to a customer or website. Systems for online trading, electronic ticketing, payroll, order entry and processing, and retail distribution all rely on CICS.</p>
<p>If you have ever withdrawn money from an ATM, taken out an insurance policy, paid a utility bill or shopped in a large retail store, then you have most likely used CICS. In all likelihood, you probably use CICS every week. And that’s really what CICS is all about—processing the world’s transactions. Securely and reliably. Day in and day out. Phil Manchester’s quote in Personal Computer Magazine sums it up:
<em>“CICS is probably the most successful piece of software of all time. … It is the mainstay of business computing throughout the world. … Millions of users unknowingly activate CICS every day, and if it were to disappear the world economy would grind to a halt.”</em></p>
<p>CICS was initially delivered around the time astronauts first landed on the moon. It was rapidly adopted by many of the world’s largest banking, insurance, telecommunications, retail, manufacturing, utilities and government organizations to run their core applications. More than 40 years after CICS was first made generally available, IBM’s commitment to deliver innovative and reliable CICS-based technologies means the world’s largest enterprises still trust CICS to run the applications that run their businesses today.</p>
<p><em>“Forty years young, with at least 40 more years to go, CICS is the backbone of so many companies around the world. When I think of CICS, I think of industry-leading transaction processing. I think of something that is really at the very core of IBM’s value proposition. CICS is clearly one of IBM’s flagship products and a great part of our portfolio.”</em><br />
Steve Mills, Senior Vice President and Group Executive, Software and Systems, IBM July 2009</p>
<p>Before CICS was invented, most application programs used batch processing, which originally involved physically loading batches of punched cards into computers for processing. Over time, this evolved into computer programs processing batches of digital data, usually overnight. CICS provided the capability to process transactions instantaneously; in other words, real-time transactions.</p>
<p>CICS also provided a collection of standard general-purpose programs, which were delivered as functions that customers could include in their own applications. With CICS providing capabilities such as security, recovery and scalability, IT programmers were free to focus on developing applications that advanced their businesses. No longer did each and every programmer have to spend time developing common capabilities that made up the “middle” layer between their applications and the computers on which they ran. The value was immediate: more powerful, more reliable applications, developed more quickly and more easily. This radical concept rapidly became the multi-billion dollar software market segment that is now known as “middleware.” And even today, CICS is a market leader.</p>
<p>The origins of CICS date back to the mid-1960s, when IBM’s Data Processing Division assembled a small team to figure out what needed to be done to improve telephone customer service. Company employees at the time had to look up paper customer records in a filing cabinet to be able to process them. Companies such as Michigan Bell Telephone, for example, wanted to see a customer’s telephone records on a screen while a company representative was talking to the customer on the phone. A feature like that would dramatically improve telephone service.</p>
<p>Ben Riggins, an IBM systems engineer, joined the IBM team with his ideas for a general online customer service application. The news soon spread that IBM had a prototype solution, and queries poured in from nearly every type of industry including airlines, utilities and financial services. Five years later, worldwide responsibility for CICS moved to the IBM development laboratory in Hursley, United Kingdom. Under the direction of Malcolm Beaver, the manager of the Hursley Programming Center at the time, IBM Hursley soon became an increasingly important center of innovation in the IT industry.
In July 1976, just two years after taking over the CICS mission, Ken Davies, a senior IBM programmer, led a team of developers in designing and implementing a major new innovation in the CICS product: the High Level Programming Interface (HLPI). This elegant replacement for the more complex macro-based capabilities allowed programmers to access CICS services and resources more easily. This new method of invoking CICS capabilities meant that writing and enhancing CICS applications, and upgrading the IT infrastructure that ran them, could all be done in a fraction of the time it used to take. Programmers who were untrained in CICS were becoming productive within weeks rather than months. This was revolutionary in the complex software industry of the 1960s, and is still pretty impressive today. HLPI technology remains at the heart of all CICS applications to this day.</p>
<p>Throughout the rapid economic development of the last 40 years, CICS has been the backbone of many of the world’s largest companies; continually evolving to enable these companies to grow to their full potential. Through the opening of the financial markets in the 1980s, the explosion of Internet commerce in the 1990s, and realizing the promise of service-oriented architecture in the 2000s, CICS has led the way.</p>
<p>Back in 1969, the problem for business was how to securely process thousands of transactions per day. Volume has grown exponentially, and the problem now is how to process thousands of transactions per second.
Each and every day, CICS secures tens of billions of online transactions. So next time you go shopping, spare a thought for IBM’s best kept secret.</p>Iain HoustonI hope IBM will forgive me for reproducing the CICS: Securing Online Transactions article below which they published in 2009 and which is as true now, I expect, as it was then. I was motivated to post this after having met up with my newly rediscovered good friend and first line manager of the RDO project in CICS 1.7, Chris Nix.Is Imperialism the only way?2022-12-20T00:00:00+00:002022-12-20T00:00:00+00:00https://iainhouston.com/Is_Imperialism_the_only_way<p>So China is quietly colonising the sources of raw materials of manufacture in Africa and South America by “partnering” with and lending to its governments. Russia’s Wagner mercenaries are ruthlessly “protecting” sources of raw materials in Africa.</p>
<p>Between them, China, Russia, and the USA are the de facto sources of a critical amount of the world’s raw materials. From rare earth to the gas, oil and metals that Europe and the UK need to create wealth or even grow an infrastructure independent of oil and gas, it is dependent upon a system of international trade that is not being sustained by the imperial reality of China, Russia, and the USA and the diminished status of Europe and the UK. Pre-Brexit EU may have been the biggest trading / consuming block but it also had an inescapable international dependency upon its sources of raw materials.</p>
<p>Looking back in history, to when Nazi Germany surprised even itself with its swift overrunning of France in 1940, it was then faced with the reality of Russia, the USA and British Empire controlling the oil and metals it needed to sustain its domain and which thus militated its suicidal attempt to capture those resources from its East.</p>
<h3 id="the-question">The Question</h3>
<p>The question is then, given that European countries, Britain and its former colonies, do not aggressively pursue the ownership of their essential raw materials in the same way as do China, Russia and the USA, can the dream of international trade with trusted counterparts ever be realised? Is the concept of Global Britain ever likely to happen before Britain is swallowed up by the new imperial powers?</p>
<h4 id="ps">PS.</h4>
<p>You might argue that Britain’s financial industry, and its creation of an alternative offshore economy, has already been swallowed up by Russian Chinese and US “investments”. And after Britain’s financial industry has swallowed up the UK’s governing party what exactly is left after Britain’s financial industry has sold off British assets like ARM and many NHS services?</p>Iain HoustonSo China is quietly colonising the sources of raw materials of manufacture in Africa and South America by “partnering” with and lending to its governments. Russia’s Wagner mercenaries are ruthlessly “protecting” sources of raw materials in Africa.Using MacOS Sidecar with iPad and Affinity2022-12-08T00:00:00+00:002022-12-08T00:00:00+00:00https://iainhouston.com/Sidecar_with_Affinity_Designer<p>Neither Apple’s nor Affinity’s documentation was precisely informative when I wanted to use MacOS Ventura’s Sidecar capability with my iPad and Affinity Designer 2.</p>
<p>Below is what I found worked quickly to allow me to draw on my iPad with the Apple Pencil whilst using Affinity Designer 2.</p>
<p>To preserve the iPad’s real estate, I leave the Designer tools and Studios etc on the Mac and use the iPad for drawing.</p>
<h2 id="separate-designers-drawing-canvas-from-studio-controls-etc">Separate Designer’s drawing canvas from studio controls etc.</h2>
<ol>
<li>
<p>Open Designer 2 on the Mac</p>
</li>
<li>
<p>Click on <code class="language-plaintext highlighter-rouge">Window >> Float View to Window</code></p>
</li>
<li>
<p>In the MacOS menu bar click on the Displays icon and then click <code class="language-plaintext highlighter-rouge">Mirror or Extend to >> Iain's iPad</code></p>
</li>
<li>
<p>Then choose <code class="language-plaintext highlighter-rouge">Use as sparate display</code></p>
</li>
<li>
<p>Now slide Designer’s separated drawing window right out of the Mac display and thus onto the iPad’s display</p>
</li>
</ol>
<p>Now I can select brushes and colours etc. on the Mac and zoom and draw with the Apple Pencil on the iPad</p>Iain HoustonNeither Apple’s nor Affinity’s documentation was precisely informative when I wanted to use MacOS Ventura’s Sidecar capability with my iPad and Affinity Designer 2.Debugging Drupal with Xdebug and Atom2022-05-12T00:00:00+00:002022-05-12T00:00:00+00:00https://iainhouston.com/Xdebug_in_Atom<h1 id="summary">Summary</h1>
<p>Atom’s docs and most web articles still refer to <strong>Xdebug 2</strong> configuration and older versions of PHP rather than the current versions of both.</p>
<p><a href="https://atom.io">Atom</a> is an excellent debugging client when connected to a remote Drupal (with <strong>Xdebug 3</strong> and <strong>PHP 8.1</strong>) and sufficient for my Drupal dev needs. Here’s how I set it up.</p>
<p>This is <em>opinionated</em> advice drawn from my experience with my simple use case. I hope it is useful to you and others with similar needs to mine.</p>
<h1 id="motivation">Motivation</h1>
<p>We aim for <em>Low Code</em> site development, to do the minimum Drupal programming of custom modules and themes, but occasionally it is necessary for me to dust off the development tools and then try to remember what I was doing those months ago when I last had to. “IDE” tasks I need to do are chiefly code editing with some debugging and inspection using Atom as an Xdebug client.</p>
<h1 id="objectives">Objectives</h1>
<p>Providing that Xdebug is set uo on the dev server with the correct <code class="language-plaintext highlighter-rouge">php.ini</code> configuration, Atom can be set up as an Xdebug client really easily so I thought that I would record what is needed. I have the following items in my component stack.</p>
<h2 id="server-side">Server-side</h2>
<ul>
<li>
<p><strong>Drupal</strong> 9.3.12</p>
</li>
<li>
<p><strong>Apache</strong> 2.4.41 running with PHP-fpm and APCu cacheing on Ubuntu 20.04 Linux. (In the past I haven’t reliably been able to use Xdebug with Nginx and PHP-fpm)</p>
</li>
<li>
<p><strong>PHP 8.1</strong> - (PHP 8.0 and later require Xdebug 3)</p>
</li>
<li>
<p><strong>Xdebug 3</strong> which has different, simpler <code class="language-plaintext highlighter-rouge">php.ini</code> settings than previously was the case in <em>Xdebug 2</em></p>
</li>
</ul>
<h2 id="client-side">Client-side</h2>
<ul>
<li>
<p><strong>MacOS</strong> (Monterey 12) as the <em>host</em> machine in which Atom is the Xdebug client and PHP Xdebug running in the <em>guest</em> virtual dev machine</p>
</li>
<li>
<p><strong>Parallels</strong> Desktop providing the virtualisation for …</p>
</li>
<li>
<p>the virtual <strong>Ubuntu</strong> Linux guest dev machine</p>
</li>
</ul>
<p>So we want to have:</p>
<ol>
<li>
<p>a connection between Xdebug running on the remote testing virtual server and</p>
</li>
<li>
<p>the Xdebug client running on our host machine as an Atom plugin.</p>
</li>
<li>
<p>We also want a browser set up with the ability to switch Xdebug step tracing on and off for each http request it makes to the remote server.</p>
</li>
</ol>
<h1 id="atom-setup">Atom setup</h1>
<p>Atom (developed by GitHub and contributors) is free of charge and a delight to use. It has (contributed) plugin packages that together can provide IDE-like functionality for many Drupal dev tasks.</p>
<ol>
<li>
<p>I assume that you have the latest <a href="https://atom.io">Atom</a> installed.</p>
</li>
<li>
<p>Install Atom’s PHP-Debug package.<br />
You will be asked to install several other dependent packages. I ended up disabling the optional packages IDE-PHP and ATOM-IDE-UI which caused errors as they seemed to be out of sync with PHP-Debug and didn’t seem to offer any extra functionality worthy of spending time investigating the error. There <em>is</em> one dependent package that is required ATOM-DEBUG-UI.
Note that <a href="https://github.atom.io/packages/php-debug">PHP-Debug’s GitHub page</a> has example <code class="language-plaintext highlighter-rouge">php.ini</code> settings for <em>Xdebug 2</em>. Ignore these. We’ll be using <em>Xdebug 3</em> because it is compatible with PHP 8.1 (more on this below)</p>
</li>
<li>
<p>Configure Atom’s PHP-Debug settings.</p>
<ol>
<li>
<p>Path mappings. Drupal programs are in the <code class="language-plaintext highlighter-rouge">themes/contrib</code> and <code class="language-plaintext highlighter-rouge">modules/contrib</code> subfolders of <code class="language-plaintext highlighter-rouge">/var/www/drupal/web</code> of the guest-remote VM whereas I am editing them in the host-local subdirectories of <code class="language-plaintext highlighter-rouge">/Users/me/mysite/web</code> and so the Atom PHP-Debug path mapping is a JSON dictionary expression like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[{</span><span class="nl">"remotePath"</span><span class="p">:</span><span class="s2">"/var/www/drupal/web"</span><span class="p">,</span><span class="nl">"localPath"</span><span class="p">:</span><span class="s2">"/Users/me/mysite/web"</span><span class="p">}]</span><span class="w">
</span></code></pre></div> </div>
</li>
<li>
<p>Port <br />
The <em>default</em> for Xdebug 3 is 9003 but I set mine to 9000 on the remote and left Atom’s PHP-Debug default 9000 on the client.</p>
</li>
<li>
<p>IDE Key.
In Xdebug 3 this is just another <em>trigger</em>. In any case I set PHP-Debug’s IDE key as <code class="language-plaintext highlighter-rouge">xdebug-atom</code> and will specify this as the trigger on the remote VM.</p>
</li>
</ol>
</li>
</ol>
<h1 id="remote-testing-server-setup">Remote testing server setup</h1>
<ol>
<li>
<p>Install Xdebug on the remote / guest VM<br />
I used <code class="language-plaintext highlighter-rouge">sudo apt install php8.1-xdebug</code></p>
</li>
<li>
<p>Configure <code class="language-plaintext highlighter-rouge">php.ini</code> on remote.
<code class="language-plaintext highlighter-rouge">apt install</code> will have created <code class="language-plaintext highlighter-rouge">/etc/php/8.1/fpm/conf.d/20-xdebug.ini</code> containing only <code class="language-plaintext highlighter-rouge">zend_extension=xdebug.so</code></p>
<p>Edit <code class="language-plaintext highlighter-rouge">/etc/php/8.1/fpm/conf.d/20-xdebug.ini</code> as follows:</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">; file /etc/php/8.1/fpm/conf.d/20-xdebug.ini
</span>
<span class="nn">[XDebug]</span>
<span class="py">zend_extension</span><span class="p">=</span><span class="s">xdebug.so</span>
<span class="py">xdebug.mode</span><span class="p">=</span><span class="s">debug</span>
<span class="py">xdebug.start_with_request</span><span class="p">=</span><span class="s">trigger</span>
<span class="py">xdebug.trigger_value</span><span class="p">=</span><span class="s">xdebug-atom</span>
<span class="py">xdebug.discover_client_host</span><span class="p">=</span><span class="s">True</span>
<span class="py">xdebug.client_port</span><span class="p">=</span><span class="s">9000</span>
<span class="py">xdebug.log</span><span class="p">=</span><span class="s">/tmp/xdebug.log</span>
</code></pre></div> </div>
<p>For good measure you should make <code class="language-plaintext highlighter-rouge">/etc/php/8.1/apache2/conf.d/20-xdebug.ini</code> identical to <code class="language-plaintext highlighter-rouge">/etc/php/8.1/fpm/conf.d/20-xdebug.ini</code></p>
<p>For a full explanation of these settings look at Xdebug’s <a href="https://xdebug.org/docs/upgrade_guide">Upgrade Guide</a> for Xdebug 2 -> 3 and <a href="https://xdebug.org/docs/">Xdebug docs</a>.</p>
</li>
<li>
<p>Restart the remote system’s web server.<br />
AFAIK it is sufficient to <code class="language-plaintext highlighter-rouge">sudo systemctl restart php8.1-fpm</code> but for good measure <code class="language-plaintext highlighter-rouge">sudo systemctl restart apache2</code> also.<br />
Maybe someone would be kind enough to comment below as to whether restarting Apache is really necessary when using PHP-fpm. I don’t think that it is.</p>
</li>
</ol>
<h1 id="browser-setup">Browser setup</h1>
<p>I use <a href="https://www.mozilla.org/en-GB/firefox/developer/">Firefox Developer Edition</a> but I know that similar <a href="https://addons.mozilla.org/en-US/firefox/addon/xdebug-helper-for-firefox/">Xdebug Helper</a> extensions exist for Chrome and <a href="https://apps.apple.com/us/app/xdebug-key/id1441712067?mt=12">for Safari</a>.</p>
<ol>
<li>
<p>Install <a href="https://addons.mozilla.org/en-US/firefox/addon/xdebug-helper-for-firefox/">Xdebug Helper for Firefox by BrianGilbert</a></p>
</li>
<li>
<p>Edit the extension’s settings to use the IDE Key <code class="language-plaintext highlighter-rouge">xdebug-atom</code></p>
</li>
</ol>
<p>If you mainly want to set breakpoints and inspect variables then I think the concepts of <code class="language-plaintext highlighter-rouge">IDE Key</code> and <code class="language-plaintext highlighter-rouge">trigger</code> are somewhat merged in Xdebug 3, so I leave the profile triggers and trace triggers blank.</p>
<h1 id="set-up-breakpoints">Set up breakpoints</h1>
<p>In Atom <em>“Move the cursor to a line you want to break on and set a breakpoint by pressing <code class="language-plaintext highlighter-rouge">Alt+F9</code>.
If everything worked correctly, you can now use the various buttons/commands to step through the script.”</em></p>
<p>The Atom client’s <em>PHP Console</em> panel indicates when the server is connected. The <em>PHP Debug</em> panel lists the breakpoints you have set and allows you to inspect the values of variables at such breakpoints.</p>
<h1 id="a-note-on-phpstorm">A note on PHPStorm</h1>
<p>I have been a JetBrains subcriber for many years but don’t need PHPStorm at this time. PHPStorm is a superb product and very useful and, as I have blogged previously, is not only a useful Xdebug client, but a fully-fledged IDE with extensive and excellent help resources.</p>
<p>But PHPStorm is not easy to set up for Xdebug. There are several PHPStorm-specific concepts to grasp when configuring it and so, having done this several times over the last decade, I am of the opinion that the learning curve is actually more demanding than that of setting up Xdebug from scratch. So, you can justify your investment in time and money in relation to the scale of the project in which you and your team are engaged.</p>Iain HoustonSummary Atom’s docs and most web articles still refer to Xdebug 2 configuration and older versions of PHP rather than the current versions of both.From Swiftmailer to Symfony Mailer2022-05-12T00:00:00+00:002022-05-12T00:00:00+00:00https://iainhouston.com/swiftmail_to_symfony_mailer<p>Drupal’s Swiftmailer module is no longer supported as our site’s Status Report reminds us and also tells us that we must switch to Drupal’s Symfony Mailer. So what to do?</p>
<h1 id="summary">Summary</h1>
<p>We switched from Drupal’s <code class="language-plaintext highlighter-rouge">mailsystem</code> and <code class="language-plaintext highlighter-rouge">swiftmailer</code> combination to Drupal’s new <a href="https://www.drupal.org/project/symfony_mailer">Symfony Mailer</a> (<code class="language-plaintext highlighter-rouge">symfony_mailer</code>) on both Live and Dev servers whilst finding a simple solution to retaining our use of Mailhog to capture outgoing email on our Dev system.</p>
<p>Although I’ve gone into some detail below, making the switch really was quite straightforward. Those who’ve done Drupal theming before shouldn’t have any problems.</p>
<h1 id="why-is-this-important-to-us">Why is this important to us?</h1>
<p>In addition to Drupal’s immediate reasons for encouraging sites to switch from Swiftmailer to Symfony Mailer there are likely long-term benefits to be gained. The <a href="https://symfony.com/doc/current/mailer.html">Symfony Mailer docs</a> show that Symfony is looking at the broader picture by having integrated into its services some of the things, like DKIM signing for example, that we have to integrate into our <code class="language-plaintext highlighter-rouge">postfix</code> instance ourselves.</p>
<p>Drupal’s Symfony Mailer, <a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/features-and-status">with its current set of features</a> addresses several scenarios for integration of the Symfony facilities into Drupal. But here I am only addressing the narrow issue of using Drupal Symfony Mailer’s <em>Legacy Mode</em> to get our use of Simplenews up and running with the new mailer.</p>
<p>We don’t use any third party mailing service like Mailchimp. We send email directly from our server’s <code class="language-plaintext highlighter-rouge">postfix</code> MTA. So testing the functionality and appearance of our simplenews issues is important to us.</p>
<p>To do this we use Mailhog (and thus <code class="language-plaintext highlighter-rouge">mhsendmail</code>) to capture emails on our Dev system. The Dev system runs in a Parallels virtual machine provisioned using our fork of <a href="https://www.drupalvm.com">Jeff Geering’s Drupal-VM</a> which has <a href="https://github.com/geerlingguy/ansible-role-mailhog">Mailhog provisioning built in</a>.</p>
<p>Our system deploys several Simplenews “issues” (distribution lists), key to the functioning of our community website. Although we send a low volume of emails, each one is important to us as several of our content types fulfil a statutary requirement.</p>
<p>We were keen to switch our Simplenews and site emails from Mail System and Swiftmailer to Symfony Mailer so long as we could do so with as few differences between Live and Dev configuration as possible.</p>
<h2 id="history---and-how-we-abandoned-our-previous-approach">History - and how we abandoned our previous approach</h2>
<p>When we were using Swiftmailer, our Dev system’s <code class="language-plaintext highlighter-rouge">php.ini</code> was configured with:</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="py">sendmail_path</span><span class="p">=</span><span class="s">/opt/mailhog/mhsendmail</span>
</code></pre></div></div>
<p>and on our our live system the <code class="language-plaintext highlighter-rouge">sendmail_path</code> was left as the default sendmail binary. In this way we could maintain an identical Drupal configuration between Dev and Live systems thus minimising potential errors by eliminating any Drupal configuration differences.</p>
<p>Were we to use the default <code class="language-plaintext highlighter-rouge">sendmail</code> binary on our Dev system, attempts to send email into the ether would not be successful as connecting MTAs would reject them as eminating from the private subnet IP adddress of the Dev virtual machine.</p>
<p>Since Symfony Mailer’s <em>Native</em> Transport would use whatever was configured in the PHP <code class="language-plaintext highlighter-rouge">sendmail_path</code> this seemed the way to go.</p>
<p>So, that <em>was</em> our approach which we tried to maintain as we switched to Symfony Mailer at which point mailing produced no errors yet neither did it produce any email output! We tried various combinations of <code class="language-plaintext highlighter-rouge">mhsendmail</code> parameters ( -t -bs -i) with varying degrees of failure. Turns out that <code class="language-plaintext highlighter-rouge">mhsendmail</code> ignores all these flags anyway.</p>
<p>I should note that to have got this far we had to have gone through the <a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/getting-started#s-installation">extra Symfony Mailer installation tasks</a>. More on this later when we talk about Symfony Mailer <em>Policies</em>.</p>
<p>So, in the end, we could not meet our objective of having both Dev and Live Symfony Mail configuration use <em>Native</em> Transport. This is the approach we abandoned as it would seem that, in the nature of the <code class="language-plaintext highlighter-rouge">sendmail</code>-style interaction between the two processes (Mailhog and Symfony), each has different expectations of the other.</p>
<p>BTW you can check that <code class="language-plaintext highlighter-rouge">mailhog</code> is working at any time as mail sent from the Linux <code class="language-plaintext highlighter-rouge">mail</code> command should correctly invoke <code class="language-plaintext highlighter-rouge">mhsendmail</code> and messages then routed to the Mailhog Web UI for inspection. e.g.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mail <span class="nt">-s</span><span class="s2">"Test Subject"</span> <span class="nt">-aFrom</span>:webmaster@my.website.uk anyemail@me.com <span class="o"><<<</span> <span class="s2">"Test Body"</span>
</code></pre></div></div>
<h1 id="our-solution">Our solution</h1>
<p>We <em>do</em> have a workable solution that allows us to:</p>
<ul>
<li>
<p>maintain a consistent Drupal configuration across both Dev and Live systems</p>
</li>
<li>
<p>trap emails in Mailhog in Dev and</p>
</li>
<li>
<p>transmit actual messages from the Live system</p>
</li>
</ul>
<p>We do this</p>
<ul>
<li>
<p>via the <em>Sendmail</em> Transport on the live system and the <em>SMTP</em> Transport on the Dev system, both using the same Drupal Symfony Mailer config settings</p>
</li>
<li>
<p>the <em>default</em> Transport in the GUI is permanently set to <code class="language-plaintext highlighter-rouge">sendmail</code> but we also have a (non-default) SMTP Transport configuration set to Host 127.0.0.1 and Port 1025 (Mailhog’s port).</p>
</li>
</ul>
<p>So the <em>Sendmail</em> configuration gets picked up in the Live system. However it gets overriden by <em>SMTP</em> in the Dev system because we do a config override in Dev’s <code class="language-plaintext highlighter-rouge">settings.php</code> thus:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$config</span><span class="p">[</span><span class="s1">'symfony_mailer.settings'</span><span class="p">][</span><span class="s1">'default_transport'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'smtp'</span><span class="p">;</span>
</code></pre></div></div>
<h2 id="theming">Theming</h2>
<p>We are talking about formatting the HTML messages generated by Simplenews when we send a node as a Newsletter Issue either as</p>
<ul>
<li>
<p>a test message to one recipient or</p>
</li>
<li>
<p>to all recipients of an issue’s distribution</p>
</li>
</ul>
<h2 id="inspecting-outgoing-email">Inspecting outgoing email</h2>
<p>We use Mailhog to be able to capture the email message and see whether it has the correct HTML and is formatted correctly according to our CSS.</p>
<p>So, first we need to enable TWIG Debugging in <code class="language-plaintext highlighter-rouge">services.yml</code> so that we can see the exactly which pieces of code are processing which pieces of markup.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">parameters</span><span class="pi">:</span>
<span class="s">twig.config</span><span class="pi">:</span>
<span class="na">debug</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>Then, in the source markup of outgoing emails, we will see the names of the TWIG templates that Drupal is looking for when it is rendering a node.</p>
<p>For example, this is what we see in the <em>Source</em> tab of the Mailhog web page after having sent a Simplenews Newsletter:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
.
<span class="c"><!-- THEME DEBUG --></span>
<span class="c"><!-- THEME HOOK: ‘node’ --></span>
<span class="c"><!-- FILE NAME SUGGESTIONS:
* node—501–email-html.html.twig
* node—501.html.twig
x node—article—email-html.html.twig
* node—article.html.twig
* node—email-html.html.twig
* node.html.twig --></span>
<span class="c"><!-- BEGIN OUTPUT from ‘themes/contrib/our_theme/templates/node—article—email-html.html.twig’ --></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">“mailpix”</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">class=</span><span class="s">“mailpix_img”</span>
<span class="na">src=</span><span class="s">“http://my.website.uk/themes/contrib/our_theme/images/email_hero.jpg”</span>
<span class="na">alt=</span><span class="s">“My</span> <span class="na">Web</span> <span class="na">Site</span> <span class="na">banner</span> <span class="na">hero</span> <span class="na">image</span><span class="err">”</span>
<span class="na">style=</span><span class="s">“max-width:</span> <span class="err">100%;</span> <span class="na">border:</span> <span class="err">0;</span> <span class="na">height:</span> <span class="na">auto</span><span class="err">;</span> <span class="na">outline:</span> <span class="na">none</span><span class="err">;”</span> <span class="nt">></span>
<span class="nt"></div></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">“link-to-page”</span>
<span class="na">style=</span><span class="s">“font-family:</span> <span class="na">serif</span><span class="err">;</span> <span class="na">font-size:18px</span><span class="err">;</span> <span class="na">line-height:</span> <span class="err">26</span><span class="na">px</span><span class="err">;</span> <span class="na">text-align:</span> <span class="na">center</span><span class="err">;</span> <span class="na">width:</span> <span class="err">100%;</span> <span class="na">margin:</span> <span class="err">0;”</span><span class="nt">></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">“http://my.website.uk/node/501”</span>
<span class="na">style=</span><span class="s">“text-decoration:</span> <span class="na">none</span><span class="err">;</span> <span class="na">font-family:</span> <span class="na">Arial</span><span class="err">,</span><span class="na">sans-serif</span><span class="err">;</span> <span class="na">font-size:</span> <span class="err">12</span><span class="na">px</span><span class="err">;</span> <span class="na">line-height:</span> <span class="err">16</span><span class="na">px</span><span class="err">;</span> <span class="na">color:</span> <span class="na">#0d77b5</span><span class="err">;”</span><span class="nt">></span>
You can view this message in our web browser
<span class="nt"></a></span>
<span class="nt"></p></span>
.
.
<span class="c"><!-- END OUTPUT from 'themes/contrib/our_theme/templates/node—article—email-html.html.twig' --></span>
</code></pre></div></div>
<p>Inspecting the above <em>DEBUG</em> information, there are several things to note here:</p>
<ol>
<li>The above example is just one (the first) set of <em>THEME HOOK</em> HTML comments in an hierarchy of the many components’ TWIG templates involved in rendering a <code class="language-plaintext highlighter-rouge">'node'</code> into an email message.</li>
<li>Symfony Mailer has its own template naming conventions, so we need to rename any templates that we previously used (more below).</li>
<li>Theme functions corresponding to theme template names will also need to be renamed. (more below).</li>
<li>The suggested template names will appear following the <em>FILE NAME SUGGESTIONS</em> line with an <em>x</em> against the one actually chosen.</li>
<li>The <em>BEGIN OUTPUT</em> line will show whether the template was chosen from our own theme or one of its base themes.</li>
<li>CSS from our theme’s <code class="language-plaintext highlighter-rouge">email.css</code> is injected inline to the html as email clients generally only handle inline styling and have no means of including external CSS files.</li>
<li><a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/getting-started#s-installation">Symfony Mailer’s installation docs</a> describe how <code class="language-plaintext highlighter-rouge">our_theme.libraries.yml</code> is configured to point to <code class="language-plaintext highlighter-rouge">email.css</code></li>
</ol>
<h2 id="renaming-theme-templates">Renaming theme templates</h2>
<p>Symfony Mailer has its own template naming conventions, so we will need to rename the templates that we used with Swiftmail. For example, after switching to Symfony Mailer our site has templates named like this:</p>
<p><code class="language-plaintext highlighter-rouge">node--article--email-html.html.twig</code> which is for our <em>Article</em> content type node being displayed in its <code class="language-plaintext highlighter-rouge">email-html</code> view.</p>
<p>And now we will probably need Xdebug to see exactly what data we are being passed in <code class="language-plaintext highlighter-rouge">$variables</code> so at this point I will refer those of you who, like me are without PHPStorm by choice or necessity, to a previous post where I describe how to simply deploy <a href="https://iainhouston.com/atom_xdebug_client/">PHP 8.1’s Xdebug 3</a> with my editor of choice, Atom (April 2022).</p>
<p>Then we have <code class="language-plaintext highlighter-rouge">email.html.twig</code> which takes the place of <code class="language-plaintext highlighter-rouge">swiftmail.html.twig</code> which is used to wrap the <em>body</em> of our simplenews messages. (I think we based the following Microsoft client-friendly HTML on a helpful Mailchimp blog post):</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span> <span class="na">xmlns:o=</span><span class="s">"urn:schemas-microsoft-com:office:office"</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">http-equiv=</span><span class="s">"Content-Type"</span> <span class="na">content=</span><span class="s">"text/html; charset=utf-8"</span><span class="nt">/></span>
<span class="c"><!--[if mso]>
<xml>
<o:OfficeDocumentSettings>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml>
<![endif]--></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="c"><!--[if mso]>
<table role="presentation" border="0" cellpadding="0" cellspacing="0" style="margin:0 auto; width:600px;"><tr><td>
<![endif]--></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"body-wrapper"</span><span class="nt">></span>
{{ body }}
<span class="nt"></div></span>
<span class="c"><!--[if mso]>
</td></tr></table>
<![endif]--></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>I would like to refer you to the <a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/getting-started#s-installation">Drupal Symfony Mailer documentation pages</a> for more info on theme file naming</p>
<h2 id="drupal-symfony-mailer-policy-markup">Drupal Symfony Mailer Policy markup</h2>
<p>As I mentioned before, it is necessary to import previously-used <code class="language-plaintext highlighter-rouge">simplenews</code> and <code class="language-plaintext highlighter-rouge">swiftmail</code> config settings into Symfony Mailer as a step in its <a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/getting-started#s-installation">installation process</a>.</p>
<p>I haven’t completely got my head around the thinking and practice behind everything the Drupal module <code class="language-plaintext highlighter-rouge">symfony_mailer</code> does with its <em>Policies</em> but we can see that they get created when we follow the installation instructions and import existing configuration data. And we can see that some pretty clever analysis is going on. For example, since Symfony Mailer is concerned with all the component parts of a mail message (<em>Body, From, Subject</em> etc.), it creates <em>Policy</em> markup configurations for each of those components as they occur in our existing Simplenews configuration YAML.</p>
<p>In the example below see what Symfony Mailer has picked up from Simplenews during import:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file symfony_mailer.mailer_policy.simplenews_newsletter.node.councillors.yml</span>
<span class="na">uuid</span><span class="pi">:</span> <span class="s">9b952dfc-...</span>
<span class="na">langcode</span><span class="pi">:</span> <span class="s">en</span>
<span class="na">status</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">dependencies</span><span class="pi">:</span>
<span class="na">config</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">simplenews.newsletter.councillors</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">simplenews_newsletter.node.councillor</span>
<span class="na">configuration</span><span class="pi">:</span>
<span class="na">email_from</span><span class="pi">:</span>
<span class="na">addresses</span><span class="pi">:</span>
<span class="pi">-</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">clerk@bmy.website.uk</span>
<span class="na">display</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Our</span><span class="nv"> </span><span class="s">Parish</span><span class="nv"> </span><span class="s">Clerk'</span>
</code></pre></div></div>
<p>… whilst retaining a dependency on the Simplenews configuration item <code class="language-plaintext highlighter-rouge">simplenews.newsletter.councillors</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file simplenews.newsletter.councillors.yml</span>
<span class="na">uuid</span><span class="pi">:</span> <span class="s">581c5a46...</span>
<span class="na">langcode</span><span class="pi">:</span> <span class="s">en</span>
<span class="na">status</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">dependencies</span><span class="pi">:</span> <span class="pi">{</span> <span class="pi">}</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">Councillors</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">councillors</span>
<span class="na">description</span><span class="pi">:</span> <span class="s1">'</span><span class="s">This</span><span class="nv"> </span><span class="s">list</span><span class="nv"> </span><span class="s">of</span><span class="nv"> </span><span class="s">email</span><span class="nv"> </span><span class="s">addresses</span><span class="nv"> </span><span class="s">are</span><span class="nv"> </span><span class="s">"subscribed"</span><span class="nv"> </span><span class="s">by</span><span class="nv"> </span><span class="s">the</span><span class="nv"> </span><span class="s">Parish</span><span class="nv"> </span><span class="s">Clerk</span><span class="nv"> </span><span class="s">and</span><span class="nv"> </span><span class="s">is</span><span class="nv"> </span><span class="s">used</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">summons</span><span class="nv"> </span><span class="s">councillors</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">meetings;</span><span class="nv"> </span><span class="s">circulate</span><span class="nv"> </span><span class="s">documents,</span><span class="nv"> </span><span class="s">etc.'</span>
<span class="na">format</span><span class="pi">:</span> <span class="s">html</span>
<span class="na">priority</span><span class="pi">:</span> <span class="s"><0></0></span>
<span class="na">receipt</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">from_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Our</span><span class="nv"> </span><span class="s">Parish</span><span class="nv"> </span><span class="s">Clerk'</span>
<span class="na">subject</span><span class="pi">:</span> <span class="s1">'</span><span class="s">[[simplenews-newsletter:name]]</span><span class="nv"> </span><span class="s">[node:title]'</span>
<span class="na">from_address</span><span class="pi">:</span> <span class="s">clerk@my.website.uk</span>
<span class="na">hyperlinks</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">allowed_handlers</span><span class="pi">:</span> <span class="pi">{</span> <span class="pi">}</span>
<span class="na">new_account</span><span class="pi">:</span> <span class="s1">'</span><span class="s">off'</span>
<span class="na">access</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">weight</span><span class="pi">:</span> <span class="m">0</span>
</code></pre></div></div>
<p>And Drupal’s Symfony Mailer GUI allows us to edit what we have imported.</p>
<p><img src="/assets/images/SimplenewsPolicy.png" alt="Editing Simplenews Policy" /></p>
<h2 id="going-live">Going Live</h2>
<p>Drupal’s Symfony Mailer alpha release doesn’t yet have every facilty that every Simplenews user will need. (Have a look at the <a href="https://www.drupal.org/docs/contributed-modules/symfony-mailer-0/features-and-status">Legacy TODO list</a>) but running Dev tests gave us confidence that we could go live with the alpha. Something others will also consider carefully.</p>
<h2 id="finally">Finally</h2>
<p>If your own experience would help improve this article, then do please leave a comment below. Thanks.</p>Iain HoustonDrupal’s Swiftmailer module is no longer supported as our site’s Status Report reminds us and also tells us that we must switch to Drupal’s Symfony Mailer. So what to do?Two important tones2021-10-07T00:00:00+00:002021-10-07T00:00:00+00:00https://iainhouston.com/Two_important_tones<p>We sit in front of the piano keyboard and someone points to seven white notes named <code class="language-plaintext highlighter-rouge">ABCDEFG</code>.</p>
<p>They ask us to play any two of these notes together and say which of these intervals are dissonant and which are consonant.</p>
<p>The answers to this question are, of course subjective and very much subject to the context in which they are played, but let us suppose that we agree with convention and that adjacent notes (a <em>step</em> apart, or the interval of a <em>second</em>) are in some sense dissonant and that the interval between the <code class="language-plaintext highlighter-rouge">F</code> and the <code class="language-plaintext highlighter-rouge">B</code> (a three tone <em>skip</em>, or the interval of a <em>tritone</em>) is dissonant and that all other combinations of two notes played together are consonant (just bear with this assumption for a moment).</p>
<p>Remove the <code class="language-plaintext highlighter-rouge">F</code> and the <code class="language-plaintext highlighter-rouge">B</code> from <code class="language-plaintext highlighter-rouge">ABCDEFG</code> and what remains is a five note or <em>Pentatonic</em> note set or scale <code class="language-plaintext highlighter-rouge">ACDEG</code>. And, as we know, the vast majority of blues, rock, folk melodies and much else are built on the consonant pentatonic scale.</p>
<p>Yes, the Pentatonic scale is usually the very first thing that we are introduced to as children as we sit at the piano, but not usually via the white notes of the <code class="language-plaintext highlighter-rouge">A minor</code> pentatonic scale <code class="language-plaintext highlighter-rouge">ACDEG</code>. So …</p>
<p>Transpose our five white notes up (or down) to the black note <code class="language-plaintext highlighter-rouge">Eb</code> and you have the scale in which we play <em>Chopsticks</em>: the five notes of the <code class="language-plaintext highlighter-rouge">Eb minor</code> pentatonic scale: <code class="language-plaintext highlighter-rouge">Eb Gb Ab Bb Db</code>. Some people think it would frighten the children to call the Chopsticks scale <em>Eb minor Pentatonic</em>. Note that we have tansposed our white notes up (or down) a Tritone. Note also that it doesn’t matter whether you do so <em>up</em> or <em>down</em> as a Tritone interval divides an octave exactly in two.</p>
<p>Transpose our five white notes up a minor third to <code class="language-plaintext highlighter-rouge">C</code> and we have <code class="language-plaintext highlighter-rouge">CDEGA</code> which is the <em>C Major Pentatonic</em> scale.</p>
<p>As an aside, knowledge of Pentatonic scales: having them “under the fingers” is quite useful for the composer and improvisor, particularly as Pentatonic melodies can be played simultaneously with a variety of chords, and very simple rules (later!) allow the composer and improvisor to control the degree of dissonance of the melodic lines against the prevaling harmony. We digress … just saying!</p>
<p>But let’s switch our attention back to the 2-note <em>Tritone</em> that we extracted from our <em>Heptatonic</em> 7-note scale to give us our 5-note <em>Pentatonic</em> scale because the two notes of the Tritone can be so vital in directing the harmonic motion in a piece of music, harmonic motion being derived largely from setting up a harmonic tension and then resolving it.</p>
<p>Let’s take a typical harmonic <em>cadence</em>.</p>
<blockquote>
<p>Cadence: a progression of two or more chords that concludes a piece of music.</p>
</blockquote>
<p>And let’s take a particular type of cadence that occurs commonly in many jazz tunes: the <code class="language-plaintext highlighter-rouge">ii-V7-I</code> with the Dominant 7th <code class="language-plaintext highlighter-rouge">V7</code> resolving to the tonic <code class="language-plaintext highlighter-rouge">I</code>. In the key of <code class="language-plaintext highlighter-rouge">C</code> that would be <code class="language-plaintext highlighter-rouge">G7</code> resolving to <code class="language-plaintext highlighter-rouge">CMAJ</code>. The dissonant Tritone <code class="language-plaintext highlighter-rouge">BF</code> supplies the tension in <code class="language-plaintext highlighter-rouge">G7</code> that needs to be resolved, <code class="language-plaintext highlighter-rouge">BF</code> being the 3rd and flatted 7th of the 4-note chord <code class="language-plaintext highlighter-rouge">G7</code>. <code class="language-plaintext highlighter-rouge">F</code> and <code class="language-plaintext highlighter-rouge">B</code> are not chord tones of the tonic <code class="language-plaintext highlighter-rouge">CMAJ</code>.</p>
<p>So, when we’re short of what to play when presented with a Dominant 7th chord, we can do a lot worse than reach for the two notes of the Tritone (Degrees <code class="language-plaintext highlighter-rouge">3</code> and <code class="language-plaintext highlighter-rouge">7</code> of <code class="language-plaintext highlighter-rouge">G7</code>e.g. <code class="language-plaintext highlighter-rouge">B</code> and <code class="language-plaintext highlighter-rouge">F</code>)</p>
<p>Since the Tritone <code class="language-plaintext highlighter-rouge">BF</code> (equally <code class="language-plaintext highlighter-rouge">FB</code>) is the “business” part of the Dominant 7th chord, we can play <code class="language-plaintext highlighter-rouge">C#7</code> which will have the same effect; <code class="language-plaintext highlighter-rouge">F</code> being the 3rd and <code class="language-plaintext highlighter-rouge">B</code> being its flatted 7th. This is an example of the so-called <em>Tritone Substitution</em> of Dominant 7ths. Whilst we get an extra bit of tension from the <code class="language-plaintext highlighter-rouge">C#</code> and the <code class="language-plaintext highlighter-rouge">G#</code> of <code class="language-plaintext highlighter-rouge">C#7</code>, we get a smooth, descending bass line through the chords’ roots, the semitones <code class="language-plaintext highlighter-rouge">D C# C</code> instead of skipping around <code class="language-plaintext highlighter-rouge">D G C</code>, the roots of the <code class="language-plaintext highlighter-rouge">ii-V-I</code>.</p>
<p>What else can we say about the Tritone? Well, as we saw above, it divides an octave exactly into two (e.g. <code class="language-plaintext highlighter-rouge">C F# C</code>). Divide that octave into four and we have <code class="language-plaintext highlighter-rouge">C D# F# A</code> which is, of course, a Diminished arpeggio or, looking at it another way, two Tritones <code class="language-plaintext highlighter-rouge">C F#</code> and <code class="language-plaintext highlighter-rouge">D# A</code> set a minor (diminished) 3rd apart.</p>
<p>Let me know if you can think of more neat properties of a Tritone.</p>Iain HoustonWe sit in front of the piano keyboard and someone points to seven white notes named ABCDEFG.Transcription of Jean Sibelius’s Tapiola2021-10-02T00:00:00+00:002021-10-02T00:00:00+00:00https://iainhouston.com/Tapiola_transcription<p>You are welcome to <a href="https://iainhouston.com/assets/downloads/Tapiola - The Full Score.pdf">view the orchestral score in your browser as a PDF file</a></p>
<p>You are welcome to <a href="https://iainhouston.com/assets/downloads/Tapiola_audio_simulation.mp3">listen to an audio file</a> too. It’s a computer simulation of a performance generated by interpreting the orchestral score.</p>
<p>Both the orchestral score and the audio file were generated by the software into which I transcribed the printed score which was published in 1959.</p>
<p>I haven’t spent any time enhancing the performance by tweaking the software’s playback settings or whatever. What you hear is simply what the computer makes of the musical notation. This serves the purpose of this project: to have on my computer the means to explore this wonderful piece of music and how the composer uses the orchestra to produce these amazing sounds.</p>
<p>If you’d like to know more about the software into which I transcribed the orchestral score, then take a look at <a href="/Dorico_iPad_Experiences/">my previous post</a></p>Iain HoustonYou are welcome to view the orchestral score in your browser as a PDF fileExperiences with Dorico for iPad2021-09-25T00:00:00+00:002021-09-25T00:00:00+00:00https://iainhouston.com/Dorico_iPad<p>I have just completed the note input phase of an orchestral work.</p>
<p>It is more than 630 bars in length for an orchestra of 34 player / section staves before Divisi.</p>
<p>The majority of note input was done on Dorico for iPad.</p>
<p>The proof reading and condensing phase is now underway, mostly on the desktop.</p>
<p>(See work in progress <a href="/tapiola_transcription/">in this post</a>)</p>
<h2 id="overall-impression">Overall impression</h2>
<p><em>TL;DR:</em> quite brilliant.</p>
<h2 id="dorico-for-ipad-and-dorico-for-macos---in-combination">Dorico for iPad and Dorico for MacOS - in combination</h2>
<p>Together they most certainly have stood up to the demands of my project, and exceeded them.
The main point of the exercise was to learn about this piece of music and I was worried I’d be side-tracked by all sorts of software limitations - as I have done in the past - but, quite to the contrary, <em>Dorico</em>, and <em>NotePerformer</em>, (with a little help from MacOS’s built-n <em>Preview</em>!) far from getting in the way have really enhanced my learning experience: being able to isolate sections of the piece, play and replay different parts has been wonderful.</p>
<h1 id="note-input">Note input.</h1>
<p>Each note was input using the <code class="language-plaintext highlighter-rouge">QWERTY</code> keyboard. Left hand on <code class="language-plaintext highlighter-rouge">ABCDEFG</code>, right hand on the keystrokes for note length, accidentals and accents.</p>
<h2 id="performance-and-reliability">Performance and reliability</h2>
<p>Both Dorico 3.5 (Desktop) and Dorico for iPad have been remarkably stable. There have been a few crashes but with no loss of work to speak of, so auto save on both platforms works when needed. On such a big project (the project file is greater than 3MB) there are noticeable periods of unresponsiveness on the iPad when auto saving, but since the program has kept all my work intact over six weeks of working on a single file, I put that down as a plus. (Yes, I have made manual and Time Machine backups continuously, but unnecessarily as it turned out).</p>
<p>Any changes whilst looking at a condensed page of the complete project could take around 45 seconds on my 2014 Mac Mini so clearly all major editing work is done without the Edit > Condensing option ON and I can’t wait until I can afford an M1 Mac next year! But regular editing with the Edit > Condensing option OFF is absolutely fine even on the slower processor. This is not in any sense a complaint, more amazement that my ancient 2-core Intel processor-fueled Mac Mini is actually capable of doing this. It’s at least 16 times slower than a moderm Mac with an Apple M1 processor.</p>
<p>I suppose I could break the project up onto smaller chunks, but I haven’t done that yet as proofreading is quite a slow process anyway.</p>
<p>Both on iPad and Desktop, crashes or stalls occurred on several occasions when dragging linked dynamics back and forth, so it was very difficult to come up with repeatable diagnostic files and scenarios with which to make a useful report to the developers.</p>
<h2 id="response-from-the-dorico-team-and-community">Response from the Dorico team and community</h2>
<p>Absolutely first class. Through the forum, they have patiently answered my questions and considered my suggestions and accepted the one or two bug reports I have made as issues on the forum.</p>
<h2 id="bugs">Bugs</h2>
<p>I have only identified a couple of genuine bugs in Dorico for iPad - concerning editing very long notes; a dialog with a numeric field that wouldn’t accept input; erroneous feedback when assigning key commands. In other words: specific to the new platform and probably the Qt framework. Considering this is a first release, and the size of my project, this has been a very good experience.</p>
<p>There are one or two features missing, and probably one or two existing features I have missed.</p>
<h2 id="missing-or-missed-features">Missing (or missed) features</h2>
<p>I made a couple of feature request for both platforms which I have raised in separate posts.</p>
<h2 id="dorico-for-ipad-ui-user-interface-comments">Dorico for iPad UI (User Interface) comments</h2>
<p>There are a couple of things worth mentioning:</p>
<h1 id="the-apple-pencil">The Apple Pencil.</h1>
<p>I certainly wouldn’t have been able to do so much on the iPad were it not for the Apple Penci, it’s been invaluable just selecting individual notes, deselecting other objects and dragging dynamics’ endpoints etc. and not just because of my large fingers.</p>
<h1 id="accidentally-editing-notes-not">Accidentally editing notes (Not).</h1>
<p>I know many people might initially baulk at the selection extension buttons in the UI of iPad Dorico which require extra clicks than you have been used to on MacOS Dorico. But in more than six weeks, zooming in and out, swiping up, down, left right and touching the screen in all sorts of ways, I can’t recall an occasion when any of this caused me to make mysterious and difficult to find accidental edits. In that respect the UI design has been very successful in protecting the investment of effort into a project.</p>
<h1 id="noteperformer">NotePerformer.</h1>
<p>The project sounds wonderful on Dorico for MacOS. I’ve yet to tweak levels in the Play Mode’s dynamics and velocity lanes to any great extent. Wouldn’t it be great to have something like this on iPadOS!</p>
<h1 id="preview-macos">Preview (MacOS)</h1>
<p>Preview’s Annotation tools have been really useful for managing source charts. It’s absoltely essential to actually know which bar you’re in and what instrument you should focus on as you flit between source score and transcribed score.</p>Iain HoustonI have just completed the note input phase of an orchestral work.Adventures in composing2021-07-10T00:00:00+00:002021-07-10T00:00:00+00:00https://iainhouston.com/Composing<p>I had an idea for an opening number. I could imagine a tune where we launched off at a fairly energetic rate in the style of one of Bob Berg’s tunes on his <em>Second Sight</em> album from 1993.</p>
<p>At this stage I have very little idea where this going to go after the initial phrase which I’d imagined for Tenor Sax in unison with the rhythm section of piano, bass and drums. Vaguely tthat it will be a good blowing number with not too complex harmonic structure to improvise over, but paying attention to the rhythm.</p>
<p>But I thought it would be worth documenting the steps I took in case it works out and I or someone else could follow the same path in the future.</p>
<h1 id="capturing-the-initial-rhythmic-idea">Capturing the initial rhythmic idea</h1>
<p>I wanted to capture the opening phrase to be led by the tenor sax. I had a good idea for the rhythm and would later experiment with melody notes from some of the altered seventh / melodic minor scales with which I’d become very familiar over recent months.</p>
<p>If I didn’t capture the idea now, it would disappear or morph into something else. Luckily my iPhone was on hand.</p>
<p>So I made a <em>Voice Memo</em> recording of my fingers beating the rhythm on the kitchen table.<br />
I tapped a bar count-in and, as it turned out, a further 8 bars of rhythm. Interesting how 8-bar melodic phrases seem to be a natural unit of composition in jazz, and I expect in other forms of music.</p>
<p><strong>Note to self:</strong> Sounds like an opportunity to set up and have some fun with the listener’s expectations by varying the phrase length when developing the tune.</p>
<h1 id="writing-down-the-rhythmic-idea">Writing down the rhythmic idea</h1>
<p>Because <em>Voice Memo</em> is an iCloud app, the recording was already on my Mac when I was ready to write down the 8 bars of rhythm.</p>
<p><em>Voice Memo</em> doesn’t use the file names you give the recordings in the app, but creates a file name using the time-stamp of the recording, so it’s easy to find it in<code class="language-plaintext highlighter-rouge">~/Library/Application Support/com.apple.voicememos/Recordings</code></p>
<p>I created a directory for the project on my Mac and copied the <em>Voice Memo</em> recording to it.<br />
I then opened the recording in <a href="https://www.seventhstring.com">Transcribe!</a> to add some bar markers. It was at this point I first realised that it was an 8 bar phrase. Because I’d recorded a count-in bar, it made it easy to tap in the barlines with the <strong>M</strong> key on my Mac keyboard.</p>
<p>So, on the top half of my screen I had <code class="language-plaintext highlighter-rouge">Transcribe!</code> open and beneath it I had <a href="https://new.steinberg.net/dorico/">Dorico</a> opened at a new project with just a single staff for a Drum Kit and playback options set to <em>Swing</em> eighth notes.</p>
<p>In this way I was able to go through my phrase bar by bar, writing it down in <em>Dorico</em> and using its playback mode to check that my notation corresponded with my recording and that I’d got the accents as clear as possible to read.</p>Iain HoustonI had an idea for an opening number. I could imagine a tune where we launched off at a fairly energetic rate in the style of one of Bob Berg’s tunes on his Second Sight album from 1993.